21 results for: evaluation

An arXiv paper introduces a DKPS-based approach that uses cached model outputs to predict benchmark scores while substantially reducing the number of queries.

An arXiv preprint introduces MPMMine, a benchmark suite built to supply the domain artifacts and structured data constraint-acquisition methods need for reproducible evaluation.

A position paper argues AI evaluation must publish item-level benchmark responses and ships OpenEval - 10M model responses across 155k items - to prove the point.

A new study reveals that multimodal large language models struggle with clinical dermatology tasks.

A recent paper argues that alignment evaluation cannot solely rely on model-level assessments.

OpenClassGen introduces a comprehensive dataset of Python classes, enhancing LLM evaluation.

Aymara AI unveils a platform for custom safety evaluations of large language models, revealing performance gaps.

A new framework aims to improve the assessment of medical AI benchmarks, addressing key shortcomings.

ATBench unveils domain-specific benchmarks, ATBench-Claw and ATBench-Codex, enhancing trajectory safety evaluation.

A new study evaluates LLMs' legal reasoning using the Japanese bar exam's writing component.

A novel LLM-based framework provides flexible evaluation of mathematical reasoning, addressing limitations of symbolic methods.

A recent study critiques Shapley values, finding misalignment in evaluation metrics and human utility.

A new test-time matching method improves compositional reasoning in AI models, achieving state-of-the-art results.

ml-intern is an open-source agent that automates literature review, dataset discovery, training script runs, and iterative evaluation for LLM post-training work.

The RARE framework addresses evaluation flaws in redundancy-heavy document retrieval, particularly in legal and financial sectors.

A new framework exposes vulnerabilities in language model safety evaluations through concept-specific manipulations.

Mozilla patched 271 vulnerabilities after an initial security evaluation that used an early Claude Mythos Preview in collaboration with Anthropic.

An arXiv paper introduces a quantitative-plus-error-analysis benchmark for Vietnamese legal text, comparing GPT-4o, Claude 3 Opus, Gemini 1.5 Pro and Grok-1.

vla-eval decouples model inference from simulator execution with a WebSocket+msgpack protocol and Docker isolation, supporting 14 benchmarks and six model servers.