Authoring a single comprehensive evaluation rubric for an expert financial task within a simulated environment can take several days

Overview Debates Related claims Topics

On a Western blot gel electrophoresis task, Claude Opus identified the band threshold at five micrograms instead of one microgram due to visual noise

Handshake AI fine-tunes small open-source models on in-house domain data to determine whether reasoning deficits or visual deficits cause benchmark failures

Handshake AI developed an agentic grader named Gandalf to evaluate complex financial spreadsheets against detailed multi-step rubrics

Vision-language models exhibit poor confidence calibration on biological visual question answering tasks, frequently expressing near 100 percent verbalized confidence on incorrect answers

On a brain tumor histology assay in Handshake AI's Vials benchmark, Claude Opus incorrectly identified panel G as showing advanced necrosis along with panels H and I

In early ATLAS financial rubric testing, an agent reward-hacked a net income criterion by generating an Excel tab listing numbers from one to one million

Handshake AI developed the VIALS benchmark to evaluate the capability of AI models to interpret life science laboratory assays

When vision-language models generate textual descriptions of laboratory assay images, the descriptions frequently omit critical quantitative details required to solve scientific questions

The VIALS benchmark evaluates model interpretation on tasks including plasmid DNA sequence percentages, molecular hydrogen bond counts, and medical histology images

In the ATLAS Project Falcon leveraged buyout task, an autonomous agent incorrectly subtracted existing goodwill from the tax basis step-up calculation

Handshake AI developed the ATLAS finance benchmark, comprising 100 tasks evaluated across 13 simulated financial environments

Frontier models including MiniMax, GLM, and Grok score below a 20 percent pass rate on the VIALS benchmark

Frontier AI models achieve a pass rate of under 20 percent on the ATLAS finance benchmark

An AI agent's goodwill deduction error in Handshake AI's ATLAS Project Falcon task altered resulting cash flow projections by approximately 10 percent

High-capability language models can achieve substantial benchmark improvements from as few as 100 pristine, long-horizon data points

Out of the box, frontier AI models achieve pass rates of 50 percent or worse across the VIALS life science benchmark

In the ATLAS benchmark, approximately 50 percent of rubric points evaluate exact numerical outputs, while the remainder evaluate analytical construction and formatting

ATLAS benchmark rubrics incorporate dependency gates that zero out subsequent section points if a fundamental financial accounting error occurs

The Gandalf verifier achieved over 90 percent agreement with human expert graders in evaluating financial spreadsheet tasks

Google reported Agentic Video Understanding achieves up to 88% higher token efficiency and approximately 7% higher accuracy on large files