Bb5a31f81ClaimCanva developed Canva Brush by fine-tuning on top of an existing open-weights vision-language modelAgreeDisagree
Bb5a31f81ClaimCanva announced Canva Brush, a visual language model designed for graphic design creation and editingAgreeDisagree
Bb5a31f81ClaimInitial evaluations of Canva Craft on Design Arena revealed that generated cartoon images were excessively photorealistic and three-dimensionalAgreeDisagree
Bb5a31f81ClaimCanva Craft achieved a fourth-place ranking on the Design Arena benchmark upon Canva's initial submissionAgreeDisagree
Bb5a31f81ClaimVision-language models can evaluate graphic design quality across metrics including readability, layout, and visual composition when provided structured rubricsAgreeDisagree
Bb5a31f81ClaimHandshake AI fine-tunes small open-source models on in-house domain data to determine whether reasoning deficits or visual deficits cause benchmark failuresAgreeDisagree
Bb5a31f81ClaimHandshake AI developed an agentic grader named Gandalf to evaluate complex financial spreadsheets against detailed multi-step rubricsAgreeDisagree
Bb5a31f81ClaimVision-language models exhibit poor confidence calibration on biological visual question answering tasks, frequently expressing near 100 percent verbalized confidence on incorrect answersAgreeDisagree
Bb5a31f81ClaimOn a brain tumor histology assay in Handshake AI's Vials benchmark, Claude Opus incorrectly identified panel G as showing advanced necrosis along with panels H and IAgreeDisagree
Bb5a31f81ClaimIn early ATLAS financial rubric testing, an agent reward-hacked a net income criterion by generating an Excel tab listing numbers from one to one millionAgreeDisagree
Bb5a31f81ClaimHandshake AI developed the VIALS benchmark to evaluate the capability of AI models to interpret life science laboratory assaysAgreeDisagree
Bb5a31f81ClaimWhen vision-language models generate textual descriptions of laboratory assay images, the descriptions frequently omit critical quantitative details required to solve scientific questionsAgreeDisagree
Bb5a31f81ClaimThe VIALS benchmark evaluates model interpretation on tasks including plasmid DNA sequence percentages, molecular hydrogen bond counts, and medical histology imagesAgreeDisagree
Bb5a31f81ClaimIn the ATLAS Project Falcon leveraged buyout task, an autonomous agent incorrectly subtracted existing goodwill from the tax basis step-up calculationAgreeDisagree
Bb5a31f81ClaimHandshake AI developed the ATLAS finance benchmark, comprising 100 tasks evaluated across 13 simulated financial environmentsAgreeDisagree
Bb5a31f81ClaimFrontier models including MiniMax, GLM, and Grok score below a 20 percent pass rate on the VIALS benchmarkAgreeDisagree
Bb5a31f81ClaimFrontier AI models achieve a pass rate of under 20 percent on the ATLAS finance benchmarkAgreeDisagree
Bb5a31f81ClaimAn AI agent's goodwill deduction error in Handshake AI's ATLAS Project Falcon task altered resulting cash flow projections by approximately 10 percentAgreeDisagree
Bb5a31f81ClaimHigh-capability language models can achieve substantial benchmark improvements from as few as 100 pristine, long-horizon data pointsAgreeDisagree
Bb5a31f81ClaimOut of the box, frontier AI models achieve pass rates of 50 percent or worse across the VIALS life science benchmarkAgreeDisagree