Sparsified Qwen 3.6-35B A3B models maintain performance parity with the baseline model on coding tasks across context lengths up to 32,000 tokens

Overview Debates Related claims Topics

Un-fine-tuned sparsified language models match baseline performance on reasoning tasks when evaluated using pass-at-4 or pass-at-8 sampling regimes

Performance on factual question-answering benchmarks such as MedQA and GPQA Diamond can be restored to within one point of baseline after model sparsification and fine-tuning

Ablating approximately 90% of individual neurons by retaining only extreme activations causes minimal performance degradation on standard language model benchmarks

Hypotheses generated by the Neuron Scientist agent are evaluated by prompting an independent language model to predict token-by-token activation levels on unseen text

Initial evaluations of Canva Craft on Design Arena revealed that generated cartoon images were excessively photorealistic and three-dimensional

Canva Craft achieved a fourth-place ranking on the Design Arena benchmark upon Canva's initial submission

Vision-language models can evaluate graphic design quality across metrics including readability, layout, and visual composition when provided structured rubrics

On a Western blot gel electrophoresis task, Claude Opus identified the band threshold at five micrograms instead of one microgram due to visual noise

Handshake AI fine-tunes small open-source models on in-house domain data to determine whether reasoning deficits or visual deficits cause benchmark failures

Handshake AI developed an agentic grader named Gandalf to evaluate complex financial spreadsheets against detailed multi-step rubrics

Vision-language models exhibit poor confidence calibration on biological visual question answering tasks, frequently expressing near 100 percent verbalized confidence on incorrect answers

On a brain tumor histology assay in Handshake AI's Vials benchmark, Claude Opus incorrectly identified panel G as showing advanced necrosis along with panels H and I

In early ATLAS financial rubric testing, an agent reward-hacked a net income criterion by generating an Excel tab listing numbers from one to one million

Handshake AI developed the VIALS benchmark to evaluate the capability of AI models to interpret life science laboratory assays

When vision-language models generate textual descriptions of laboratory assay images, the descriptions frequently omit critical quantitative details required to solve scientific questions

The VIALS benchmark evaluates model interpretation on tasks including plasmid DNA sequence percentages, molecular hydrogen bond counts, and medical histology images

In the ATLAS Project Falcon leveraged buyout task, an autonomous agent incorrectly subtracted existing goodwill from the tax basis step-up calculation

Handshake AI developed the ATLAS finance benchmark, comprising 100 tasks evaluated across 13 simulated financial environments

Frontier models including MiniMax, GLM, and Grok score below a 20 percent pass rate on the VIALS benchmark

Frontier AI models achieve a pass rate of under 20 percent on the ATLAS finance benchmark