Anthropic developed an internal benchmark named CoBench containing approximately 500 engineering and research problems solved by Anthropic staff

Overview Debates Related claims Topics

Inference efficiency can be measured using metrics including GPU utilization, key-value cache size per token, floating-point operations per forward pass, and latency

The GPT-4 technical report stated that OpenAI predicted GPT-4's final training loss using smaller models trained with 10,000 times less compute

The GPT-4 technical report stated that final model loss was predictable using models trained with 10,000 times less compute

Deep learning inference efficiency can be verified by measuring GPU utilization, KV cache size per token, floating-point operations per forward pass, and latency

ByteDance released EdgeBench as a benchmark to measure how rapidly AI models learn unfamiliar environments and tasks

Anthropic developed CoBench, an internal benchmark containing approximately 500 tasks previously resolved by Anthropic researchers and engineers

ByteDance released EdgeBench to evaluate the capability of AI models to learn on the fly

Language model pre-training progress can be verified using loss and perplexity benchmarks on held-out test datasets

DevRev Computer used 75% to 80% fewer tokens than Claude Code when tested on identical tasks using the same Anthropic Claude Opus model

Closed-source machine learning models can be evaluated for private data leakage by issuing targeted queries through application programming interfaces

DevRev's Enterprise Bench launched with benchmark evaluations covering Level 1 and Level 2 autonomous tasks

AlphaGenome models operate with high precision and low recall, yielding few false positives but numerous false negatives

On protein design benchmarks, Mythos 5.2 scored 85.1%, outperforming Mythos 5.1 by 6 percentage points

Un-fine-tuned sparsified language models match baseline performance on reasoning tasks when evaluated using pass-at-4 or pass-at-8 sampling regimes

Performance on factual question-answering benchmarks such as MedQA and GPQA Diamond can be restored to within one point of baseline after model sparsification and fine-tuning

Ablating approximately 90% of individual neurons by retaining only extreme activations causes minimal performance degradation on standard language model benchmarks

Hypotheses generated by the Neuron Scientist agent are evaluated by prompting an independent language model to predict token-by-token activation levels on unseen text

Sparsified Qwen 3.6-35B A3B models maintain performance parity with the baseline model on coding tasks across context lengths up to 32,000 tokens

Initial evaluations of Canva Craft on Design Arena revealed that generated cartoon images were excessively photorealistic and three-dimensional

PresentBench is a public benchmark comprising 200 tasks evaluating slide-related and presentation-related generation capabilities