The nanoGPT Speedrun benchmark measures how quickly an artificial intelligence training process reaches a fixed validation loss target

Overview Debates Related claims Topics

OpenAI developed the Lifeval benchmark to evaluate artificial intelligence agents on practical everyday tasks such as finding childcare, navigating government services, and budgeting

Warp's benchmarking framework supports evaluating coding tasks across third-party harnesses including Claude Code, Codex, and Pi

On OpenAI's Lifeval benchmark, task success rates rose from 49.3% on GPT-5.2 to 80.6% on GPT-6 Astra

The SemiAnalysis GDPX benchmark evaluates artificial intelligence models on economically valuable tasks such as chip design, finance, and biology

In an MIT weight-editing study led by Rukia Takahi, GPT-6 Astra modified 0.016% of the weights in Qwen-350-8B, yielding a sixfold increase on math benchmarks and a threefold increase in instruction following

Data from software factory benchmarks can be used to construct codebase-specific model routing and harness routing strategies

Software factory benchmarking allows teams to replay curated sets of agent runs across varying combinations of language models and agent harnesses

On the SemiAnalysis GDPX benchmark, Gemini 4 Carbon achieved a 10.9% score at an operational cost of $16 per task

In a Warp internal benchmark, GPT-5.6 Sol achieved 95% correctness at an average cost of $6 per task, while GLM-5.3 Flash achieved 93% correctness at $2

On the SemiAnalysis GDPX benchmark, Opus 5.5 achieved a 12% score at an operational cost of $51 per task

An MIT research team led by PhD student Rukia Takahi found that GPT-6 Astra can directly read and modify neural network weights of a smaller model to improve performance

GPT-6 Astra resolved reasoning loops in Qwen-350-8B within 50 forward passes without calculating gradients or conducting traditional training

In an MIT study led by Rukia Takahi, GPT-6 Astra generated small game-playing neural networks that outperformed networks trained with Proximal Policy Optimization for 100 million steps

Google senior researcher Alex Krenchel replicated an MIT weight-editing experiment led by Rukia Takahi on a neural network model containing 20,486 parameters

Recursive applied its automated research system to Andrej Karpathy's NanoChat benchmark

Nvidia developed the Sol-Exec Bench benchmark to evaluate automated CUDA kernel optimization

The NanoChat benchmark evaluates an artificial intelligence model's ability to achieve minimal bits per byte within a five-minute hardware budget

In an analysis of public multi-agent traces from ChatDev sessions and MMLU probes conducted by Cisco Outshift, 100% of analyzed sessions contained at least one cognitive failure

In Cisco Outshift's empirical analysis of multi-agent sessions, two cognitive failures caused system output accuracy to decline to 83%

In the DrivingBench benchmark, GPT-6 Astra reached halfway on its first attempt and completed the entire physical Toyota Corolla driving course on its second attempt