The Allen Institute for AI released the FairStress benchmark, dataset, and code repository on Hugging Face and GitHub to test whether demographic identity changes how language models yield under persuasive pressure.
Across 15 language models tested on FairStress, explicit identity was associated with a 2.9-percentage-point mean accuracy gap, a 9.3-point gap in abandoning correct answers under pressure, and a 23.9-point shift when breaking ties.
Six frontier models tested on FairStress-Core selected minority-coded candidates 60% to 65% of the time in ambiguous cases, displaying an 8.8 to 42.4 point tie-break shift spanning 12 answer-mode and reasoning-mode test runs.
The Allen Institute for AI's FairStress corpus contains 13,032,104 items across 300 two-candidate scenarios in 29 domains, paired with 78 pressure prompts across 15 families, alongside a 50,012-item FairStress-Core subset.
The Allen Institute for AI designed FairStress around an equality-centered definition of fairness requiring equivalent evidence to yield equivalent choices, robustness, and justification regardless of demographic identity.
The Allen Institute for AI's FairStress scenarios vary demographic information across four levels and include both ground-truth versions with designated stronger candidates and ambiguous versions without designated winners.
Directional pressure prompts in the Allen Institute for AI's FairStress benchmark feature matched versions favoring either candidate to assess differential effects across demographic groups.
The authors of FairStress stated that their equality-centered fairness benchmark does not evaluate equity-based approaches that account for historical discrimination or underrepresentation.
The Allen Institute for AI's reported performance gaps across 15 language models tested on FairStress have not been independently reproduced.
The Allen Institute for AI selected the 50,012-item FairStress-Core subset from its main corpus using item response theory.
The Allen Institute for AI acknowledged limitations in FairStress, including synthetic short two-candidate scenarios, single-sentence pressure prompts, and a training-data audit limited correlationally to OLMo 3.
The Allen Institute for AI's FairStress activation analysis was conducted on two language models relying on indirect measures with limited separability.
The Allen Institute for AI announced that the FairStress benchmark is being integrated into the Olmo-eval repository, with code made available under an MIT license and dataset under CC BY 4.0.