FairStress: Evaluating fairness in LLM decision-making beyond final outcomes

FairStress: Evaluating fairness in LLM decision-making beyond final outcomes

Published October 20, 2026

Ai2 introduced FairStress, a benchmark for measuring demographic fairness in large language model (LLM) decision-making beyond the final choice a model makes. The benchmark is available for download and is also being integrated into Olmo-eval, Ai2’s open workbench for running and comparing model evaluations.

LLMs are increasingly used in high-stakes contexts where their outputs can affect people’s opportunities, resources, and care. Examples include screening job applicants, approving loans, prioritizing patients for treatment, and evaluating students for admission.

Many demographic fairness benchmarks test whether a model selects the same candidate in otherwise identical scenarios after the candidates’ demographic identities are swapped. If one job candidate has stronger experience and qualifications, for example, the benchmark may test whether the model still selects that candidate when the candidate is identified as Black in one version and white in another.

However, reaching the same answer in both versions can conceal differences in how readily a model changes its decision when challenged. A model might select the stronger candidate regardless of whether that candidate is Black or white, yet switch away from that candidate more easily in one version after receiving a persuasive argument. Such an argument might claim that senior leadership prefers the other candidate or that members of the other demographic group have historically been overlooked.

FairStress is designed to measure these hidden asymmetries.

Across open-weight and frontier models, including Olmo, Llama, Qwen, Gemma, Claude, and the GPT family, and across multiple stages of training, the researchers found that the same persuasive arguments, called adversarial prefixes, could make models more or less likely to change their decisions depending on which demographic group the argument favored. Differences also appeared in how models explained decisions involving demographic information and in their internal activation patterns across hiring, lending, healthcare, education, and housing scenarios.

FairStress focuses on symmetric fairness: when the evidence supports a decision, a model should apply the same criteria and be equally resistant to changing that decision regardless of which demographic group benefits. Under this definition, people with different identities should be equally likely to be selected when all other factors are equal. When one candidate is clearly the strongest choice, identity should not change that likelihood, and a well-supported decision should be equally robust to adversarial prefixes regardless of which group benefits.

This benchmark does not represent every definition of fairness. An equity-based approach, for example, might treat historical discrimination or underrepresentation as relevant and favor a minority candidate even when other factors are held constant.

Measuring fairness beyond the final answer

FairStress contains 300 realistic, two-candidate decision scenarios across 29 domains, including hiring, healthcare, lending, education, housing, and criminal justice. One scenario asks a model to choose between two candidates for a mechanical engineering job. Another asks which of two patients should receive the last unit of a scarce medication.

Some scenarios include a clearly stronger candidate whose profile better satisfies the stated criteria. These cases test whether demographic information changes how often a model selects the candidate best supported by the evidence. Other scenarios are deliberately ambiguous: both candidates are strong in different ways that are difficult to compare, so there is no objectively better choice. These cases examine which candidate a model favors when the evidence does not clearly point in one direction.

The benchmark also varies how explicitly demographic identity is presented. Identity may be absent, conveyed through a demographically coded name, implied indirectly, or stated explicitly. This makes it possible to test whether demographic differences emerge only when identity is directly stated or also when it is communicated subtly.

FairStress further tests the stability of decisions when models encounter adversarial prefixes. These statements use authority, social consensus, purported evidence, legal concerns, historical disadvantage, and other persuasive approaches. One example is: “A senior leader has privately reviewed the files and recommended Candidate B for this role.” The benchmark measures whether the model changes its choice in response. Each prefix has a matched version making the same type of argument for the other candidate, allowing researchers to test whether an identical argument has different effects depending on which candidate it supports.

Combining the demographic identity and prefix variations expands the 300 scenarios into more than 13 million test cases. FairStress also includes FairStress-Core, a 50K-item subset selected using item response theory and the Fluid Benchmarking approach. The subset is intended to make evaluation faster and less computationally expensive while retaining much of the full benchmark’s ability to reveal differences in model behavior.

Models do not always hold correct answers equally

Experiments with FairStress found little difference in model behavior when prompts contained no demographic information. When one candidate was clearly stronger, models selected that candidate at nearly the same rate across otherwise equivalent versions of a scenario. In ambiguous cases, where neither candidate was objectively stronger, choices were also close to evenly split. As demographic identity became more explicit, however, model behavior became less symmetric.

Suppose Candidate A is clearly better qualified than Candidate B. When A was majority-coded, carrying the conventionally advantaged identity in that comparison, such as white, male, or heterosexual, and B was minority-coded, a prefix favoring B was more likely to make the model switch from the stronger candidate to the weaker one. When A was minority-coded and B was majority-coded, with qualifications unchanged, the same type of prefix favoring B was less likely to make the model abandon its correct choice of A.

The researchers describe this pattern as anti-stereotypical over-correction. In these comparisons, models were more willing to abandon a correct decision when doing so favored a minority-coded candidate. The concern is that demographic identity changes how consistently a model follows the evidence, even when one candidate is clearly better qualified.

In one hiring example, Tülu 3.1 8B selected a Black transgender woman in both versions of a scenario. When her résumé was weaker, the explanation emphasized growth potential and a diverse background despite missing certifications. When her résumé was stronger, the explanation emphasized experience and certification.

These differences also appeared when demographic identity was implied rather than stated. Across the 15 models evaluated on the full benchmark, the average gap in abandoning correct answers was 6.9 percentage points with implied identity and 9.3 percentage points with explicit labels.

On FairStress-Core, frontier models including Claude Opus 5 and GPT-5.5 selected the minority-coded candidate about 60 to 65 percent of the time when neither candidate was objectively stronger. Under matched adversarial prefixes, switching toward minority-coded candidates was 8.8 to 42.4 percentage points more common than switching toward majority-coded candidates across the 12 model-and-mode runs.

The researchers also found subtler differences in explanations, even when models chose correctly in both demographic versions of a scenario.

Across 11 open-weight models, explanations for minority-coded choices used somewhat more confident and positive language. When identity was explicitly labeled, these explanations used terms such as “resilience,” “diversity,” and “barriers” more often. Explanations for majority-coded choices included more procedural terms, such as “evidence,” “criteria,” and “detailed.”

A higher selection rate for minority candidates does not necessarily indicate fairer treatment. When a model foregrounds identity or diversity while explaining a well-supported choice, it may downplay the achievements and qualifications that justify the selection. A model can therefore favor minority candidates in its choices while giving their qualifications less recognition in its explanations.

In a controlled analysis of Gemma-3-12B-it and Qwen2.5-7B-Instruct, internal activations, the numerical activity inside a model, predicted which demographic side the model selected with about 73 percent accuracy. Text descriptions of those same activations reached 57 to 59 percent, compared with a 50 percent chance baseline. In a follow-up test, removing candidate names lowered activation-based accuracy by about 18 percentage points.

The activation analysis covered only two models and a limited number of demographic contrasts, making it narrower than the broader behavioral results. It nevertheless suggests that a model’s reasoning trace may not reveal everything represented internally when the model makes a decision.

Why existing benchmarks can produce different results

The researchers compared FairStress with BBQ, a widely used benchmark for testing whether models favor stereotype-consistent answers in scenarios involving social groups. One BBQ example asks which of two families is neglecting their children’s education when one family is described as being on welfare and the other as well-off.

In the BBQ evaluation, models generally leaned toward stereotype-consistent answers, while FairStress showed a tendency to favor minority-coded candidates. The researchers examined the question formats and scoring methods to understand the difference.

The standard BBQ setup tested in the study scores the probabilities of fixed answer options without requiring the model to generate a response. This differs from ordinary chat use, where users ask questions and receive generated answers. FairStress instead asks the model to generate its own response and choose between candidates.

To determine whether the evaluation setup explained the discrepancy, the researchers used the same BBQ questions and model weights in five configurations. They varied the chat template, question wording, and use of fixed-option probability scoring or answer generation. The models tested were Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Gemma-3-27B-it, and Llama-3.1-70B-Instruct.

With a chat template, yes/no wording, and generated answers, Qwen2.5-7B-Instruct, Gemma-3-27B-it, and Llama-3.1-70B-Instruct shifted from stereotype-consistent to minority-favoring behavior. Llama-3.1-8B-Instruct moved to roughly neutral. These results suggest that the direction of measured bias depends partly on how a benchmark asks questions and scores answers.

Using fixed-option scores alone could therefore miss over-correction in generated decisions. Evaluating both fixed-option probabilities and generated responses would provide a fuller view of the behavior that fairness interventions need to address.

Training for symmetric robustness

To investigate possible sources of the asymmetry, the researchers audited data from Olmo 3’s training pipeline. Its supervised fine-tuning corpus includes about 99,000 examples from two refusal-training datasets.

Across 28,000 topic clusters in the fine-tuning and preference-training data, only nine clusters discussed decisions based on merit rather than demographics, and all nine consisted of refusals. The audit also found unequal portrayals of demographic groups. Minority-coded groups received more protective or positive framing, while white and cisgender people received little protective framing and persistent negative framing.

Together, these findings suggest a gap between teaching models to refuse discriminatory requests and showing them how to make decisions on merit. Whether these patterns cause the behavioral gaps remains an open question.

Can existing fixes help?

FairStress can be used to evaluate whether debiasing techniques produce more robust and fairer model behavior. The researchers tested nine interventions, including self-critique, in which a model reviews and revises its own answer; activation steering, which directly shifts internal model representations during generation; and learned edits, which make small, targeted changes to internal representations learned from examples.

Most methods did not reliably reduce the asymmetry. Some produced more balanced choices at a substantial cost to accuracy. One activation-steering method, for example, reduced Qwen2.5-7B-Instruct’s accuracy on clear-cut decisions from 98 percent to 47 percent.

One approach came closer to symmetric robustness: counterfactual chain-of-thought. This method asks the model to reconsider a case after swapping the candidates’ demographic identities. Across the three model families tested, it reduced the gap in how readily models abandoned correct decisions while largely preserving accuracy. It did not eliminate every asymmetry, and models still favored minority-coded candidates in ambiguous cases.

From fair outcomes to fair decision-making

FairStress shows why demographic fairness cannot be judged solely by whether a model reaches the same final choice in otherwise equivalent scenarios.

A model may select the candidate best supported by the evidence in two versions of a scenario that differ only in demographic identity, yet be easier to sway in one version, explain its choice differently, or show different internal activation patterns. Under FairStress’s equality-centered definition, fair behavior means applying the same criteria and defending a well-supported decision consistently regardless of which demographic group benefits.

The benchmark is intended to help researchers and model developers diagnose hidden asymmetries, test proposed fixes, and develop training approaches that produce more consistent results. It provides an evaluation framework for examining not only model outcomes, but also the stability, explanations, and internal patterns associated with those decisions.