Artificial Analysis Decision Index v2 Adds Calibration-Focused Scoring and New Benchmarks

Artificial Analysis Decision Index v2 Adds Calibration-Focused Scoring and New Benchmarks

Artificial Analysis has released version 2 of its Decision Index, an evaluation designed to measure the real-world utility of decision models while reducing the effect of benchmark gaming. Private datasets now account for 45% of the Index, compared with 25% in version 1.

Drex Max v1.1 from Nace AI leads the decision models with a score of 73, one point ahead of GLiDE v1.2 from Fastino Labs and Mammoth 27B from Baseten, which scores 72.

What changed in Decision Index v2

• Decision Score: The new headline metric rewards both correctness and calibrated confidence. Each question receives a score from 0 to 100. Correct answers score above 50, incorrect answers score below 50, and confidence in the selected answer determines the position within either range.
• AA-Paperclip, 30%: This private benchmark contains 1,500 documents, including resumes, receipts, invoices, and artificial intelligence and machine learning research papers. Models must make decisions and extract facts from each document. A document is counted as correct only when all 15 answers are correct.
• AA-Symmetry, 15%: This benchmark tests whether models change their answers when equivalent questions are presented with different wording. It uses 120 questions with reordered options, swapped labels, and negated yes/no questions, then measures consistency in the models' probabilities.
• AA-Omniscience-Decision v1.1 and HLE-MultiChoice v1.1, 15% each: Both evaluations now vary the wording of the option that allows a model to say, "I don't know," across ten alternatives. HLE-MultiChoice adds the same ten phrasings as an option to decline an answer.
• EvalSafe-O*NET-AA, 15%: The Index now includes Artificial Analysis' implementation of TypeSafe AI's O*NET benchmark.

When2Call from NVIDIA remains in the Index at a reduced weighting of 10%, down from 25%. WildGuardTest was removed because of its high saturation.

Key results

• Three decision models are within one point of the top of the Index. Drex Max v1.1 and GLiDE v1.2 both score 73, while Mammoth 27B from Baseten scores 72.
• Cost per 1 million tasks varies by more than 60 times across decision models. Celeris-1.1 Decision at $14, Decider v1.2 at $68, and Drex Max v1.1 at $102 sit on the Decision Index versus Cost Pareto frontier. Clef costs $859.
• Token pricing is not the only cost factor. Most decision models, including Drex Max v1.1 and Decider v1.2, bill a document's input once for all its questions. GLiDE v1.2 and Decider Flash v1.2 bill the input for every question. GLiDE v1.2 therefore costs $780, nearly eight times more than Drex Max v1.1 despite list prices that are only 33% higher.
• General language models can achieve higher scores, but they also require more time and cost more per task. Claude Haiku 5.5 (Max) scores 77, and GPT-6 Luna (Max) scores 76, about three points above Drex Max v1.1. GPT-6 Luna (Max) costs $1,530 per 1 million tasks compared with $102 for Drex Max v1.1, or 15 times as much. It takes 11.34 seconds per task, compared with 275 milliseconds for Drex Max v1.1, making it approximately 40 times slower.

Decision Score

The Decision Score rewards correct answers and well-calibrated confidence. In practical use, a decision model's output may be acted on when its confidence exceeds a threshold. Otherwise, the task can be passed to a person or a more capable model. The Decision Score measures the average payoff across all possible thresholds rather than depending on one selected cutoff.

For each question, the score is:

50 × (1 + c − (p − c)²)

Here, p is the probability assigned to the selected answer, and c indicates whether the answer is correct. c equals 1 for a correct answer and 0 for an incorrect answer. Across an evaluation, the same measure is expressed as:

50 × (1 + Accuracy − Brier score)

Confidence can affect the overall leaderboard. On AA-Paperclip, Clef passes 36% of documents based on raw accuracy, while Celeris-1.1 Decision passes 24%. However, Celeris-1.1 Decision matches Clef on the Decision Score because its calibration is better, with a Brier score of 0.15 compared with Clef's 0.27.

Across the full Index, GLiDE v1.2 leads Drex Max v1.1 on accuracy metrics, scoring 60 compared with 58. Drex Max v1.1 moves ahead when calibrated confidence is included in the Decision Index.

AA-Paperclip

AA-Paperclip evaluates fact extraction and decision-making on real-world documents. Its 1,500 private documents each contain five decisions and ten facts to extract. The primary metric is the percentage of documents for which the model answers all 15 questions correctly.

For the Decision Score, confidence for a document is calculated as the product of the probabilities assigned to its 15 answers.

Claude Haiku 5.5 (Max) passes 92% of documents, while GPT-6 Luna (Max) passes 86%. Among decision models, Drex Max v1.1 leads with 63%, followed by GLiDE v1.2 at 62% and Decider v1.2 at 55%.

The all-pass requirement reveals differences that may not be visible in individual-answer accuracy. Drex Max v1.1 answers 97% of questions correctly, close to Claude Haiku 5.5 (Max) at 99%, but passes 63% of documents compared with Haiku's 92%.

Decision models lose most documents on the decision questions rather than on extraction. Drex Max v1.1 makes all five decisions correctly on 71% of documents and extracts all ten facts correctly on 88% of documents.

AA-Symmetry

AA-Symmetry measures consistency when models receive equivalent questions. It starts with a fixed task set and introduces changes such as reordered options, swapped labels, and negated yes/no questions. The benchmark contains 120 base questions and 440 variations, measuring consistency both in the selected answer and in the assigned probability.

Models from Baseten lead AA-Symmetry, partly because of a high abstention rate. They consistently indicate that they do not know an answer with high confidence across variations, which produces high consistency scores.

Mammoth 27B scores 96 on AA-Symmetry, while Elephant 9B scores 93. Mammoth 27B selects the same answer across variations for 99% of questions, compared with 91% for Drex Max v1.1, 88% for GLiDE v1.2, and 67% for Clef.

Cost per task

Decision model costs range from $14 per 1 million tasks for Celeris-1.1 Decision to $859 for Clef. Celeris-1.1 Decision, Decider v1.2, and Drex Max v1.1 sit on the Decision Index versus Cost Pareto frontier. At higher scores, GPT-6 Luna (Max) extends the frontier at $1,530, and Claude Haiku 5.5 (Max) extends it at $6,240.

Token prices are only one part of decision-model costs. GLiDE v1.2 and Decider Flash v1.2 process each question in a separate pass and bill the input each time. As a result, an AA-Paperclip task uses about 63,000 input tokens for these models, compared with about 9,000 for Drex Max v1.1.

GLiDE v1.2 costs $780 per 1 million tasks at $0.04 per 1 million input tokens. Decider Flash v1.2 costs $196, nearly three times the cost of Decider v1.2 despite having half its token price.

Time per task

Celeris-1.1 Decision records the lowest time per task at 56 milliseconds. Elephant 9B follows at 70 milliseconds, and Jev 1.14 records 92 milliseconds.

Claude Haiku 5.5 (Max), by comparison, has a time per task of 32.65 seconds. Mammoth 27B scores within four points of GPT-6 Luna (Max) on the Decision Index and is 118 times faster.