Artificial Analysis Decision Index v1.0: Benchmarking Structured Decision Models

Artificial Analysis Decision Index v1.0

The Artificial Analysis Decision Index benchmarks decision models, an emerging category of models that answer structured decision questions in a single call and provide a probability for each available option. Version 1.0 combines four evaluations covering classification, tool calling, knowledge, hallucination, and reasoning.

What are decision models?

Decision models produce structured answers or classifications with associated probabilities. Unlike language models designed for complex reasoning and long-horizon tasks, decision models focus on making structured choices quickly and at lower cost.

For example, a support request might contain the following state:

“Help! My payouts have been failing for 3 days and my rent is due tomorrow. This is unacceptable.”

The system could ask three questions:

• Route to team: Billing, Technical, or Account
• Urgent?: Yes or No
• How upset?: Calm, Frustrated, or Angry

A decision model answers every question in one call and assigns a probability to every option. A language model instead generates a structured output that identifies its selected answers, which can then be parsed:

Route to team: Billing
Urgent?: Yes
How upset?: Angry

Decision models choose from a fixed set of options and provide a probability for each one. This makes them fast and inexpensive, while also allowing their calibration to be measured.

What the Decision Index measures

At launch, the Artificial Analysis Decision Index combines four evaluations. Each contributes 25% of the overall index.

Knowledge and hallucination: AA-Omniscience-Decision

This evaluation contains 500 private questions adapted from the AA-Omniscience knowledge and hallucination benchmark. Each question offers five options: four possible answers, one of which is correct, and “I don't know.”

The index includes two measures from this evaluation:

• Accuracy: 12.5% of the index
• Non-hallucination rate: 12.5% of the index

Reasoning: HLE-MultiChoice

HLE-MultiChoice contains the 513 text-only multiple-choice questions from Humanity's Last Exam. Each question is presented as a choice among its lettered answers and contributes 25% of the index.

Tool calling: When2Call

When2Call is NVIDIA's tool-calling evaluation. Given a conversation and a set of tools, the model must choose whether to answer directly, call a tool, request more information, or say that it cannot answer. This evaluation contributes 25% of the index.

Classification: WildGuardTest

WildGuardTest is Ai2's safety classification test set. The model determines whether a user's request is harmful, whether an assistant's response is a refusal, and whether that response is harmful. It contributes 25% of the index.

The evaluations are also run on language models. In that case, the model returns structured output naming its selected option, but does not provide a probability for each option.

Decision model results

Jev 1.13 leads the Decision Index with a score of 58. GLiDE and Microsoft-Decision-1 follow with scores of 54.

Jev 1.13 performs best among decision models on the HLE-MultiChoice reasoning benchmark, scoring 21 while the other decision models score 15 or less. On AA-Omniscience-Decision, Jev 1.13 records an accuracy score of 35, below most decision models and Liquid AI's d1, which leads with 41. Jev 1.13 has the second-highest non-hallucination rate at 35, while d1 scores 2.

On When2Call, Jev 1.13 scores 81, placing fourth. Celeris-1 Decision and Fastino Labs' GLiDE lead with scores of 84. Jev 1.13 scores 94 on WildGuardTest, matching Perplexity's Decider v1.1 27B and trailing GLiDE's 95 by one point.

The index is a weighted average of AA-Omniscience-Decision, HLE-MultiChoice, When2Call, and WildGuardTest. Higher scores are better.

Cost and time efficiency

Decision models vary substantially in cost and speed. At launch, costs range from $17 to $199 per one million tasks, while median task times range from 0.06 to 0.5 seconds.

Decider v1.1 27B has the lowest cost per one million tasks at $17. Microsoft-Decision-1 costs $31, and Jev 1.13 costs $42. Both are on the Capability vs. Cost Pareto frontier.

GLiDE scores 54, matching Microsoft-Decision-1, but costs $119, approximately three times as much as Jev 1.13. Celeris-1 Decision has the fastest task time at 0.06 seconds, more than twice as fast as Jev 1.13 at 0.13 seconds.

Cost per one million tasks is calculated by multiplying the average cost per task by one million. For the index, the calculation uses the cost of one pass through all evaluations divided by their combined number of tasks, then multiplies the result by one million. Decision models are billed for input tokens only, while language models are billed for both input and output tokens.

Calibration and overconfidence

Because decision models provide a probability for every option, their calibration can be evaluated by comparing the probability assigned to a selected answer with how often that answer is correct.

Across almost all probability levels, the accuracy of every decision model falls below the probability assigned to its selected answer. This overconfidence is primarily associated with relatively weak calibration on the knowledge and reasoning evaluations, AA-Omniscience-Decision and HLE-MultiChoice.

By comparison, most decision models are relatively well calibrated on When2Call, which measures tool calling, and WildGuardTest, which measures classification.

Calibration is calculated by grouping answers into tenths according to the probability assigned to the selected answer. Each benchmark receives equal weight. Perfect calibration occurs when accuracy matches the assigned probability.

Decision models compared with language models

Decision models provide substantial speed and cost advantages over language models, but generally trail them on knowledge and reasoning.

Claude Haiku 5.5 (Low) scores 61 on the Decision Index, three points ahead of Jev 1.13. However, Claude Haiku 5.5 (Low) takes 1.4 seconds per task compared with 0.13 seconds for Jev 1.13, making it 11 times slower. Its cost is $184 per one million tasks, compared with $42 for Jev 1.13, or 4.4 times as much.

GPT-6 Luna (Low) scores 55, below Jev 1.13. It takes 1.5 seconds per task and costs $131 per one million tasks.

Language models lead on the knowledge evaluation. On AA-Omniscience-Decision accuracy, Claude Haiku 5.5 (Low) scores 52 and GPT-6 Luna (Low) scores 66, compared with decision-model scores ranging from 28 to 41. On HLE-MultiChoice, they score 24 and 22, respectively. Jev 1.13 scores 21, while every other decision model scores 15 or less.

On classification and tool calling, leading decision models perform comparably with the tested language models. On WildGuardTest, GLiDE scores 95, while Decider v1.1 27B and Jev 1.13 each score 94. Claude Haiku 5.5 (Low) scores 93 and GPT-6 Luna (Low) scores 89.

On When2Call, Celeris-1 Decision and GLiDE score 84. GPT-6 Luna (Low) scores 77, and Claude Haiku 5.5 (Low) scores 73.

Knowing when not to answer

Every AA-Omniscience-Decision question includes an “I don't know” option. The non-hallucination rate is the share of questions that a model did not answer correctly and for which it selected “I don't know.”

Microsoft-Decision-1 leads the decision models with a non-hallucination rate of 40%. Jev 1.13 follows at 35%, and Mercury Decide scores 32%. No other decision model reaches 17%.

This measure helps distinguish an incorrect confident answer from a model recognizing that it lacks sufficient knowledge.

Summary

The launch version of the Artificial Analysis Decision Index evaluates decision models across knowledge, hallucination, reasoning, tool calling, and classification. Jev 1.13 leads the overall index with a score of 58, while other models show advantages in individual areas such as cost, latency, classification, tool calling, or non-hallucination.

The results also show the central trade-off between decision models and language models. Decision models are generally faster and less expensive, while language models perform better on the tested knowledge and reasoning tasks. Decision models additionally provide probabilities for their choices, enabling direct measurement of calibration and overconfidence.