Artificial Analysis Intelligence Index v5 Adds Private Benchmarks and Progress Tracking

Artificial Analysis Intelligence Index v5 Adds Private Benchmarks and Progress Tracking

October 27, 2026

Artificial Analysis has released Intelligence Index v5, its composite measure of model capabilities across tasks intended to reflect current real-world usage. The update is the largest since version 4 launched in January. Private test sets or held-out answers now account for 60% of the Index.

The release also introduces the Artificial Analysis Progress Index v1, which tracks changes in model capability over time on a continuous scale.

Changes in Intelligence Index v5

New evaluations

• FrontierCode 2.0: Cognition's private coding benchmark, consisting of real-world repository tasks. Each task is graded according to whether a repository maintainer would merge the agent's change.
• Terminal-Bench-Science: A set of 70 agentic scientific research tasks performed in a terminal environment.

Upgraded evaluations

• HLE-Diamond: Replaces Humanity's Last Exam. It is a refined, 1,000-question version split evenly between reasoning and knowledge questions.
• AA-Omniscience v1.1: Removes saturated and public-set questions, adds new questions, and revises ambiguous or erroneous items.
• CritPt v2: Uses a private, 40-question hard subset revised by the CritPt authors and evaluated with a code-execution tool.

Evaluations retained

AA-Briefcase, AutomationBench-AA, GDP.pdf, and Terminal-Bench 4.0 remain in the Index.

Evaluations removed

GDPval-AA v2.1, SciCode, and AA-LCR v1.1 were removed.

Index weighting

The 100-point Index is divided evenly among four areas:

• Knowledge Work, 25%: AA-Briefcase, 15%; AutomationBench-AA, 5%; GDP.pdf, 5%
• Coding, 25%: FrontierCode 2.0, 15%; Terminal-Bench 4.0, 10%
• Science, 25%: Terminal-Bench-Science, 15%; CritPt v2 hard subset, 10%
• General, 25%: AA-Omniscience v1.1, 15% (10% accuracy and 5% non-hallucination); HLE-Diamond, 10%

Overall results

Claude Opus 5.5 (max) leads the Index with a score of 58. GPT-6 Astra (max) follows at 57, and GPT-6.1 Sol (max) scores 55. Anthropic and OpenAI occupy the top five positions. Gemini 4 Argon (high) is next at 50.

The gap between the leading model and open weights models has increased from three points to 12. Kimi K3.1 Pro, whose weights have not yet been released, is the highest-scoring open weights model at 46. Among models with released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32, 26 points behind the overall leader and below Claude Haiku 5.5 (max) at 36.

Costs also vary considerably. Among the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the least expensive at $0.96 per Intelligence Index task, while Claude Fable 5.1 (max) is the most expensive at $13.04.

FrontierCode 2.0

FrontierCode 2.0 is Cognition's updated private coding benchmark. It contains 132 tasks drawn from real open-source repositories. A task is graded according to whether a maintainer would merge the agent's change. Any attempt that fails a blocking check receives a score of zero. The tasks remain private to reduce contamination, and each is run three times in Stirrup, an agentic harness.

Claude Opus 5.5 (max) leads with 46%. GPT-6 Astra (max), GPT-6.1 Terra (max), and Claude Sonnet 5.5 (max) each score 42%. DeepSeek V4.1 Flash 1017 (max) is the highest-scoring open weights model at 28%, 18 points behind the leader.

Terminal-Bench-Science

Terminal-Bench-Science evaluates whether agents can complete scientific research workflows in a terminal. Experts curate and review the tasks, and a task passes only when every test passes.

GPT-6 Astra (max) leads at 63%, followed by Claude Opus 5.5 (max) at 59% and GPT-6.1 Sol (max) at 58%. Scores decline sharply below these models. Kimi K3.1 Pro, the highest-scoring open weights model, reaches 34%, while Claude Haiku 5.5 (max) scores 20%. DeepSeek V4.1 Flash 1017 (max) is the best model with released weights at 13%, and most evaluated open weights models score below 10%.

HLE-Diamond

HLE-Diamond replaces Humanity's Last Exam in the Index. Developed from the refined 1,000-question version of Humanity's Last Exam from the Center for AI Safety (CAIS) and Scale AI, it contains equal numbers of reasoning and knowledge questions.

Claude Opus 5.5 (max) leads at 68%, followed by GPT-6 Astra (max) at 64% and Claude Fable 5.1 (max) at 60%.

Knowledge questions are more difficult than reasoning questions for the leading models. Claude Opus 5.5 (max) and GPT-6 Astra (max) each score 78% on reasoning questions, compared with 58% and 50%, respectively, on knowledge questions.

AA-Omniscience v1.1

AA-Omniscience v1.1 is a revised knowledge and hallucination benchmark. Saturated and public-set questions were removed, new questions were added, and ambiguous or erroneous questions were revised.

Results remain broadly aligned with version 1. Models score an average of two points lower on the AA-Omniscience v1.1 Index, with a median difference of 0.6 points. A public subset, AA-Omniscience-Public-v1.1, has also been released on Hugging Face. It was recalibrated so that accuracy, hallucination rate, and overall performance remain consistent between the public and private sets.

CritPt v2 hard subset

The Index uses a private, 40-question hard subset of CritPt v2. It contains 70 research-level physics questions minus the 30 questions that seven frontier models solved in all five attempts at launch. Models work in an offline Python sandbox in Stirrup.

GPT-6 Astra leads at 72% on high effort, followed by GPT-6.1 Sol at 69% on max effort. Claude Opus 5.5 (max), the overall Index leader, scores 61%. Kimi K3.1 Pro is the highest-scoring open weights model at 45%.

No model leads every evaluation

The v5 results leave room for progress across all nine evaluations. No single model leads every category:

• Claude Opus 5.5: FrontierCode 2.0, 46%; HLE-Diamond, 68%; AA-Omniscience v1.1, 48
• GPT-6 Astra: Terminal-Bench-Science, 63%; CritPt v2 hard subset, 72%
• Claude Sonnet 5.5: AA-Briefcase, 1812 Elo
• Kimi K3.1 Pro: Terminal-Bench 4.0, 67%
• Gemini 4 Argon: AutomationBench-AA, 78%
• Muse Wonder Full: GDP.pdf, 33% all-pass

Open weights models

Open weights models have fallen further behind in v5. Kimi K3.1 Pro, whose weights have not yet been released, is the leading open weights model at 46, 12 points behind the overall leader. In v4.3, the gap was three points.

Among models with released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) tie at 32. Claude Haiku 5.5 (max) scores higher than all of them.

The largest differences appear on HLE-Diamond and Terminal-Bench-Science. Kimi K3.1 Pro scores 37% and 34% on those evaluations, compared with 68% for Claude Opus 5.5 (max) and 63% for GPT-6 Astra (max), respectively.

Cost and effort settings

Costs differ substantially among leading models. Claude Opus 5.5 (max) costs $10.04 per Intelligence Index task, GPT-6 Astra (max) costs $6.24, and GPT-6.1 Sol (max) costs $1.27. GPT-6.1 Sol scores within four points of Claude Opus 5.5 at approximately one-eighth of the cost per task.

Effort settings have a greater effect in v5 than in v4.3. Claude Opus 5.5 scores 35 at low effort and 58 at max effort, while its cost per task rises from $0.98 to $10.04. Claude Sonnet 5.5 has a 29-point spread between low and max effort, up from 20 points in v4.3. GPT-6.1 Sol's spread doubles from 10 points to 20.

Token usage

The v5 tasks require substantially more work from models. Every evaluated model generates more output tokens per Intelligence Index task than in v4.3. The median increased by two thirds, from 35,000 to 59,000 tokens.

Agentic evaluations account for most of the token use. Terminal-Bench-Science, Terminal-Bench 4.0, AA-Briefcase, and FrontierCode 2.0 represent 55% of the Index weighting but approximately 90% of median output tokens.

Terminal-Bench-Science is the most token-intensive evaluation, accounting for about one third of output tokens per task by itself.

Artificial Analysis Progress Index v1

Alongside Intelligence Index v5, Artificial Analysis introduced Progress Index v1. It places language models released since November 2022 on a single capability scale across a broad range of benchmarks.

The Progress Index is intended to compare how underlying capability has changed over time, including comparisons between models released years apart. The Intelligence Index remains the measure for comparing models for current use cases and assessing the present frontier.

The Progress Index design incorporates lessons from the Epoch Capabilities Index developed with Epoch AI and Anthropic.