FrontierCode 1.1 Evaluates Whether AI-Generated Code Would Be Merged

FrontierCode 1.1 Evaluates Whether AI-Generated Code Would Be Merged

FrontierCode 1.1 is a coding benchmark from Cognition that measures whether AI agents can make changes to real open source repositories that maintainers would accept. The benchmark evaluates more than whether tests pass. It also examines the size and scope of changes, unnecessary file modifications, excessive diffs, and the usefulness of added tests.

Leading models still fail more than 40% of the time. Claude Opus 5.5 (max with fallback) leads the results with an average score of 52% and a pass rate of 58%.

FrontierCode 1.1 Main is the primary 100-task subset of the benchmark's 150 tasks. Every model runs in the same agent harness for consistent comparisons, and the tasks remain private to reduce contamination.

The benchmark is published as a standalone leaderboard and is not included in the Artificial Analysis Intelligence Index. It is planned for inclusion in the Coding Agents leaderboard.

How FrontierCode Works

Cognition developed FrontierCode with maintainers from 36 open source repositories. The maintainers spent more than 40 hours on each task. Cognition announced the benchmark in June 2026 and released version 1.1 in July 2026, adding rules governing fair internet use.

Each task asks an agent to modify a real repository. The FrontierCode verifier then evaluates the attempt using deterministic tests and qualitative review, including:

• Hidden tests and command checks
• Limits on which files can be modified
• Limits on how much code can be changed
• An LLM review based on criteria provided by repository maintainers

Checks are classified as blocking or non-blocking. An attempt that fails any blocking check receives a score of zero. An attempt that passes all blocking checks receives a score between 70% and 100%, depending on its performance against desirable non-blocking criteria.

Methodology

• Task set: The 100 tasks in FrontierCode 1.1 Main, with three attempts per task
• Agent harness: Stirrup, an open source agent harness, with a limit of 250 turns per attempt
• Grading: GPT-6 Luna (medium) evaluates the rubric's review criteria
• Internet use: Attempts flagged by Cognition's trajectory scanner for looking up solutions receive a score of zero
• Scoring: The three attempts for each task are averaged, then task scores are averaged with equal weight

Key Findings

• Leading models fail more than 40% of the time. Claude Opus 5.5 (max with fallback) leads with a 52% score and passes every blocking check in 58% of attempts. GPT-6.1 Sol (max) and Claude Fable 5.1 (max with fallback) follow at 49%.
• The top of the leaderboard is tightly grouped. Nine models from six labs score between 43% and 52%.
• MiMo-V2.6-Pro is the leading open weights model at 45%, followed by DeepSeek V4.1 Flash (max) and Kimi K3 (max) at 44%.
• GPT-6.1 Sol (max) has the strongest score relative to cost, scoring 49% at $0.81 per task. This matches Claude Fable 5.1 (max with fallback) while costing less than one-tenth as much per task as the leading Anthropic models.
• Nine of the 100 tasks are passed by no model on any attempt. No task is passed by every model on every attempt.

Pass Rate

Each task is run three times. An attempt passes when it meets every blocking check. Pass rates range from 58% of attempts for Claude Opus 5.5 (max with fallback) to 16% for Muse Glimmer (high).

Cost

Cost per task varies by more than 100 times, from $0.09 for GPT-6 Luna (max) to $12.00 for Claude Fable 5.1 (max with fallback).

Four models form the score-versus-cost Pareto frontier:

• GPT-6 Luna (max): 42% at $0.09 per task
• MiMo-V2.6-Pro: 45% at $0.18 per task
• GPT-6.1 Sol (max): 49% at $0.81 per task
• Claude Opus 5.5 (max with fallback): 52% at $10.15 per task

The final step from GPT-6.1 Sol (max) to Claude Opus 5.5 (max with fallback) adds three percentage points while increasing the cost per task by more than 12 times.

GPT-6.1 Sol (max) scores two points higher than GPT-6 Astra (max), while costing substantially less per task: $0.81 compared with $4.22. Claude Sonnet 5.5 (max with fallback) uses approximately 50% more output tokens per task than Claude Opus 5.5 (max with fallback). However, its lower per-token price results in a lower total cost, $8.26 compared with $10.15.

Evaluation costs are divided among input, cache-hit, cache-write, reasoning, and answer-token pricing when canonical token counts are available.

Token Usage and Time

GPT-6 models lead in token efficiency. GPT-6 Astra (max) and GPT-6.1 Sol (max) use approximately 27,000 and 33,000 output tokens per task, respectively. Claude Opus 5.5 (max with fallback) uses approximately 157,000 output tokens per task, while Claude Sonnet 5.5 (max with fallback) uses approximately 236,000, the highest figure in the evaluation.

Time per task depends on both token usage and live output speed. It ranges from approximately three minutes for Inkling (xhigh) to approximately 46 minutes for Qwen3.8 Max (0902).

FrontierCode 1.1 Resources

• The FrontierCode 1.1 evaluation page contains the leaderboard.
• The methodology documentation describes the implementation, grading, and scoring process.
• Cognition's posts, Introducing FrontierCode and FrontierCode 1.1, provide additional information about the benchmark and its design.
• Stirrup, the open source agent harness used for FrontierCode, is available on GitHub.