🔍 Read the full analysis: FrontierCode: A Maintainer’s Questions About AI-Written Code on ThorstenMeyerAI.com
TL;DR
Artificial Analysis evaluated 23 models on FrontierCode, a benchmark built with open-source maintainers to judge whether AI-written changes meet repository standards. Claude Opus 5.5 scored highest at 52.2%, but cleared every blocking check in 57.7% of attempts. The results show that passing tests alone does not establish that a patch is ready to merge.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark designed with open-source maintainers to assess whether they would accept a proposed repository change. Claude Opus 5.5 ranked first with a 52.2% score, but it passed all blocking checks in 57.7% of attempts, leaving substantial room for human review.
FrontierCode’s results differ from benchmarks that primarily count whether code passes tests. It grades patches on six dimensions, including hidden tests, commands that check for regressions, the usefulness of the agent’s own tests, limits on files and diff size, and an AI code review against maintainers’ criteria. Any failed blocking check gives an attempt a zero, regardless of how many other tests pass.
After Opus, GPT-6.1 Sol scored 49.1% and Claude Fable 5.1 scored 49.0%. The next results included GPT-6 Astra at 47.0%, Claude Sonnet 5.5 at 45.8%, Xiaomi’s MiMo-V2.6-Pro at 45.4%, DeepSeek V4.1 Flash at 44.0% and Kimi K3 at 43.6%. Artificial Analysis reported that nine models from six labs scored between 43% and 52%.
The benchmark also reports a separate pass rate: the share of attempts that clear every blocking check. Opus’s pass rate was 57.7%, while GPT-6.1 Sol’s was 54.7%. The source says nine of the 100 tasks were not solved by any model on any attempt. Those figures concern this evaluation’s task set and setup; they do not establish how often models succeed across all software projects.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Maintainers May Still Need to Review
The results put a measurable gap between generating a plausible change and producing one that meets a maintainer’s acceptance criteria. Since a single blocking failure zeroes an attempt, a model can produce code that passes some tests and still fail the benchmark on correctness, regression risk, scope or another required check. Passing tests is not the same as earning a merge under FrontierCode’s rules.
That distinction matters to teams considering AI agents for repository work. The top model did not clear every blocking check in more than four in ten attempts in this evaluation. The benchmark does not measure the full cost or quality of real-world human review, but its results caution against treating a successful test run as proof that an AI-generated pull request is ready to accept.
Results also show that model choice can depend on the measure used. In the source’s comparison with Artificial Analysis’s Intelligence Index, Claude Sonnet 5.5 ranked second among the listed models on that index but fifth on FrontierCode; Grok 4.7 ranked 18th on FrontierCode. General capability rankings may not predict repository-level acceptance, although this comparison covers only the models and results provided.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Repository Changes
FrontierCode was developed by Cognition with open-source maintainers and announced in June 2026. Version 1.1 followed in July with rules intended to prevent agents from searching online for task solutions. More than 20 maintainers created tasks from 36 repositories, including Celery, Budibase, uppy and Mattermost. The source says maintainers spent more than 40 hours on each task.
The full benchmark contains 150 tasks; Artificial Analysis used its 100-task Main set. Tasks are private so models could not have encountered them in training, according to the source. Artificial Analysis ran each model through the same open-source agent harness, Stirrup, with a cap of 250 turns per attempt and three attempts per task. GPT-6 Luna (medium) performed the review step.
Scoring also accounts for matters that can make a patch difficult to review: changes to unrelated files, oversized diffs and unhelpful tests can reduce a score. The source quotes Celery maintainer Tomer Nosrati describing the benchmark as grading “like a tech lead,” rather than like a continuous-integration pipeline.
„Where others grade like a CI, FrontierCode grades like a tech lead.“
— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement
software development code analysis software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Available Results
The results do not show how the models would perform on every repository, task type or production workflow. They reflect 100 tasks in Artificial Analysis’s Main set, the specified agent harness and turn limits, and a review step judged by GPT-6 Luna (medium). The source does not provide enough information to determine how the scores compare with human developers completing the same tasks.
Several models had no reported FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the published ranking is not a comparison of every model discussed in the source. The source also gives task costs for several models, but cost figures alone do not establish value for a particular team or workflow.
As an affiliate, we earn on qualifying purchases.
More Model Results May Change the Ranking
Artificial Analysis’s October 12 evaluation is a snapshot, and additional model results could alter the leaderboard. The source specifically identifies Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as lacking FrontierCode scores as of October 13. It does not provide dates for when those results might appear.
For now, readers comparing coding agents can use the published scores and pass rates as evidence about performance on this benchmark, while keeping its task set and test setup in view. Whether a model can reliably produce changes that a project’s maintainers would accept in day-to-day work remains a broader question than these results alone can answer.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition with open-source maintainers. It evaluates whether AI-generated repository changes meet a set of checks and maintainer-oriented review criteria, rather than relying only on ordinary test results.
Which model scored highest?
Artificial Analysis ranked Claude Opus 5.5 (max with fallback) first, with a FrontierCode score of 52.2%. Its pass rate—attempts clearing every blocking check—was 57.7%.
Does a FrontierCode score mean a model’s code is ready to merge?
No. The benchmark reports performance on its own tasks and rules; it does not certify that a model’s code is suitable for a particular project. The source’s results also show that the top-ranked model failed at least one blocking check in a substantial share of attempts.
How many tasks and models were included in the evaluation?
Artificial Analysis tested 23 models on FrontierCode’s 100-task Main set. The full benchmark contains 150 tasks, according to the source.
What remains unknown?
The results do not establish how scores translate to every software project or how models compare with human developers on the same tasks. Several models also had no reported result as of October 13, 2026.
Source: ThorstenMeyerAI.com
