The Best Coding Agent Passed 70% of Acceptance Criteria but Fully Finished Only 26.4% of Real Tasks. Who Checks the Rest?

OpenAI’s GPT-6 Astra, the top model among ten coding agents tested on real user requests, passed about 70% of each task’s acceptance criteria but fully finished only 26.4% of tasks, according to Co-Design Bench, a benchmark from researchers at Alibaba and five universities published on 26 Oct 2026. The other nine models scored lower. Opus 5 fully finished 18.3% of tasks, and the lowest-scoring model passed 51.1% of criteria.

The article covers what the benchmark found, why the unfinished part of agent work passes a diff review, and how to review each pull request (PR) against the ticket behind it.

TL;DR

The best agent finished 26.4% of tasks in full. OpenAI’s GPT-6 Astra passed an average of 69.9% of the hidden acceptance criteria per task, and 73.6% of tasks kept at least one criterion unmet, according to the Co-Design Bench paper.
The tasks are hard on purpose, and the grading has limits. The researchers picked requests with a history of rephrasing and rollbacks, graded with LLM judges and ran every model in one harness, the agent software that lets a model read, edit and run code. Rankings changed when the harness changed.
Most unfinished tasks are mostly done, and the missing part is a requirement from the ticket that the agent never built. A task that meets nine criteria of ten still reads as complete in a PR, because the tenth, such as a discount that never reaches the order summary, is code that should exist and doesn’t, so a diff review has nothing to flag. The review conclusion is Qodo’s inference, since the paper measures agents, not code review.
Qodo reviews every PR against its linked ticket from Jira, Linear, Azure DevOps, GitHub Issues, GitLab Issues or Monday.com, and can flag missing or incomplete requirements as findings in the PR, so teams merge agent-generated code sooner with fewer rollbacks.

What Co-Design Bench Found About Coding Agents

Co-Design Bench asks coding agents to revise working web applications the way real users asked for changes, sometimes with an image, audio clip, video or document attached. The researchers built 273 tasks from real coding sessions and wrote 2,331 hidden acceptance criteria for them. Each agent got the request and a runnable app, never the criteria.

Judges then built and ran each finished app in a browser and graded every criterion as Pass, Partial or Fail. A task counted as fully finished only when every criterion passed.

What was measured or reported	Detail	Source
Tasks	273 tasks curated from 37,463 deduplicated queries across 26,314 real user coding sessions	Co-Design Bench paper, 26 Oct 2026
Acceptance criteria	2,331 hidden, expert-authored criteria	Co-Design Bench paper
Best model	GPT-6 Astra passed 69.9% of criteria per task on average and fully finished 26.4% of tasks	Co-Design Bench paper, Table 5
By track	GPT-6 Astra fully finished 41.0% of 100 multimodal editing tasks and 17.9% of 173 repository editing tasks	Co-Design Bench paper, Table 2
Next models	Opus 5 at 66.1% of criteria and 18.3% of tasks; Qwen3.8-Max at 65.3% and 16.5%	Co-Design Bench paper, Table 5
Unfinished tasks	73.6% of tasks kept at least one criterion that was not fully met, for the best model	Co-Design Bench paper
Breadth of change	Repository edits spanning three or four change types scored 59.1% of criteria, against 65.4% for one or two, across all ten models	Co-Design Bench paper
Harness	Every model ran in the Claude Code harness; swapping harnesses moved Qwen3.8-27B between 56.0% and 74.9% on repository edits	Co-Design Bench paper, Appendix H
Task selection	Requests ranked by difficulty, verifiability and past failure signals such as reformulations and rollbacks	Co-Design Bench paper
Grading	An LLM judge, a separate LLM verifier that challenges every positive verdict, and two experts who checked 200 outputs	Co-Design Bench paper

The best of ten models in Co-Design Bench passed 69.9% of acceptance criteria per task on average but fully finished 26.4% of 273 tasks.

Earlier benchmarks look very different. On WebCompass-edit, a 2026 web editing test, Opus 4.8 reached 99.4% feature completion when it could run and fix its own code, the paper reports. The paper notes the two tests use different task pools and grading. In most cases, “the agent handles the bulk of the work but leaves at least one detail unfinished,” The Decoder reported on 8 Nov 2026.

The paper names its own limits. The tasks were picked for difficulty, each one is a single fixed request with no follow-up questions, and the authors write that retained evidence “cannot eliminate judgment error.” Alibaba researchers built the benchmark and Alibaba’s Qwen models placed third and fourth by share of criteria passed, though an OpenAI model ranked first. All tasks are web applications, not backend services.

Even read with every caveat, the result points at one place in the workflow, the moment a person decides the agent’s work is done.

Why Co-Design Bench Matters: Agent Work Looks Finished Before It Is

Every coding agent reports a task as done when it stops working on it, and a mostly finished change looks the same as a finished one. “Substantial coverage of individual requirements can coexist with limited complete-task success,” the authors write. A task where the agent met nine criteria of ten scores well on coverage and still fails the user.

Many teams assume that an agent writing code from a user story covers the story’s acceptance criteria, so PR review adds little. Co-Design Bench tests that assumption directly, with hidden criteria the agent never saw, and the best model met every criterion in about one task in four.

Understanding whether code matches its intent is the greatest AI code review challenge for 13.2% of developers in Qodo’s 2026 survey of 500 US developers, as Qodo’s AI code review guide reports. A reviewer who reads only the diff answers a narrower question, whether the code that changed looks right.

A mostly finished change hides its unmet ticket requirement in plain sight. The cost shows up in four places.

An example ticket with three requirements and a pull request diff that covers two, with a gap where the third requirement's code should be. Example data.

What Breaks When Code Review Reads Only the Diff

The missing piece is an unmet ticket requirement, which no diff shows. A diff shows what changed. An unmet criterion is code that should exist and doesn’t, such as a discount that never reaches the order summary, so a line-by-line review finds nothing wrong with the lines it sees.
Broad changes miss more. Repository edits spanning three or four change types scored lower than edits spanning one or two, across all ten models, and 166 of the 173 repository tasks required more than one type. Each extra layer, state or content change is one more place for a detail to slip.
“Done” depends on the harness as much as the model. The same model scored between 56.0% and 74.9% of criteria on repository edits across three harnesses, so a team can’t infer from a leaderboard how finished its own agent’s PRs will be.
Leaders can’t see how often agent PRs ship unfinished. Keeping visibility into what code agents create or modify is one of the biggest quality and governance gaps, named by 38.3% of engineering leaders in the same Qodo survey, and a missing requirement leaves no trace in the merged code.

All four breaks share one cause. The requirements live in the ticket, and a diff review never opens the ticket.

So Who Checks the Rest of an Agent’s Work?

Code review checks the rest of an agent’s work only when the review reads the ticket alongside the diff, because the unfinished part looks done in a diff. The benchmark measures agents, not reviewers, so the review conclusion is Qodo’s inference. A review that catches unfinished agent work needs four parts, one for each break.

The linked ticket’s requirements inside the review, so a missing piece shows up as a gap against the ticket.
The ticket’s acceptance criteria and subtasks read item by item, so broad changes get checked part by part.
The same check on every PR, whichever agent wrote it, in whatever harness, with findings the agent can act on.
One record for leaders across repositories and PR authors.

A ticket that is only a vague idea, with no written acceptance criteria, gives review nothing to check against. Requirement checks start with written requirements.

How Qodo Reviews Agent PRs Against the Ticket

Qodo, the AI Code Review Platform, is the quality and governance layer for teams moving toward a more autonomous SDLC (software development lifecycle). Qodo answers the four breaks in order, with ticket context in every review, acceptance criteria and subtasks, the same review for every agent and analytics for leaders.

Pull requests from three coding agents, each linked to a ticket, reviewed by Qodo against the ticket's acceptance criteria, with findings shown per PR. Example data.

Qodo Checks Each PR Against Its Linked Ticket

Qodo links a ticket to a PR when the PR description contains a link to the ticket or the branch name matches a ticket reference such as feature/PROJ-123. During review, Qodo evaluates how well the changes align with the ticket’s stated intent and reports what it finds as findings in the PR, including missing or incomplete requirements, according to Qodo’s ticket context documentation.

Requirement-aware feedback is on by default. Qodo supports Jira Cloud and Jira Data Center, Linear, Azure DevOps work items, GitHub Issues, GitLab Issues and Monday.com.

Acceptance Criteria and Subtasks Come Into the Review

Qodo fetches each ticket’s title and description, plus subtasks or child items where the provider has them. Acceptance criteria come from a Jira custom field or from the dedicated field in Azure DevOps work items. Qodo also notes when an implementation looks correct but needs manual validation, such as UI testing.

A team can turn on detection of unrelated changes, so Qodo flags changes that don’t match the ticket’s intent. Enterprise engineering teams get one review against the same tickets across many repositories and coding agents.

The Same Review Runs Whichever Agent Wrote the PR

Qodo runs a review automatically when a PR is opened, reopened or marked ready for review, whichever agent wrote the change, in whatever harness. Review Resolver, an Agentic Toolbox skill in Qodo’s Codex and Claude Code plugins, lets the agent that wrote the code retrieve Qodo’s findings and resolve the ones a developer asks for, with an explicit outcome for each.

Each time Qodo reviews a new commit, it re-checks every open finding and marks it resolved when the changes address it. High-velocity teams keep the coding agents they already use and get every PR checked, with setup in about 5 minutes on GitHub or Bitbucket Cloud and no new process.

Code Review Analytics Shows Leaders How Teams Act on Findings

Qodo’s code review analytics shows admins how teams respond to findings across PRs, repositories and PR authors. Leaders filter by severity, category and finding type over ranges such as the last 30 or 90 days, see which high-severity findings remain unaddressed, and compare accepted findings across repositories.

Where Unfinished Agent Work Slips Through: 2 Examples

Unfinished work example 1: a discount that never reached the order summary

A product team gave Claude Code a Jira story for a checkout page with five acceptance criteria in a custom field. The PR added the discount field, passed its tests and read cleanly in the diff. The criterion that the order summary shows the discounted total was never implemented.

The branch name carried the ticket key, so Qodo pulled the story and its acceptance criteria into the review and flagged the summary total as a missing requirement. Claude Code retrieved the finding through Review Resolver and added the update, and Qodo marked the finding resolved on the next commit.

Unfinished work example 2: a broad repository change with an extra edit

A team asked Codex to rework a settings page across layout, state and content, tracked as an Azure DevOps work item with acceptance criteria in its dedicated field. The PR covered every criterion but one and also renamed a shared component the work item never mentioned.

With unrelated-change detection on, Qodo flagged the rename, reported the incomplete requirement and noted that a theme toggle needed manual UI testing. The developer reverted the rename, Codex resolved the gap, and a reviewer checked the toggle by hand.

Hands-On: Review Agent PRs Against the Ticket With Qodo

The steps follow Qodo’s docs as of 9 Nov 2026. Ticket keys and settings values are examples.

Install Qodo on your Git provider. Installation takes about 5 minutes on GitHub and on Bitbucket Cloud, and around 5 to 10 minutes on GitLab and Azure DevOps. Reviews then run automatically when a PR opens.
Connect your ticketing system. For Jira Cloud, go to the Qodo Integrations page, click Connect Jira Cloud and click Accept, as Qodo’s Jira guide describes. Jira Data Center uses a personal access token in the configuration file.
Write acceptance criteria where Qodo reads them. In Jira, keep them in a custom field. In Azure DevOps, use the work item’s Acceptance Criteria field.
Link each ticket to its PR. Put the ticket link in the PR description, or put the key at the start of the branch name or right after a slash, such as feature/PROJ-123-add-discount. For Jira Cloud branch names and short ticket IDs, add jira_base_url under [jira] in the configuration file. In Azure DevOps, link the work item under Work items to link on the PR.
Turn on unrelated-change detection in the configuration file, if you want it. Requirement-aware feedback is already on by default.

[pr_reviewer]
check_pr_additional_content = true

Resolve findings from the agent. After Qodo reviews an open PR, ask the agent that wrote it to resolve the findings you choose through Review Resolver.

Fix all high-severity findings.

Read the record. As an admin, open Analytics in the Qodo portal, choose Last 30 days, and break the data down by repository, PR author or finding type.

Summary

Co-Design Bench gave ten coding agents 273 real user requests with 2,331 hidden acceptance criteria. The best model, GPT-6 Astra, passed 69.9% of criteria per task on average and fully finished 26.4% of tasks, with Opus 5 second. The tasks were picked for difficulty, LLMs did the grading and the harness changed the results.

Most unfinished agent work is mostly done, and the missing part is a requirement from the ticket that the agent never built. A diff can’t show code that doesn’t exist, so a review that skips the ticket passes it. Qodo reviews every PR against its linked ticket and acceptance criteria, can flag missing or incomplete requirements as findings, lets Codex and Claude Code resolve them through Qodo’s plugins, re-checks open findings on every commit and shows leaders the record in analytics.

Teams merge agent-generated code sooner with fewer rollbacks. Enterprise engineering teams get one review against the same tickets across many repositories and coding agents. High-velocity teams keep their coding agents, get every PR checked and finish setup in about 5 to 10 minutes on cloud Git providers. Qodo’s AI code review guide covers the basics, and its post on where code rules should live covers review across several coding agents.

FAQs

What is Co-Design Bench?

Co-Design Bench is a coding agent benchmark of 273 tasks built from real user coding sessions, with 2,331 hidden acceptance criteria. Agents revise working web applications from user requests, and LLM judges grade each criterion in a browser. Researchers from Alibaba and five universities published it on arXiv on 26 Oct 2026.

Which coding agent scored highest on Co-Design Bench?

OpenAI’s GPT-6 Astra scored highest on Co-Design Bench, passing 69.9% of criteria per task on average and fully finishing 26.4% of tasks. Opus 5 came second with 18.3% of tasks fully finished, and Qwen3.8-Max third with 16.5%. All ten models ran in the Claude Code harness.

Does 26.4% mean coding agents get most real tasks wrong?

A 26.4% full-pass rate does not mean coding agents get most real tasks wrong. The best model met 69.9% of criteria per task on average, so most unfinished tasks were mostly done, with one or more acceptance criteria unmet. The tasks were also picked for difficulty.

Why do coding agent benchmarks disagree?

Coding agent benchmarks disagree because they use different tasks, grading and harnesses. On WebCompass-edit, an earlier web editing test, Opus 4.8 reached 99.4% mean feature completion, while the best model on Co-Design Bench met 69.9% of criteria per task, and the paper notes that the two tests use different tasks and grading. The paper also shows the same model scoring very differently in different harnesses.

How do you check that an agent PR meets its acceptance criteria?

Checking an agent PR against its acceptance criteria starts with linking the ticket to the PR and writing the criteria down. Qodo reads the linked ticket from Jira, Linear, Azure DevOps, GitHub Issues, GitLab Issues or Monday.com and can flag missing or incomplete requirements as findings in the PR.

Does Co-Design Bench test code review?

Co-Design Bench does not test code review. The benchmark measures how completely coding agents meet hidden acceptance criteria. The link to code review is Qodo’s inference, since unmet criteria are code that never appears in a diff.