A new benchmark for AI coding agents has a number I keep coming back to. The best model met about 70 percent of the acceptance criteria on average. It fully delivered 26.4 percent of the tasks.
Those two numbers describe the same runs. The difference between them is where I think most of the real work in AI-assisted development now sits.
What the benchmark measured
Co-Design Bench was published by researchers from Fudan University, Alibaba and several other universities. It has 273 tasks taken from real coding sessions: change requests against existing web apps, from small tweaks to larger overhauls, some with images, audio, video or documents as input. The authors deliberately picked sessions where users had to rephrase a request or roll back a change, which is to say the kind of work that is hard in practice.
Grading is strict. An AI judge tests the finished app in a browser against hidden acceptance criteria, and a second judge rechecks every pass, looking for mock-ups and hard-coded results. A task only counts as solved when every criterion works in the running app.
The headline: across ten models, the best strict completion was 69.9 percent of criteria, but the best full-solve rate was 26.4 percent. On older editing benchmarks such as WebCompass-edit, models score above 99 percent. Every model did worse when a change combined several kinds of work, such as layout, interaction and content at once.
"Mostly done" is a different state from "done"
The project page sums it up in one line: criterion coverage is not complete delivery. Anyone who has run a delivery team knows this without a benchmark. A change that does seven of ten things the user asked for is not 70 percent of value. Often it is a review, a conversation and a second round of work, because the three missing things are the ones that made the change worth doing.
What the benchmark adds is a measurement of how big that gap is when an agent does the work, and a reminder that the gap does not show up if you only look at averages. "It meets 70 percent of requirements" sounds like a productivity story. "It finishes one task in four" sounds like a planning problem. Both are true.
The harness matters as much as the model
The finding I found most useful as an architect is not about which model won. In a separate test, the same open model scored 75 or 56 percent on larger overhauls depending only on the harness, meaning the environment where the agent reads, writes and runs code. Which of two models ranked higher also depended on the harness. The authors recommend reporting model and harness together.
That matches how I think about it. When a team says "we tried model X and it didn't work for us", the question I want to ask is: in what setup? With what tests to run against? With what instructions and context? A model comparison that ignores the environment is comparing two things and naming one of them.
What I take from it for my own work
None of this makes me less enthusiastic about agents. It changes where I put my attention.
- Write down what "done" means before the agent starts. The benchmark only works because every task has explicit acceptance criteria. If my own definition of done lives in my head, I cannot tell a 70 percent result from a 100 percent one without reading every line.
- Make the checks able to fail. The second judge looks for hard-coded results and mock-ups that make a test pass without the feature existing. That is the same question I ask in code review: could this test have come out differently if the code were wrong?
- Keep changes narrow. Every model got worse as a change mixed more kinds of work. Splitting a change into smaller, single-purpose steps is not just good practice for humans; it is where agents are measurably stronger.
- Review the plan, not just the diff. A Google repository has open-sourced a plugin for coding assistants where one agent drafts a specification with the full context and fresh-context agents review it without the earlier conversation. I like the idea because unstated assumptions are cheaper to catch in a spec than in a pull request. The project itself says the reviews do not replace tests or human judgement (and that it is not an officially supported Google product), and I agree with that framing.
Caveats worth keeping
The benchmark has limits, and the authors name them. Each task is a single fixed request with no follow-up questions or mid-task corrections, which is not how I work with an agent. The judges are AI models and can be wrong. And web-app changes are one kind of work, not all of software development.
So I would not use 26.4 percent as a forecast for any particular team. I would use it as a prompt: if you measure your agent-assisted work only by how much gets produced, you are measuring the 70 percent. The value is in the part that gets finished.
Sources
- Co-Design Bench project page: https://codesignbench.github.io
- Wei et al., Co-Design Bench paper (arXiv 2610.35138): https://arxiv.org/abs/2610.35138
- The Decoder, "AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test" (8 November 2026): https://the-decoder.com/ai-coding-agents-still-need-human-cleanup-on-nearly-three-out-of-four-tasks-in-a-real-world-test/
- Google, Elephant-Goldfish Model (GitHub): https://github.com/google/elephant-goldfish-model