The Same Coding Model Produced a Threefold Cost Gap

A coding assistant can receive the same model, the same software bug and a very different bill. The difference may lie in everything the tool sends along with the request: instructions, descriptions of available tools and the growing record of what the agent has already tried.

In an October 3 research preprint, Yangze Liu and Zhongyi Han compared three ways of putting models to work inside software projects. For one model, the estimated average token cost of attempting the same 45 difficult tasks ranged from 0.56 to 1.81 Chinese yuan. The more expensive setup passed more tasks in that run, but the small test could not establish a reliable performance advantage.

That distinction matters. A tool can produce a clearly larger bill before an evaluation can show that it produces better work.

The Software Around the Model Has Its Own Cost

The researchers tested Claude Code, mini-SWE-agent and OpenCode. These are not three interchangeable names for AI models. They are software environments that give a model instructions, let it inspect or edit files, run commands, and decide what information to carry forward.

Researchers call this surrounding machinery a harness. Think of the model as the component proposing the next action, and the harness as the working environment that makes that action possible. Changing either can change the result.

The study used SWE-bench Verified, a collection of real software issues with tests that check whether a proposed fix works. Eight tasks requiring public web services were excluded. The remaining 492 were split into 45 difficult tasks, tested across five models, and 447 other tasks, tested only on two locally run Qwen models.

Those are different comparisons, not one giant test of every model under every tool. The researchers also repeated selected configurations to see how much a result could move without changing the setup at all.

A Threefold Bill Is Not Threefold Value

For GLM-5.3-Flash on the 45 difficult tasks, the released data give these approximate costs per attempt:

• Claude Code: 1.81 yuan, with 22 tasks passed.
• mini-SWE-agent: 0.97 yuan, with 20 tasks passed.
• OpenCode: 0.56 yuan, with 18 tasks passed.

The highest estimate is about 3.2 times the lowest. These are the paper's historical token-price assumptions applied to recorded usage, not today's prices, subscription fees or invoices paid by Vastkind. Each average includes failed attempts. It is not a cost per successful fix.

The score differences are also real observations, but they do not establish a dependable ranking. On this small task set, the paired comparisons could not distinguish the apparent harness advantages from no difference. A four-task lead is not proof of no benefit; it is insufficient evidence to pin down the benefit confidently.

The larger test offers a more useful constraint. On the two Qwen models tested across the 447-task pool, Claude Code and mini-SWE-agent met the study's equivalence criterion: their difference lay within a margin of five percentage points. That is a bounded result for those models and tasks, not a finding that all coding tools perform identically.

Repeated Context Can Become the Expense

Language models process text in units called tokens. A coding agent does not usually make one request and receive a finished fix. It may ask to inspect a file, receive the result, edit something, run a test and try again.

Each new request can bring along instructions and earlier context. The tool therefore affects both how much text the model processes per step and how many steps it takes.

The authors attribute much of the cost difference to that repeated context, rather than simply to different amounts of source code returned by individual tools. Their breakdown of hidden instructions and retained reasoning is a reconstruction from recorded behavior, not a direct inspection of every internal system prompt. The released trial table supports the cost totals; it does not make every proposed mechanism an independently reproduced fact.

There is a revealing counterexample in the same study. With DeepSeek-V4-Flash, the lightweight mini-SWE-agent took more steps and cost slightly more per attempt than Claude Code: about 1.02 versus 0.98 yuan under the paper's assumptions. A smaller starting prompt did not guarantee a smaller final bill.

Caching complicates the picture further. Providers may charge less when an input repeats earlier text, so a million cached input tokens need not cost the same as a million new ones. A useful comparison needs the types of tokens and their prices, not just an impressive total.

The cost question is therefore not simply whether a tool is elaborate. It is what the model keeps reading, how long the conversation grows and when the tool stops.

The Wrapper Still Matters When It Fails

The study did find a clearer performance deficit. On the larger pool with Qwen3.8-27B, OpenCode trailed Claude Code and mini-SWE-agent by roughly eight to nine percentage points; those differences survived the paper's correction for multiple comparisons.

The authors linked part of the gap to runs that stopped at an output limit before a useful patch was finished. Other observed failure modes included overflowing the model's context window and tools that could not initialize offline.

That is a concrete reason not to read the paper as an argument that software around a model is pointless. Recovering from an interrupted response, keeping necessary context and making a search tool available can prevent a capable model from failing to finish.

The comparison was not perfectly symmetrical either. Every trial had a budget of 300 model calls, but some locally run Qwen configurations had substantially different time limits: up to ten hours for Claude Code versus 96 minutes for OpenCode. The paper examines those failures, but a whole-tool comparison still bundles many design choices together.

Nor does the result rank today's products. The researchers tested pinned software versions. Their closed frontier-model reference ran in Claude Code only, so it cannot establish how that model would perform in the other environments.

A Better Benchmark Starts With a Repeat

The unusually useful part of this work is the control experiment: run the same configuration again. On the difficult tasks, the median share of outcomes that flipped was about 13% both when changing the harness and when repeating the same one.

If a tool's score moves without a software change, a small improvement after a change needs more evidence than a single leaderboard number. Cost, success rate and repeatability answer different questions. So does the underlying infrastructure: Arm's agent-work replay experiment isolates CPU behavior by replaying work rather than asking the model to make fresh decisions.

This study does not identify a universally cheapest or smartest coding assistant. It shows why choosing one is not just choosing a model. The surrounding software can charge for more activity, prevent avoidable failures, or do both. The bill tells you how much work was processed. It takes a separate, repeatable test to show how much useful work was gained.

Sources and Calculation

Paper and methods, October 3, 2026 · Released trial data and analysis code. Vastkind recalculated the quoted costs from the released trial table and the paper's stated prices; it did not rerun the coding experiments. Values are rounded, and the approximately 3.2-fold comparison uses unrounded means. The study is a preprint, not an independently replicated result across all coding products.