Arm Replays an AI Agent’s Work to Test Its CPU

One of Arm’s new CPU tests runs through 259 software-engineering tasks without asking the AI agent to invent a fresh route through each one. It replays the work. The score is tasks per minute, not how many new problems the model can solve.

That distinction is the useful idea inside Arm’s October 13, 2026 announcement. The company wants to measure the processor work behind AI agents: compiling code, running tools and coordinating services. Its supporting methodology and an Arm-commissioned Signal65 report were already public on October 12. They address a practical problem: a faster agent can reflect a better model, better infrastructure, or simply a different sequence of decisions.

Replay helps separate those possibilities by holding the work still.

Freeze the commands, then compare the machines

A coding agent does more than generate text. It can read files, invoke a compiler, launch tests and inspect errors before choosing its next action. The model chooses; the surrounding computer executes. Both consume time, but they are different jobs.

Run the same assignment twice and the agent might choose different commands. A machine that finishes sooner may have executed less work, not executed the same work faster. That makes a simple stopwatch a poor way to isolate the CPU.

Arm’s methodology proposes recording the tool calls and command inputs from an agent’s path through a task, then replaying that fixed sequence. The model does not have to regenerate the strategy on every run. The method also calls for separating CPU execution from time spent waiting for external services.

The result is a more controlled question: how quickly can this system carry out these instructions?

Signal65’s test table identifies its 259-task SWE-bench workload specifically as a replay and reports throughput in tasks per minute. That label matters. It is not interchangeable with the familiar benchmark score for an AI model fixing previously unseen software issues. The report also includes a vLLM test that measures CPU-side serving overhead while bypassing the model’s neural-network computation. Neither number should be read as the speed of a complete conversation with an AI.

Replay does not make the original choices representative of every agent. A trace captures one route through a problem. Choosing enough representative routes remains part of the measurement problem, and Arm presents its methodology as a framework for discussion, not a finished universal benchmark.

The chip result and the rack result are different claims

On the SWE-bench replay, Signal65 reports 1.13 times the throughput per chip for Arm’s tested system relative to AMD’s EPYC 9755 baseline. Its rack-level figure is 2.00 times. The extra gain does not mean the software suddenly runs twice as fast on each processor: the rack model fits more servers within a power budget.

Across seven workloads, the report gives arithmetic averages of 1.20 times per chip and 1.99 times per rack. The latter uses a 24-kilowatt rack model and rated server power, not a measured comparison of complete racks at the wall socket.

Memory allocation changes that model substantially. The baseline sizes memory per hardware thread, giving the AMD servers about twice the memory of the Arm servers. Signal65 says sizing it per physical core instead would reduce the seven-workload rack average from 1.99 to about 1.81 times. As with Microsoft’s Maia rack-throughput claim, the system boundary and configuration help explain the multiplier.

Arm commissioned the report from Russ Fellows, Signal65’s VP of Labs, and ran the tests at its own facilities. Signal65 watched remotely and analyzed the original logs, configurations and scripts. That is external scrutiny of vendor-run testing, not an unrelated lab independently operating both systems.

Faster execution still has to meet the real task

Infrastructure can genuinely change whether an agent succeeds. In a separate February 5, 2026 experiment, Anthropic kept the model, test harness and tasks fixed while varying resource configurations. The gap between the most constrained and least constrained Terminal-Bench 2.0 setups was six percentage points. That result is not a validation of Arm’s hardware, but it shows why the computer around a model belongs in the evaluation.

The two kinds of test answer different questions. Replay helps attribute time to the machinery. A full task test asks whether the agent, with its actual decisions, tools and resource limits, gets the job done. Confusing them changes what a benchmark result means.

A faster CPU can shorten the work an agent has already chosen to do. Whether it chose the right work remains a separate test.