Artificial Analysis’s New AI Tests Show a Wider Open-Model Gap

A coding assistant fixes the bug it was asked to fix. The affected operation restarts correctly, but an old operation is abandoned without its cleanup code running. The requested repair works; the surrounding software can still be left in a bad state.

That is one of the failures Cognition documents in its new coding test. It helps explain why Artificial Analysis has changed its AI ranking: answering a well-specified question is not the same as completing work with consequences the user never spelled out.

The evaluator’s October 27 update to its Intelligence Index produces a wider gap between leading proprietary models and the models it groups as open weights. But the test itself has changed. The larger gap is not evidence, by itself, that models suddenly lost capability.

The Questions Changed, Not Just the Leaderboard

The new index, version 5, combines nine evaluations. Its published methodology assigns equal shares to four areas: knowledge work, coding, science and general capabilities. Within those areas, different tests carry different weights.

New tasks include scientific research workflows performed through a computer terminal and coding changes judged against the standards of a software maintainer. Several other tests have been revised or replaced. This is a different selection of work, not simply another round of the old exam.

Consider the coding example. In Cognition’s account of FrontierCode 2.0, Claude Opus 5.5 at maximum effort repairs a restart problem in Mock Service Worker, a JavaScript library used to simulate network requests. Its solution drops the old operation without closing it properly. Cleanup that should run afterward never happens, potentially leaving resources tied up or the application in an inconsistent state.

The task did not explicitly demand that cleanup. The evaluator treats it as a responsibility a competent contributor should infer. Cognition reports that all five attempts shown for this task failed. That is one documented example, not a failure rate for all coding work. Cognition also sells the Devin coding agent; its benchmark is a commercially interested test design, not an outside audit of the entire Artificial Analysis ranking.

The distinction matters beyond programming. “Did it do what I wrote?” and “Did it finish the job responsibly?” are different questions. A test that starts asking the second can expose weaknesses an earlier score concealed.

“Open Weights” Includes an Important Catch

Artificial Analysis reports a 12-point gap between the new index leader and Kimi K3.1 Pro, compared with three points on version 4.3. These are index points, not percentages of intelligence.

There is also an availability problem: Kimi K3.1 Pro’s weights had not been released at the time of the announcement. Weights are the learned numerical parameters needed to run a model independently. Access to a hosted service is not the same thing.

Among models with released weights, the announcement puts DeepSeek V4.1 Flash 1017 and GLM-5.3 at 32, against Claude Opus 5.5 at 58, all at their stated maximum-effort settings. That is a 26-point gap within this version of the index. Neither comparison establishes a universal gap across every language, tool or task.

A Better Test Still Needs Boundaries

Artificial Analysis says private test sets or held-out answers now account for 60% of the index’s weight. Keeping evaluation material out of public circulation can reduce obvious opportunities to train specifically for the test. It also makes full outside reproduction harder. Privacy is not proof that contamination is impossible.

The methodology is primarily text-based and English-language. Results also depend on the tools, surrounding agent software and reasoning effort used. Our explanation of coding-agent test setups shows why the model name alone does not describe the whole system being measured.

A changed test is not a reason to dismiss an uncomfortable result. It can reveal a real weakness. But comparing the new score with an old score as though the measuring instrument stayed fixed creates a different mistake.

The useful question is which requirements the new test adds. In the coding example, the missing requirement was not another clever answer. It was cleaning up after the work was done.