About 17 minutes into a robot-arm test, OpenAI’s GPT-6.1 Sol had built a tower that met the test’s passing criteria. Then it kept working. By the end, it had failed.
That episode, documented in Artificial Analysis’s AA-Robotics research preview published October 14, captures a problem that a leaderboard can hide. Getting the world into the right state and knowing to leave it there are different skills.
The coding agent gets a gripper
The experiment puts five language models in their coding-agent software, gives them camera views and lets them write commands for a real robot arm. Eight tabletop tasks include pouring a drink, putting blocks away and measuring a box. A human judge scores the final state, with partial credit for unfinished work. None of the five models earned full marks on all eight tasks.
Think of the loop as looking, planning, writing a movement command and looking again. The output is no longer just a file that can be edited. It changes the scene that the next decision must deal with.
The distinction from much robotics work matters. The NexArm hardware’s own documentation describes teaching motions, recording and replaying them, and learning from demonstrations. Here the researchers are asking what general coding agents can manage through a control interface, not whether a purpose-built robot controller can stack blocks.
A score is not a finished job
GPT-6 Astra led the published table with 69.4 points out of 100. That is a mean score with partial credit, not a 69.4% success rate. Count the full-score entries instead: Astra received 100 on four of the eight tasks. Its other results included 71 for the tower and zero for measuring the box.
Those two readings answer different questions. Partial credit shows useful progress. A person waiting for a task to be done needs to know whether the required result actually survives to handoff. The Sol tower separates those questions especially clearly: a passing intermediate state did not become a passing final one.
The clock is part of the result
Artificial Analysis reports roughly 30 minutes per task for Astra, against a 20-second human average. The person used their hands, not the same robot arm. It is a comparison of ways to do the job, not a clean measurement of human and machine thinking speed.
That leaves an important engineering question: how much work should a general model do itself, and how much should it hand to a faster, specialized control system? A fluent plan and a precise physical movement are not interchangeable. Our earlier examples of robots learning from failed demonstrations and navigation without generating a future video explore different ways to close that gap.
This is a small research preview, not a factory or household trial. The published method does not establish repeat-run reliability or compare the agents with specialized robot policies. Its value is the concrete behavior it makes visible.
A useful robot needs more than another good move. It needs to recognize a sufficient result, preserve it and stop. The tower was evidence of progress before the final score became evidence of failure.
