Embodied AI enters a new testing era with AA-Robotics v0.1

Embodied AI enters a new testing era with AA-Robotics v0.1

AA-Robotics v0.1 is a research preview designed to measure frontier language models on understanding, planning, and acting in the physical world. Each model controls a 6-axis robot arm with five joints and a gripper, then receives a simple objective. The initial release evaluates eight tabletop tasks.

What AA-Robotics v0.1 measures

Frontier language models have made substantial progress in controlling robot arms since the beginning of 2026. AA-Robotics v0.1 tests this progress through coding-agent harnesses that allow models to control a robot arm directly.

The results indicate that current models can manipulate an arm for simple tasks. However, they remain far too slow and expensive for production robotics. Their performance is nearly two orders of magnitude worse than human baselines in time, although this gap may change as the technology develops.

GPT-6 Astra ranked first with a score of 69, followed by GPT-6.1 Sol, Gemini 4 Argon, Claude Opus 5.5, and Claude Fable 5.1. GPT-6 Astra required an average of 30 minutes and cost $16.36 per task.

Each attempt receives a score from 0 to 100. A completed task earns 100 points, while partial results receive points based on the progress made.

The eight tasks

The benchmark initially includes eight tabletop activities covering control, dexterity, spatial reasoning, and adaptability:

1. Press the red button: Press the button with the gripper. The button must flash.
2. Place the crocodile toy in the rectangle: Put the crocodile entirely inside the rectangle.
3. Pour the drink into the cup: Transfer the drink from the bottle to the cup without spilling or breaking anything.
4. Tidy the blocks into the box: Place all four blocks in the box and replace the lid so it is flush.
5. Complete the shape matching challenge: Arrange the blocks on the board to match a picture.
6. Write hello world: Write “hello world” legibly inside the black box on the paper.
7. Build the tallest tower: Construct a freestanding tower with the maximum achievable height from the foam blocks.
8. Measure the box: Measure its length, width, and height, with each dimension within 2 mm of the correct size.

Results

The mean scores across all eight tasks were:

Results by task

GPT models ran in Codex, Claude models ran in Claude Code, and Gemini 4 Argon ran in Antigravity. All models used the high-effort setting.

How the benchmark works

Models run in isolated containers with brief instructions explaining how to access the robot through a simple HTTP API, identifying the arm as a Hiwonder NexArm, and describing how to submit an attempt as complete. No further arm specifications are provided.

Each model can view the scene through four cameras: three positioned around the table and one mounted on the arm’s wrist. Models can write and execute code in the container to plan movements and send commands.

After a model submits an attempt for review, a human judges the final state against the task’s pass criteria. A complete pass scores 100, while incomplete attempts receive partial credit. All attempts are zero-shot.

Time and cost

Models take approximately half an hour per task, compared with a human average of 20 seconds. This makes them roughly 100 times slower than a human on the benchmark.

Four of the five models averaged about 30 minutes per task. Gemini 4 Argon was the slowest, averaging 52 minutes. These averages conceal significant differences between tasks:

• GPT-6 Astra pressed the red button in 2.2 minutes.
• Claude Fable 5.1 spent about 10 minutes understanding and planning before its first action on that task.
• GPT-6.1 Sol declared the crocodile-placement task complete after five minutes.
• Claude Opus 5.5 took 45 minutes on the same task.
• Gemini 4 Argon spent an average of 21 minutes observing and reasoning before its first action, nearly twice as long as any other model.

The cost gap is smaller than the time gap. The most cost-effective model costs about 10 times as much as a person paid the average US hourly wage, compared with nearly 100 times as long in task duration.

GPT-6.1 Sol was the least expensive at $1.93 per task on average. GPT-6 Astra was the most expensive at $16.36, more than eight times the cost of Sol for a score only four points higher, 69 compared with 65.

Because the arm and scene are unfamiliar and the models process large volumes of visual information, most spending is attributable to input. Context accounts for 70% of total cost.

The human cost estimate uses a task time priced at $37.81 per hour, the average hourly earnings of all employees on US private nonfarm payrolls. The figure is based on preliminary September 2026 data from the US Bureau of Labor Statistics.

How models used their time

Models generally spent much of the beginning of each attempt calibrating the arm and interpreting the scene. After the first action, most of the remaining time went to movement planning, with short arm movements occurring between longer planning steps. Reviewing camera views and writing notes represented smaller portions of the total time.

Models also frequently continued refining an attempt after reaching a passing state. GPT-6.1 Sol built a passing tower after about 17 minutes, then continued and eventually failed the task. This illustrates the difficulty models have in judging when a physical task is complete.

The “Other” category includes waiting for code to run, writing notes, and producing the final report.

Gemini 4 Argon’s activity times are estimates because Antigravity does not timestamp every step. Steps are placed between precisely timed arm commands in proportion to their token counts.

In one pour task, the first action occurred at the following times:

• GPT-6 Astra: 2:23
• GPT-6.1 Sol: 2:39
• Claude Opus 5.5: 4:14
• Claude Fable 5.1: 7:51
• Gemini 4 Argon: 25:36

A first action is the first press, grasp, push, pour, or pen stroke. An irrecoverable mistake means the model can no longer achieve full marks.

Physical accuracy

Grasping and fine manipulation are central to AA-Robotics v0.1 because they test precise control and the ability to use objects as tools.

GPT-6 Astra and Claude Fable 5.1 recorded the highest proportions of successful grasps, at approximately 45%. GPT-6.1 Sol had the lowest rate at 28%. Gemini 4 Argon was between them at 36%; many of its failed grasps closed just above or beside the intended block.

Grasps were attempted in six tasks: Place the crocodile, Pour the drink, Tidy the blocks, Match shapes, Build the tower, and Measure the box.

A successful grasp means the model held the intended object through lifting or carrying. The other category includes grippers that failed to close on the object, closed before it slipped, dropped the object, could not lift it, or picked up the wrong object.

Common failure patterns

The observed failures fell into several broad groups.

Misjudging when a task was complete

Models sometimes incorrectly reported success or continued acting after reaching a passing state. Examples included:

• Continuing to grasp a bottle after it had fallen and spilled.
• Reporting that a board matched its picture while a red hexagon remained incorrectly seated.
• Reporting that a lid was level and aligned when it rested on a wedge stacked on a cube.
• Counting a tilted yellow hexagon as correctly placed while acknowledging that six of eight pieces were positioned.
• Continuing to adjust a crocodile that was already inside the rectangle, eventually pushing it outside the far edge three times.
• Rewriting “hello world” repeatedly after an initial illegible attempt.
• Reporting an off-center lid as flush.

Poor physical reasoning

Other failures resulted from incorrect predictions about contact, force, or object movement:

• Testing pen contact on the page and leaving marks inside the writing box.
• Pushing a box toward the table corner while trying to drag it back.
• Sliding the gripper into the crocodile and pushing it about 10 cm out of reach.
• Searching for paper by making ink marks inside the writing box.
• Pushing a crocodile that was already correctly placed back over the rectangle’s far edge.
• Pressing the wrist against the tray and moving it forward, making the bottle harder to grasp.
• Using table contact to push four loose blocks into one cluster.
• Handling blocks for 13 minutes before removing the box lid.
• Releasing a wedge from well above the rim, causing it to land on a cube.
• Allowing the pour stream to move the cup about 4 cm to the left.
• Dragging the crocodile by its tail and moving it over two edges.
• Lowering the lid onto two upright blocks and pressing down.
• Using the lid to push the box to the limit of the arm’s reach.
• Approaching an upright green triangle from an angle that prevented it from being lifted out of its opening.
• Sweeping the top two blocks from a four-block tower while reaching for a nearby cone.
• Closing on the hook end of a tape measure three times, after which the tape retracted into its case.
• Failing to account for the pour stream moving closer to the bottle as the flow slowed.
• Lifting a leaning block by its corner without anticipating that the tower would fall.

Missed grasps, misplaced objects, and drops

The benchmark also recorded many direct manipulation errors:

• Rolling the wrist incorrectly and placing the crocodile across the rectangle with both ends over the edges.
• Sweeping an unplaced yellow hexagon to the arm’s base.
• Carrying a hexagon between the fingers over the cutout, then dropping it as the arm moved away.
• Allowing the bottle to slip from a high grip on its shoulder.
• Lifting the bottle by the fingertips until it slid out and spilled.
• Knocking a red hexagon off the table with an empty grasp.
• Squeezing a red hexagon out of the gripper, moving it 5 to 8 cm farther away.
• Tipping a blue square onto its edge during a close.
• Making seven unsuccessful attempts to lift an upright triangle from the wrong cutout.
• Releasing a yellow triangle and tipping a blue square out of its cutout.
• Dragging a blue circle onto the board’s rim with an empty descent, causing it to fall.
• Pushing a blue square off the front of the board to the arm’s base.
• Making five missed grasps before the first successful hold.
• Dropping a blue circle during two carries and leaving it in the hexagon cutout.
• Closing on the rim of the box while reaching for a block and lifting the entire box.
• Knocking over a standing two-block stack while attempting to move it onto a wider base.
• Perching a cylinder on a tower from a level gripper and allowing it to fall after release.
• Releasing a triangular prism above and off-center from a cube, causing it to fall to the table.
• Repeatedly sliding a cube farther away because the fingertips landed behind it.
• Holding a long block by one corner until it slid off the cube.
• Making three consecutive closes beside or above a blue block.
• Dropping a corner-caught cube back onto the table.
• Releasing a prism several centimeters above its intended position.
• Placing a long block next to, rather than beneath, a cube.
• Making four empty closes in 80 seconds while the cube remained beside the fingers.
• Leaving a red circle tilted beside its hole after an off-target reach.
• Carrying an empty gripper after closing beside a yellow triangle.
• Releasing a blue square beside the board after a wrist correction left it tilted.
• Mistaking a round block against its base for a yellow hexagon.
• Closing well above a red cone the model believed it had reached.
• Closing about 1.3 cm above a red circle and grasping nothing.
• Releasing a cube about 2 cm too far out, causing it to fall from a cylinder.
• Allowing a hexagon held by a tilted edge to slip out during movement and leave the table.

AA-Robotics v0.1 therefore shows that language models can perform simple physical actions, but reliable control still depends on faster planning, more accurate perception, better grasping, and a stronger understanding of when a task is complete.