ToyBlocks: A Controlled World for Robot Learning and Evaluation

ToyBlocks: A Controlled World for Robot Learning and Evaluation

Robots are learning to fold laundry, sort parts, and handle unfamiliar objects. However, demonstrations of these abilities are often narrow, making it difficult to determine what a robot has learned or why it fails.

Scientists studying complex systems often begin with small, controlled environments, such as petri dishes or terrariums. These settings allow researchers to vary conditions, repeat tests, and trace the causes of observed behavior. Robotics research benefits from similar environments, where teams can change training data or model designs and evaluate the results systematically.

A framework called FoundationWorlds is designed for this purpose. Each FoundationWorld combines a controlled simulation for a particular robotics domain with tools for automatically generating tasks and training demonstrations, along with metrics for measuring robot behavior over time. These environments support research questions such as how models improve with additional training data and how high-level reasoning can be combined with precise physical control.

The first FoundationWorld is ToyBlocks, an open simulation framework built around a robot arm and blocks on a tabletop. Although the setting is deliberately simple, the robot must reason about block interactions while pushing, flipping, and stacking objects.

ToyBlocks automatically generates demonstrations, which are recordings of a robot completing tasks in simulation. This avoids the time-consuming teleoperation normally required to collect many real-world demonstrations. Researchers can vary the generated experiences, for example by producing more pushing or flipping examples and then evaluating the resulting model on tasks excluded from training.

How ToyBlocks Works

ToyBlocks builds on earlier open robot-learning work, including MolmoAct 2, MolmoBot, and MolmoSpaces. As a FoundationWorld, it is designed around four requirements:

1. Realistic physics and appearance
2. Automatic generation of tasks and demonstrations at scale
3. Challenges that combine reasoning with dexterity
4. Broad coverage of the tasks possible within the domain

Open-ended environments require models of many objects and interactions, which makes it difficult to satisfy all four requirements. ToyBlocks has a deliberately narrow scope, allowing researchers to address them within one setting.

The environment is intended to help researchers study how a robot can learn a broad range of reasonable tasks while maintaining confidence that the simulated arm and blocks behave similarly to their physical counterparts.

ToyBlocks is not limited to picking up and stacking blocks. Pictures and letters on the block faces support searching, sorting, and arranging tasks. A robot may need to rotate a block to reveal a hidden face before deciding where it belongs. Domain randomization changes properties such as block size and table height between scenes, allowing researchers to test how well models generalize to changes in geometry.

The framework automatically generates skills, which are behaviors for short tasks, and task demonstrations that can be used to train models or fine-tune previously trained models. High-level reasoning models determine what should happen, while low-level action models control the robot's movements. ToyBlocks also provides tools for designing experiments and testing hypotheses.

ToyBlocks uses reinforcement learning (RL) and motion planning to generate skills for short-horizon tasks. Skill training takes place in Isaac Lab, Nvidia's open-source framework for robot learning in simulation. Through trial and error, policies learn non-prehensile skills such as pushing a block across a table or flipping it onto another face without picking it up.

Skill generation can use privileged information, such as a block's exact position and orientation, that a simulator can provide directly but a physical robot would need to estimate from camera images. This makes it possible to automatically generate skills that would otherwise require real-world teleoperation.

To create demonstration data, a task-and-motion planner (TAMP) selects and sequences the skills required for a task and plans the robot's movements between them. Executing each plan in simulation produces recorded demonstrations and labels describing the task or skill being performed. ToyBlocks can generate and execute hundreds of plans in parallel on a single graphics processing unit (GPU), reaching up to 24 times real-time performance.

When a robot acts from sensor observations, perception becomes part of the problem. The model must estimate a block's pose, including its position and orientation, instead of receiving that information directly from the simulator. ToyBlocks therefore trains models that map raw sensor observations to actions.

A representative system uses Molmo 2 as a high-level model and ManiFlow as a low-level model. Molmo 2 selects skills such as rotation and flipping and identifies the relevant block through a pointing interface. ManiFlow then controls the simulated arm to execute the selected skill.

ToyBlocks' controllable data generator allows users to create training mixtures that emphasize particular skills or fill gaps in existing datasets. Researchers can investigate how much data a model needs and which experiences help it generalize to unseen tasks. For example, they can test whether adding rotation and pick-and-place demonstrations improves flipping, or vary the balance of examples inside and outside a defined workspace region.

Evaluation provides more than success rates. It can also record execution speeds, forces exerted on objects, and task progress, using the same specifications as the planning module.

A web interface can connect Claude or another frontier model to ToyBlocks. Users can issue natural-language instructions or try tasks, providing a way to study planning and action without requiring specialized robotics expertise.

Research Using ToyBlocks

ToyBlocks has been used to study how training data and methods affect low-level performance, spatial reasoning, generalization to new letters and block appearances, multi-step planning, transfer to physical robots, and frontier models that act through tools. The framework also allows researchers to define their own tasks and metrics.

Low-Level Action Models and Spatial Generalization

Low-level action models, including vision-language-action models (VLAs), map language instructions and visual observations to robot actions.

VLAs are expected to generalize rapidly to different scenarios or workspaces after pretraining on large-scale datasets. Evidence about this ability remains mixed. Some studies report that these models are brittle when object layouts or lighting change, while others find that they generalize well. ToyBlocks provides a simple and reproducible environment for testing this question across meaningful variations in task space.

The generalization tests vary the skill sequences required by an instruction, the letters or colors on the blocks, and the number of blocks in a scene. A stacking experiment can withhold four-block stacks during training and test whether a model can combine familiar movements to build them.

Workspace experiments distinguish interpolation, which involves new positions within the training range, from extrapolation, which involves positions beyond that range. For example, a model can be trained with objects appearing only in one region and tested in another, or trained on stack heights of two and five blocks and tested on heights of three and four.

Initial spatial-generalization results based on ManiFlow, π0.5, and MolmoAct 2 showed poor generalization despite large-scale pretraining. Additional techniques improved performance, and the experiments examined each technique's contribution. The results included the use of LoRA, or low-rank adaptation, which fine-tunes a model with a small set of trainable weight updates. These findings indicate that action-model generalization remains an open research problem, and ToyBlocks provides a controlled testbed for studying possible solutions.

High-Level Reasoning with Vision-Language Models

Long-horizon tasks require a model to plan and execute many actions. ToyBlocks supports these tasks and can provide oracle components, which are reference components that supply correct plans or actions at either the high or low level. This makes it possible to identify whether a failure came from high-level planning or low-level execution.

One example is spelling “Ai2” with a stack of blocks. Because the letters may not be visible initially, the agent must first choose actions that reveal information and then plan the sequence needed to build the stack.

In one hierarchical system, Molmo 2 serves as the high-level planner and ManiFlow serves as the low-level policy. Molmo 2 chooses the skill and the block on which to perform it, while ManiFlow executes the instruction and pauses before querying Molmo 2 for the next one.

The planning evaluation used three scenarios of increasing difficulty:

• In the easiest scenario, every character begins upright and facing the camera.
• In the medium scenario, characters may not be upright.
• In the hardest scenario, some characters are not visible at the start of the episode.

Success rates were measured on three-letter block-spelling tasks using Molmo 2 as the high-level reasoning model and either ManiFlow or a motion planner for low-level manipulation. The systems were trained on ToyBlocks and evaluated across the three difficulty levels and two generalization conditions.

High-Level Reasoning with Agentic Systems

Other experiments examined how frontier models plan and act through tool calls. One setting used Claude Opus 5.5 with ToyBlocks skills exposed through the Model Context Protocol (MCP). MCP is a standard for connecting AI applications to tools.

The agent observes the scene through camera images, calls skills such as pick, flip, or rotate by pointing to a pixel, and decides when to search, construct a result, or declare that a task is complete or impossible. The same MCP tools can execute either scripted skills used to generate ToyBlocks demonstrations, including motion-planned pick and place and RL-based rotation and flipping, or learned ManiFlow skills.

In a classical-planning experiment, the system using Claude Opus 5.5 solved all 30 block-stacking tasks in a group similar to tasks that frontier models had struggled with several years earlier. It achieved the same result while repeatedly observing camera images and choosing its next action in a closed loop.

In the search-and-spell setting, the agent had to construct stacks spelling three-letter words. It searched among as many as eight blocks, and 50% of scenes did not contain all the required letters. In those cases, the agent had to declare the task infeasible.

The agent successfully solved or correctly declared infeasible 63% of tasks. Most errors came from confusing letters with look-alike letters, which accounted for 13% of errors, or selecting an incorrect orientation for a letter in a stack, which accounted for 15%.

The agent performed more efficiently when blocks had semantic regularities, or patterns. These patterns helped it reason about occluded faces. In many infeasible scenes with unpatterned blocks, the agent examined every face before declaring that the task was impossible. With semantic patterns, it could reach that conclusion significantly faster.

A third experiment examined whether an agent could learn how its skills behave. When a skill produced an unexpected result, the agent formed a hypothesis, designed a small test, and revised its understanding based on the outcome. The agent also generated new skill descriptions from its experiments. These descriptions raised success rates from 24% with the default descriptions to 63%, exceeding the 36% achieved with descriptions written by a human expert.

Action-Conditioned World Models

Action-conditioned world models predict how a scene changes in response to a robot's actions. They may eventually serve as alternatives to model-based simulators for training and evaluating robot policies, but their usefulness depends on how accurately they predict physical interactions.

ToyBlocks provides the ground-truth simulated state, enabling controlled tests of world models in low-level object manipulation. Researchers tested whether these models could predict block motions or skill outcomes after training on example executions.

Four skills were evaluated: pick-and-place, flip, rotate, and a mixed task. For each skill, the researchers generated 900 trajectories, which are sequences of robot states and actions over time. Half were successful demonstrations, while the other half were unsuccessful because of small changes to spatial configurations. Since successful and unsuccessful trajectories were equally represented, a model that always predicted success would be correct only half the time.

The models were fine-tuned on these demonstrations and tested on their ability to predict the outcomes of action sequences for the same skills.

Three types of models were evaluated:

• Wan 2.2, a video prediction model with 5 billion parameters
• Astra, an agentic model that created its own physics simulator and tuned it to match the demonstration data
• LeWM, a latent world model

During testing, each model predicted future images, simulator states, or latent states based on an initial image and a sequence of actions. For full trajectories, the initial image was the only observed image. For short interaction segments, such as a single pick, place, flip, or rotate, the model began with the real image immediately before the interaction.

Full trajectories showed how errors accumulated over long horizons. Short segments isolated individual interactions, such as the gripper making contact with a block. The researchers compared the models by examining where each predicted that the blocks and gripper would be at different points and whether it predicted success at the end.

Wan 2.2 predicted block motion and task outcomes most accurately. Astra tracked the gripper well but struggled to simulate contact. LeWM was faster and more accurate over short interaction segments, although it drifted over full trajectories.

Additional experiments found that models with higher pose and success-prediction scores also produced better training data. Adding 100 model-generated trajectories from the best video-generation model increased a policy's success in an unseen region from 19% to 78%.

Real-World Experiments

Sim-to-real evaluations test policies trained in simulation on physical robots. Comparing simulated and real-world execution helps determine whether behaviors and failure patterns observed in simulation provide useful indications of physical performance.

ToyBlocks models were tested on physical robots without training on real-world examples. The evaluation covered ManiFlow and the individual low-level skills of pick, place, flip, and rotate. It also tested whether the hierarchical Molmo 2 and ManiFlow system could complete long-horizon, three-block spelling tasks.

The physical experiments used a Franka FR3 arm with a Robotiq gripper across 30 scene configurations. Each scene contained four to ten blocks seen during training. An episode counted as successful only when the relevant blocks were moved to their target poses without collisions. If an operator stopped the robot because of unsafe motion, the episode was marked as a failure.

ManiFlow's atomic-skill success rates were similar in simulation and the real world: 80.0% and 81.1%, respectively. The hierarchical Molmo 2 and ManiFlow system succeeded in 55.5% of simulated episodes, or 111 out of 200, and 50.0% of real-world episodes, or 15 out of 30. Further trials are needed to establish statistical confidence across tasks.

To transfer models trained exclusively on ToyBlocks data to a physical robot, the researchers addressed simulation-to-reality differences in three stages.

Before data generation, they performed system identification by tuning simulated robot parameters, including joint friction and armature, the motor and gearbox inertia reflected at the joint, to match real-robot trajectories.

During data generation, they randomized physical properties such as object size, friction, and table height, as well as visual properties including lighting, backgrounds, materials, and camera parameters.

At inference time, observations were captured with a front-facing ZED camera and a wrist-mounted ZED Mini. For policies using depth images, clean depth maps were generated from paired red-green-blue (RGB) camera images with FastFoundationStereo, rather than using noisy depth-sensor measurements.

Tools for Further Research

ToyBlocks connects data generation, training, and evaluation in one controlled environment. This allows researchers to examine how robots learn over time and how changes to training data or model design affect behavior.

The released resources include simulation assets, procedural task and demonstration generators, automatically generated demonstrations, two trained models, training code, and evaluation code with validation sets. The models are based on Molmo 2 and ManiFlow. These resources allow teams to create their own training mixtures and investigate failure modes beyond the initial experiments.

The initial data release contains 85,336 long-horizon demonstrations, each comprising approximately six to twelve skill demonstrations, depending on the dataset. The datasets cover nearly 1,600 hours of robot execution data and 144 million frames.

A subsequent release is planned to include a larger-scale dataset, additional skills, and more comprehensive benchmarking. Fully integrated RL training capability is also planned but is not included in the current package.

Future directions include bimanual manipulation with two arms and dexterous robot hands equipped with force and touch sensors. These platforms could support research into motions and tasks involving frequent physical contact. The project is also exploring interaction with unstructured piles of blocks as a way to collect data and study how new behaviors and tasks emerge through interaction.

ToyBlocks is intended to support research into how robots can reliably apply learned skills to unfamiliar tasks and environments, while providing a controlled basis for developing additional FoundationWorlds.