Named SceneSmith, the platform uses three collaborative AI agents to build detailed virtual spaces, including kitchens, hotels, and living rooms, by arranging objects, walls, and room layouts.

These digital environments allow robots to safely practice everyday tasks, test different strategies, and improve their skills before operating in the real world,

According to the team, the new model helps in reducing the need for costly physical training while accelerating the development of more capable and adaptable robotic systems.

Virtual robot training

MIT’s SceneSmith is an AI system that creates realistic 3D indoor environments where robots can safely learn and practice everyday tasks before operating in the real world.

One of the biggest challenges in robotics is collecting enough training data. Unlike AI chatbots that learn from text, robots need experience interacting with physical objects in many different environments. Teaching robots in the real world is expensive, slow, and requires countless hours of human supervision.

SceneSmith addresses this problem by automatically building detailed virtual environments that resemble real places such as kitchens, bedrooms, hotels, restaurants, offices, and garages. Robots can train inside these digital worlds, testing different ways to complete tasks without the cost or risk of physical experiments, according to a statement by the team.

The system relies on three AI agents working together. The first, called the designer, creates the room layout and places furniture and other objects. The second, the critic, checks whether the scene looks realistic and suggests improvements, such as removing objects that do not belong in a particular room. The third, the orchestrator, manages the entire process, deciding when the design is complete or when more revisions are needed.

These AI agents use OpenAI’s GPT-5.2 vision-language model, which has learned from vast collections of text and images. This gives the system an understanding of how real indoor spaces are arranged and how everyday objects are typically placed.

Users can simply describe a room using natural language. For example, asking for “a garage with a car, workbench, stacked tires, and a ladder against the wall” allows SceneSmith to generate a detailed virtual version of that space. Compared with previous systems, its scenes contain up to six times more interactive objects, giving robots many more opportunities to learn.

Robots practice better

Researchers generated more than 1,300 virtual environments using SceneSmith. These included realistic homes, offices, shops, and themed spaces. Robots practiced common household tasks such as placing fruit on plates, moving soda cans between shelves and tables, opening cabinets, and putting cups into sinks.

To evaluate the system, researchers tested robot control programs inside 100 different SceneSmith environments. An AI agent assessed whether each robot completed its assigned task. Human reviewers agreed with the AI’s evaluations more than 99 percent of the time.

The team also tested a robot controller trained mostly on real-world data. Even though it had never encountered a SceneSmith environment before, it completed tasks such as picking up an apple from a bowl and placing it on a cutting board. Researchers also remotely controlled robots inside the virtual environments, where they opened cabinets, stored bottles, and navigated between rooms, showing that the scenes could support realistic physical interactions.

In user studies involving more than 200 people, over 90 percent rated SceneSmith’s environments as more realistic than those created by earlier methods. They also found the system better at following written instructions. SceneSmith can even generate new 3D objects, assigning them realistic physical properties such as weight, friction, and movement.

However, creating a highly detailed scene can currently take several hours because the AI carefully reviews every object and layout. The researchers believe faster computing and larger 3D object libraries will significantly improve performance, helping robots gain the rich training data needed for real-world applications.

Source link