Abstract:
Apple’s research team reveals progress in research on robot “brain”. On November 4, local time, the Apple research team submitted the paper "Perceptive Humanoid Motion from Simulation" on the preprint platform arXiv, disclosing the humanoid robot autonomous control system HALO. The research team deployed the system on Yushu Technology's G1 humanoid robot for verification, allowing the robot to take the actions learned in the simulation environment and perform them in an unfamiliar real-life room.
The research team also disclosed the data generation, model training and deployment code, covering action generation, action tracking, task orchestration, simulation and real robot operation.
The Yushu G1 robot equipped with the HALO system demonstrates tasks such as crawling under the table, sitting on the sofa, and pushing boxes in different real environments. Image source: HALO research team
The project page shows the robot following instructions to push boxes, follow people, sit on chairs, sideways through a half-open door, bend over or crawl under a table, etc. These tasks require the robot to understand its surroundings and adjust its body posture according to spatial conditions.
When executing tasks, HALO is controlled collaboratively by three levels. The top layer is the task orchestration module based on the visual language model (VLM). It uses Qianwen Qwen3.6-35B-A3B by default. It runs locally on the Mac mini equipped on the robot. It shares the model service engine with the action generator and can work without the need for an Internet connection. The system can also choose to call the cloud model through API to obtain stronger reasoning and planning capabilities.
This layer needs to process the fuzzy information in the instructions and arrange actions according to the environment. For example, "sit down" does not have a clearly specified goal and requires the robot to determine where to sit from the scene in front of it; "sit on a chair" can be executed directly when the path is clear, but if there is a table blocking the way, it needs to be broken into continuous subtasks such as leaning over, approaching the chair, and sitting down. The task orchestration module continuously observes the environment through the video from the robot's perspective, determines whether the current action is successful, and advances the plan or retry accordingly, issuing a clear subtask to the action generator each time.
The action generation layer in the middle uses another Qianwen model Qwen3.5-2B. The research team uses it as a parameter-freezing encoder and cooperates with the DiT decoder responsible for generating actions to plan the whole body movements for about 5 seconds in the future based on the pictures seen by the robot and the current sub-task. This layer is replanned about three times per second, allowing the action plan to be continuously updated as the environment changes.
The bottom layer is an action tracker trained through reinforcement learning, policy distillation and other steps. It runs at a frequency of 50 times per second, combines depth images and the robot's own status, and converts action plans into joint control instructions.
The three levels are respectively responsible for task arrangement, action planning and body control, and collaboratively complete the process from understanding instructions to executing actions.
HALO's three-layer control architecture: the visual language model is responsible for task orchestration, the generation module plans the whole body movements, and the action tracker trained by reinforcement learning performs joint control. Image source: Mosaic of screenshots from the HALO project page
According to the paper, the research team conducted 360 single task tests in 6 real environments without pre-scanning or mapping, with an average success rate of 91%. In another real environment, 10 tests were conducted on three combined tasks, and the average success rate was 70%. Among them, the success rate of the combination of "leaning over the obstacle - walking towards the target - pushing the box" is 50%.
How robots obtain the data needed to complete these tasks is the focus of this research. The research team synthesized about 300 hours of primary training data in 1,000 program-generated simulated home environments to match "what the robot saw", "what instructions it received" and "how to move".
Specifically, the system first generates actions, then puts the actions into the virtual room, and finally generates images from the robot's perspective. The project page takes "Sit on an armchair" as an example, splitting it into segments such as walking forward, turning around, and sitting down. The action generation model generates action sequences based on text prompts, way points, and key postures. The actions are then screened through geometric checks to reduce problems such as foot sliding and body penetration into the ground.
Then, the system arranges the actions into the program-generated room, checks the positional relationship between the robot and the furniture, looks for a collision-free placement plan, and then uses simulation tools to render what the robot sees when it performs the action. During training, conditions such as lighting, color tone, and image blur will also be changed to make the visual input more diverse. In this way, the same process can generate supporting data including instructions, visual images and full-body movements.
HALO's synthetic data generation and training process: first generate an action trajectory, put it into the virtual scene, and then render the image from the robot's perspective to form supporting training data for training the action generator and action tracker. Image source: HALO paper
According to reports, the research team used 250 NVIDIA H100 GPUs and took about 48 hours to synthesize the main training data. The action generator is trained through imitation learning, using 64 B200s; the action tracker is trained through reinforcement learning, policy distillation and other steps, using 4 H100s.
It should be pointed out that the data generated by the simulation does not completely get rid of human participation. The team stated that the new physical tasks still require manual writing of prompt words and design of action key frames, and some actions also use human video or motion capture materials.
The research team also pointed out that currently robots mainly rely on visual perception and it is difficult to directly perceive contact situations; when objects leave the camera's field of view or are occluded, the performance of tasks involving more contacts will decline. Adding tactile perception is one of the directions for subsequent improvements.
Comments