Robot Foundation Models Explained

Robot foundation models are large neural networks pre-trained on broad datasets of robot interactions, then fine-tuned for specific tasks or embodiments. They represent a fundamental shift in how robots learn: instead of training a new model from scratch for every task, developers start with a model that already understands general physical manipulation and adapt it to their specific use case.

Think of it like this. Before foundation models, teaching a robot to pick up a cup required collecting thousands of demonstrations of that exact task, on that exact robot, in that exact environment. With a robot foundation model, you start with a network that has already seen 800,000 different robot episodes across dozens of robot types and hundreds of tasks, then fine-tune it with a few dozen demonstrations of your specific scenario.

This guide explains the complete model stack without treating a research video, downloadable checkpoint, developer API, and commercial deployment as equivalent evidence.

Current robot foundation model capability matrix

Last reviewed: August 24, 2026.

Model	Model role	Access	Weights and license	Supported robots	Training data visibility	Deployment maturity
Gemini Robotics On-Device 2	On-robot VLA policy	Controlled access	Closed	Multi-embodiment partner hardware	Model card, raw corpus not public	Partner evaluation
Gemini Robotics-ER 2	Embodied reasoning and planning	Developer ecosystem	Closed	High-level, robot-agnostic orchestration	Model card, raw corpus not public	Available reasoning layer, not a motor policy
NVIDIA GR00T N1.7	Cross-embodiment VLA	Public repository, Early Access	Apache 2.0 code, NVIDIA Open Model License weights	Bimanual, semi-humanoid, humanoid	Mixture described, full corpus not public	Early Access, not GA
Cosmos3-Edge-Policy-DROID	Post-trained world action policy	Public model and framework	OpenMDW 1.1 model license	DROID reference embodiment	DROID workflow and recipe public	On-device Jetson Thor reference, vendor benchmark
Figure Helix 02	Whole-body policy stack	Figure hardware only	Closed	Figure 03	Private fleet data	Vendor-reported deployments
SmolVLA 450M	Compact VLA	Public through LeRobot	Accessible code and weights, verify checkpoint terms	SO-100, SO-101 and LeRobot adapters	Community datasets public	Reproducible prototyping
pi-zero-point-five	Generalist VLA	Public openpi repository	Public code and checkpoints, verify terms	ALOHA, DROID and custom adaptations	Public recipes plus private pretraining	Research adaptation
OpenVLA 7B	Manipulation VLA	Public	MIT code, Llama 2 restrictions on weights	OXE manipulation embodiments	OXE mixture documented	Research and fine-tuning
Octo	Generalist diffusion policy	Public	Public code and checkpoints	Multiple manipulation arms	OXE mixture documented	Established research baseline
RT-2	Historical VLA	Paper only	Closed	Google mobile manipulator	High-level description	Research artifact

A policy model converts observations and instructions into actions. A reasoning model interprets a scene, calls tools, and creates a plan but does not necessarily command joints. A deployment model is adapted to a target robot, control rate, runtime, and safety envelope.

Google separates Gemini Robotics-ER 2 from Gemini Robotics On-Device 2. Figure packages proprietary reasoning and visuomotor layers on its own hardware. GR00T, SmolVLA, openpi, OpenVLA and Octo offer different degrees of public adaptation. See the open model comparison for licensing and reproducibility.

What makes a model a “foundation model” for robotics

A foundation model has three defining characteristics in the robotics context:

Pre-trained on broad, diverse data. The model learns from massive datasets spanning multiple robot types, tasks, and environments. This is not a specialist model trained only on one arm doing one thing.

Transferable to new tasks. After pre-training, the model can be fine-tuned with relatively small amounts of data to perform new tasks it never explicitly trained on. The general knowledge transfers.

Multi-modal inputs. Robot foundation models process multiple input types simultaneously: camera images, language instructions, proprioceptive data (joint angles, forces), and sometimes depth maps or point clouds. They output low-level motor actions.

The technical term for many of these models is Vision-Language-Action (VLA) model. They take in vision (what the robot sees), language (what the robot should do), and output actions (motor commands). Google DeepMind pioneered this architecture with RT-2 in July 2023, and the concept has since become the dominant paradigm.

How robot foundation models differ from LLMs

Large language models like GPT-4 or Claude process text and produce text. Robot foundation models share the transformer architecture but differ in critical ways:

Output type. LLMs output tokens (words, characters). Robot foundation models output continuous motor actions: joint positions, velocities, or torques that physically move a robot. The gap between predicting the next word and predicting the next joint position is enormous.

Real-time requirements. An LLM can take 500ms to generate a response. A robot arm moving at speed needs action predictions every 10 to 50 milliseconds. Robot foundation models must run inference at control frequencies of 20Hz or higher.

Embodiment awareness. An LLM does not know what a body is. A robot foundation model must understand the specific physical constraints of its embodiment: how many joints it has, their range of motion, the dynamics of its actuators, and the physics of contact.

Training data. LLMs train on trillions of tokens of text scraped from the internet. Robot data is expensive. Even the largest robot datasets contain only hundreds of thousands of episodes because each episode requires a real robot (or high-fidelity simulation) performing real movements. This data scarcity is the fundamental bottleneck.

Grounding. LLMs can hallucinate because wrong text has no physical consequence. Robot models cannot afford to hallucinate because a wrong action command can destroy hardware, damage the environment, or harm humans. The output must be physically plausible.

Safety constraints. Robot foundation models must respect joint limits, torque limits, collision avoidance, and force thresholds. These hard physical constraints do not exist in the text domain.

GR00T: NVIDIA’s accessible humanoid model line

NVIDIA announced Project GR00T in March 2024 and now publishes GR00T N1.7 for generalized bimanual and humanoid skills. The repository provides pretrained weights, fine-tuning and inference workflows, but labels the release Early Access. Stable features, fully validated benchmarks, and commercial support are described as GA items.

Architecture. GR00T N1 uses a dual-system architecture inspired by Daniel Kahneman’s thinking fast and slow framework. System 1 is a diffusion transformer that generates fluid motor actions in real-time, analogous to human reflexes and intuition. System 2 is a vision-language module that interprets the environment through camera images and language instructions, handling deliberate reasoning and planning.

Training data. NVIDIA documents a mixture of robot demonstrations and synthetic data, but it does not publish the complete training corpus for N1.7. Treat the provided checkpoints and adaptation datasets as accessible artifacts, not as full training-data disclosure.

Access and licensing. The repository and checkpoints are public, but code and model weights use different terms. Code is under Apache 2.0 and the weights use the NVIDIA Open Model License. Teams should review both before commercial use rather than treating “open” as one license category.

Ecosystem integration. GR00T integrates with NVIDIA’s broader robotics platform: Isaac Sim for simulation, Isaac Lab for reinforcement learning, and the Jetson Thor compute module for on-robot inference. This tight integration means developers can train in simulation, validate in sim, then deploy to hardware.

GR00T Dreams. An extension called GR00T-Dreams uses Cosmos world foundation models to generate synthetic training videos. Given a single image, it produces videos of the robot performing new tasks in new environments, creating synthetic training data without physical demonstrations.

RT-2: historical VLA milestone

Google DeepMind released RT-2 (Robotic Transformer 2) in July 2023. It was the first model to demonstrate that vision-language models trained on internet-scale data could be directly used for robotic control.

Architecture. RT-2 is built on large vision-language model backbones, specifically PaLI-X (5B and 12B parameters) and PaLM-E (55B parameters). The key innovation is that it represents robot actions as text tokens. Instead of outputting continuous values directly, it discretizes actions into token sequences that the language model can predict just like it predicts the next word.

Training data. RT-2 trains on a combination of internet-scale vision-language data and approximately 130,000 robot demonstration episodes from the RT-1 dataset collected on mobile manipulators.

Emergent capabilities. The most significant finding was that RT-2 exhibits emergent capabilities. By training on web-scale data, the model learned concepts it never saw in robot demonstrations. For example, it could follow the instruction “pick up the extinct animal” (a plastic dinosaur) even though no robot demonstration ever mentioned dinosaurs. The model transferred visual knowledge from web pre-training to physical manipulation.

Limitations. RT-2 is a closed research artifact. Google has not released the model weights publicly. It also requires significant compute for inference, making real-time control on embedded hardware challenging. The model was primarily demonstrated on a single robot platform (mobile manipulators at Google).

Successor. RT-2 led to Open X-Embodiment and Gemini Robotics. Current coverage should separate Gemini Robotics-ER reasoning from Gemini Robotics On-Device policy, while retaining RT-2 as historical context.

Current closed systems and reproducible alternatives

Google DeepMind’s current line separates embodied reasoning from action generation. Gemini Robotics-ER 2 handles spatial reasoning, planning and tool use. Gemini Robotics On-Device 2 is an efficient local VLA. Access remains controlled, so neither belongs in an open-model ranking.

Figure describes Helix 02 as a proprietary full-body stack on Figure 03. BMW and household results provide deployment evidence, but remain vendor-reported results on one vertically integrated platform.

SmolVLA offers a low-cost workflow through LeRobot and SO-101-class arms. Physical Intelligence’s openpi provides pi-zero, pi-zero-FAST and pi-zero-point-five checkpoints. OpenVLA remains a larger public manipulation model, while Octo remains an established baseline.

The robotics datasets and benchmarks directory documents the data layer. The sim-to-real workflow covers the validation gates required before hardware deployment.

Octo: the open-source generalist robot policy

Octo, developed by researchers at UC Berkeley and collaborators, was released in 2024 as an open-source generalist robot policy. It is specifically designed to be a practical foundation model that anyone can use and fine-tune.

Architecture. Octo is a transformer-based diffusion policy. It uses a transformer encoder to process observations (images from multiple cameras, language instructions, proprioceptive state) and a diffusion-based action decoder to output motor commands.

Training data. Octo was pre-trained on 800,000 diverse robot trajectories from the Open X-Embodiment dataset, which is the largest robot manipulation dataset assembled to date. This data spans dozens of different robot platforms and hundreds of tasks.

Flexibility. Octo is designed for adaptation. It supports flexible task definitions (language goals, image goals, or both), flexible observation inputs (any combination of cameras), and can be quickly fine-tuned to new robot hardware with different action spaces.

Fine-tuning efficiency. The key practical advantage of Octo is fine-tuning speed. Researchers demonstrated that Octo could be adapted to a completely new robot with a different action space using just a few hundred demonstrations, achieving strong performance in under an hour of training.

Fully open. Unlike RT-2, Octo is fully open-source. The model weights, training code, evaluation code, and fine-tuning scripts are all publicly available on GitHub. This makes it the most accessible robot foundation model for the research community.

How to use a robot foundation model in practice

The practical workflow for using a robot foundation model follows a pattern:

1. Select a pre-trained model. Choose based on your robot type, task, license, runtime and available adaptation path. Octo targets manipulation research, while GR00T N1.7 targets bimanual and humanoid workflows. There is no universal starting model.

2. Collect demonstrations. Record demonstrations of your specific task on your specific robot. The required number depends on task variation, embodiment mismatch, safety constraints and the base model. Do not treat a vendor or paper’s sample count as a deployment guarantee.

3. Fine-tune. Adapt the pre-trained model to your embodiment and task using your demonstration data. This updates the model’s action decoder to output commands appropriate for your robot’s specific joint configuration and action space.

4. Evaluate in simulation. Before deploying to hardware, test the fine-tuned model in simulation to verify it produces reasonable behaviors.

5. Deploy to hardware. Run inference on your robot’s compute module (Jetson, GPU workstation, etc.) and evaluate real-world performance.

The current state and what comes next

Robot foundation models in mid-2026 span research checkpoints, Early Access platforms, partner APIs and vertically integrated commercial robots. A single readiness label would be misleading.

Key challenges remain. Data scarcity is the biggest issue. The entire Open X-Embodiment dataset has 800,000 episodes. GPT-4 trained on trillions of tokens. There is a six-order-of-magnitude gap in training data availability.

Sim-to-real transfer partially addresses this by generating synthetic data in simulation, but simulated data does not perfectly match real-world physics. Domain randomization and techniques like GR00T-Dreams help bridge this gap.

Inference speed is another challenge. NVIDIA now demonstrates the 4B Cosmos3-Edge-Policy-DROID model running fully on a Jetson AGX Thor T5000. It generates action chunks rather than replanning at every observation, and the reported result is specific to NVIDIA’s DROID and RoboLab configuration. Read the Cosmos 3 Edge deployment analysis before treating on-device availability as general real-time robot control.

The practical direction is reusable pretraining plus embodiment-specific post-training. Teams should select by robot interface, data compatibility, runtime, license and measured task evidence.

FAQ

What is the difference between a robot foundation model and a regular robot learning model? A regular robot learning model is trained from scratch on data from one robot doing one task. A robot foundation model is pre-trained on diverse data across many robots and tasks, then fine-tuned for specific applications. The foundation model approach transfers general manipulation knowledge, requiring far less task-specific data.

Are robot foundation models the same as large language models? No. They share the transformer architecture but differ in outputs (motor actions versus text tokens), timing requirements (real-time control versus conversational), and training data (robot demonstrations versus internet text). Some models like RT-2 bridge the gap by representing actions as text tokens, but the underlying challenges are distinct.

Which robot foundation model should I use for my project? If you want an open-source model you can fine-tune today, start with Octo. If you are building for a humanoid robot within the NVIDIA ecosystem, use GR00T N1. If you are using Hugging Face tools, consider models available through LeRobot like Pi0 or GR00T N1.5.

How much data do I need to fine-tune a robot foundation model? It depends on how different your task is from the pre-training distribution. For tasks similar to what the model has seen (basic pick and place with a standard arm), 50 to 100 demonstrations may suffice. For novel tasks or unusual robot configurations, you might need 200 to 500 demonstrations.

Can I run a robot foundation model on embedded hardware? Yes, but it requires careful consideration. Smaller models (under 1B parameters) can run on NVIDIA Jetson Orin at acceptable frequencies. Larger models need more powerful compute or model distillation techniques to reduce size while preserving capability.

What is a Vision-Language-Action (VLA) model? A VLA model takes visual input (camera images), language input (task instructions), and outputs actions (motor commands). The concept was pioneered by Google DeepMind with RT-2 in July 2023 and has become the standard architecture for robot foundation models.

Is training data the main bottleneck? Yes. Robot demonstration data is expensive and time-consuming to collect compared to internet text data. The field is addressing this through simulation (generating synthetic training data), teleoperation datasets (crowd-sourcing demonstrations), and data augmentation techniques.

Will robot foundation models replace traditional motion planning? Not entirely, at least not in the near term. Foundation models excel at generalizable manipulation in unstructured environments. Traditional motion planning (inverse kinematics, trajectory optimization) remains superior for high-precision, repetitive industrial tasks where reliability and exactness matter more than generalization.

Sources

• NVIDIA Cosmos 3 Edge post-training and deployment workflow

• GR00T N1: An Open Foundation Model for Generalist Humanoid Robots - NVIDIA’s dual-system VLA architecture paper

• NVIDIA Isaac GR00T GitHub - public code, weights, and licensing details

• NVIDIA Isaac GR00T N1 research page - official research publication

• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control - Google DeepMind’s July 2023 paper

• What is RT-2? - accessible overview of the VLA concept

• Octo: An Open-Source Generalist Robot Policy - model overview and results

• Octo paper - pre-trained on 800k trajectories from Open X-Embodiment

• Octo GitHub repository - open-source code and model weights

• Physical Intelligence pi-zero announcement - first generalist policy blog post

• Pi-zero: A Vision-Language-Action Flow Model - technical paper on flow matching architecture