Vision-Language-Action Models Explained

Vision-Language-Action (VLA) models are the architecture class that connects internet-scale knowledge to physical robot control. They take visual observations and language instructions as input, then output motor actions directly. No hand-coded motion planners, no explicit state machines, no separate perception pipeline feeding a separate planning pipeline. One model, end to end.

This is the core idea that separates VLA models from previous approaches: instead of building a stack of specialized modules (a vision system that detects objects, a language parser that interprets commands, and a motion planner that generates trajectories), you train a single neural network that learns the entire mapping from pixels and words to joint torques.

The three-stage pipeline

VLA models process information through three connected stages. Understanding this pipeline is key to understanding why they work and where they fail.

Stage 1: Perception (Vision)

The vision component processes camera images (typically one or more RGB cameras mounted on or around the robot). Modern VLA models use vision transformers (ViTs) or similar architectures pre-trained on large image datasets. The vision encoder converts raw pixels into a rich feature representation that captures spatial relationships, object properties, textures, and scene geometry.

Crucially, these vision encoders are typically not trained from scratch on robot data. They inherit representations learned from billions of internet images, which gives them strong priors about object categories, spatial relationships, and visual semantics before they ever see a robot workspace.

Stage 2: Language understanding

The language component processes natural language instructions (“pick up the red cup,” “move the box to the left of the plant,” “fold the towel in half”). This is where VLA models gain their generalization power. By building on pre-trained language models, they inherit an understanding of object categories, spatial relationships, and task semantics that would be impossible to learn from robot demonstrations alone.

The language model does not just parse instructions. It provides a bridge between human concepts and physical actions. When you say “move the snack toward the person,” the model can infer what counts as a “snack” and what “toward” means in the current spatial context, even if it was never trained on that exact combination.

Stage 3: Action generation

The action head converts the fused vision-language representation into physical robot commands. Depending on the model, these can be joint positions, joint velocities, end-effector poses, or discretized action tokens. Some models output actions directly (like classification into action bins), while others use diffusion-based generation for smoother, more continuous motion.

The action space varies by implementation. RT-2 discretizes actions into 256 bins per dimension and treats them as text tokens. Pi-zero uses flow matching to generate continuous actions. Octo uses a diffusion-based action head. Each approach has tradeoffs in precision, smoothness, and training efficiency.

Key VLA models in detail

Current model-role matrix

System	Policy model	Reasoning model	Deployment layer	Access
Gemini Robotics 2	On-Device 2 generates actions	ER 2 handles spatial reasoning, planning and tools	On-Device 2 targets local execution	Controlled access
Figure Helix 02	Proprietary visuomotor and full-body policies	Proprietary semantic layer	Integrated on Figure 03	Closed
NVIDIA GR00T N1.7	Cross-embodiment VLA	External planning can sit above it	NVIDIA and Isaac ecosystem	Public Early Access
SmolVLA	Compact action-chunk VLA	VLM backbone, not a separate planner	LeRobot and affordable hardware	Public
openpi	Flow and autoregressive VLA policies	Language-conditioned policy	ALOHA and DROID examples	Public
OpenVLA and Octo	Manipulation policies	No separate production reasoning service	Research deployment	Public

A policy model outputs actions. A reasoning model decomposes goals and interprets a scene. A deployment model is adapted to one embodiment and runtime. Vendors may package all three, but developers should evaluate each layer separately.

Gemini Robotics 2, Helix 02, and current open workflows

Google DeepMind documents Gemini Robotics-ER 2 and Gemini Robotics On-Device 2 as separate models. ER 2 is the reasoning and tool-use layer. On-Device 2 is a local VLA that generates actions across supported embodiments. Neither is a generally downloadable open policy.

Figure’s Helix 02 is a closed, vertically integrated stack for Figure 03. Published BMW and household results are relevant deployment evidence, but remain vendor reports that outside developers cannot reproduce as a model comparison.

SmolVLA is a 450 million parameter VLA distributed through Hugging Face LeRobot. Physical Intelligence’s openpi repository provides pi-zero, pi-zero-FAST and pi-zero-point-five checkpoints. OpenVLA and Octo remain public baselines with different compute and licensing profiles.

Use the open foundation-model comparison for weights and licenses, and the dataset directory for training and evaluation resources.

RT-2 (Google DeepMind, 2023)

RT-2 (Robotic Transformer 2) was the breakthrough paper that proved VLA models could work. Published by Google DeepMind in July 2023, it demonstrated that a vision-language model trained on internet-scale data could be adapted to output robot actions, and that doing so transferred web knowledge to robotic control.

Architecture: RT-2 builds on PaLI-X (55B parameters) or PaLM-E (12B parameters) as the backbone. It fine-tunes these models on robot demonstration data, treating robot actions as text tokens. Actions are discretized into 256 bins and represented as strings (e.g., “1 128 91 241 5 101 127” for a 7-dimensional action).

Key findings:

• Emergent capabilities: RT-2 can follow commands involving concepts it never saw in robot training data, purely from its web pre-training. It can pick up objects based on abstract reasoning (“move the object that is least often used in the kitchen to the counter”).
• Chain-of-thought: Adding reasoning chains to the model improves manipulation success on novel tasks.
• Scale result in the paper: RT-2’s authors reported stronger novel-object generalization for the larger evaluated configuration. This is a within-study result, not a benchmark against current models.

Limitations: RT-2 requires massive compute (the 55B model runs inference on a TPU cluster), making it impractical for real-time control on actual robots without cloud inference. It was demonstrated on simple tabletop manipulation with a single arm.

Octo (UC Berkeley, 2024)

Octo is the first open-source generalist robot policy. Developed at UC Berkeley and released in 2024, it represents the community’s answer to the question: “What would it take to build an open, fine-tunable foundation model for robot manipulation?”

Architecture: Octo is a transformer-based diffusion policy pre-trained on 800,000 robot trajectories from the Open X-Embodiment dataset. It uses a modular design with separate tokenizers for observations (images, proprioception) and tasks (language instructions, goal images), feeding into a shared transformer backbone, which then connects to a diffusion-based action head.

Key features:

• Open source: Weights, code, and training data are publicly available.
• Flexible inputs: Supports variable numbers of cameras, different proprioception formats, and both language and goal-image task specifications.
• Fine-tunable: Designed to be quickly adapted to new robots and tasks with small amounts of data (as few as 50 demonstrations).
• Multiple embodiments: Pre-trained across multiple robot platforms (WidowX, Franka, Kuka, xArm, and others).

Size: The base model has approximately 93 million parameters, making it small enough to run on a single GPU in real time.

Limitations: Octo’s performance out of the box (without fine-tuning) is modest compared to larger proprietary models. Its strength is as a starting point for fine-tuning, not as a zero-shot generalist.

Pi-zero (Physical Intelligence, 2024)

Pi-zero is the first VLA model from Physical Intelligence, the startup founded by robotics researchers including Sergey Levine and Karol Hausman. It represents a bet that VLA models can work as truly general robot controllers, not just for simple manipulation but for complex multi-step tasks across very different robot platforms.

Architecture: Pi-zero uses a flow matching architecture built on top of a pre-trained vision-language model (PaLI-Gemma 2B as the VLM backbone). Flow matching generates continuous actions by learning a vector field that transforms noise into action trajectories. This is different from both the discrete token approach (RT-2) and standard diffusion (Octo).

Key features:

• Cross-embodiment: Trained on data from single-arm robots, dual-arm robots (ALOHA), and mobile manipulators, all in one model.
• Dexterous manipulation: Demonstrated on tasks including laundry folding, table cleaning, and box assembly.
• Zero-shot transfer: Can perform some tasks without fine-tuning by following language instructions.
• Hybrid architecture: The VLM backbone (System 2) handles reasoning while the flow matching head (System 1) handles fast reactive control.

Demonstrated tasks: Laundry folding (with a bimanual setup), clearing a table, packing items into a box, operating a toaster. These represent a significant step up in task complexity compared to the simple pick-and-place demonstrations of earlier VLA models.

Status: Physical Intelligence now publishes openpi code and checkpoints for pi-zero, pi-zero-FAST and pi-zero-point-five. These are adaptable research releases, not universal drop-in policies.

GR00T N1.7 (NVIDIA, 2026)

GR00T N1.7 is NVIDIA’s current accessible VLA for bimanual, semi-humanoid and humanoid data. NVIDIA labels it Early Access. Code and weights are public under separate licenses, while GA-level stability and support are future status gates.

Architecture: GR00T N1 uses a dual-system design:

• System 2 (slow, deliberative): A vision-language module that interprets the environment through camera images and processes language instructions. This module handles task understanding and high-level planning.
• System 1 (fast, reactive): A diffusion transformer that generates fluid motor actions in real time based on the representations from System 2.

Key features:

• Humanoid-specific: Designed for the high-dimensional action spaces of humanoid robots (30+ degrees of freedom).
• Real-time control: The dual-system architecture allows System 1 to run at high frequency (for smooth motion) while System 2 reasons at a slower rate.
• Simulation integration: Deeply integrated with NVIDIA Isaac Sim for synthetic data generation and sim-to-real transfer.
• Open weights: NVIDIA released the model weights on GitHub (Isaac-GR00T repository).

Target users: Research labs and humanoid robot companies building general-purpose humanoid policies. Not intended for direct consumer use.

Why VLA models matter for robotics

The data efficiency argument

Traditional robotic manipulation requires extensive engineering for each new task. A pick-and-place pipeline might take weeks of tuning for a single object category. VLA models change this by leveraging internet-scale pre-training. The model already “knows” what thousands of objects look like and what spatial instructions mean. This knowledge transfers to robot manipulation, drastically reducing the amount of robot-specific training data needed.

The generalization argument

The central promise of VLA models is generalization. A well-trained VLA model should handle new objects, new instructions, and new environments without retraining. RT-2 demonstrated this: it could follow commands about objects it had never manipulated, purely because it had seen those objects in web images during pre-training. This kind of zero-shot generalization was impossible with previous approaches.

The scalability argument

VLA models can be trained on data from many different robots. The Open X-Embodiment dataset combines trajectories from over 20 robot platforms. This means improvements in VLA models benefit the entire robotics community, not just one lab with one specific robot. It also means that every new robot that collects data contributes to a shared foundation.

Current limitations

VLA models are not ready to replace purpose-built industrial automation. Key limitations include:

Precision: Current VLA models achieve manipulation success rates of 60 to 90 percent on benchmark tasks. Industrial applications often require 99.9+ percent reliability. The gap is still large.

Speed: Many VLA models run inference at 2 to 10 Hz (actions per second), which is too slow for dynamic tasks. Newer models like pi-zero-FAST are pushing toward 30+ Hz, but this remains an active challenge.

Safety: A model that generates actions end-to-end from neural network inference has no explicit safety constraints unless they are engineered into the training or architecture. This is a concern for deployment around humans.

Compute requirements: Larger VLA models (RT-2 at 55B parameters) require cloud inference, adding latency and network dependency. Smaller models (Octo at 93M, pi-zero at 3B) can run on edge hardware but sacrifice some generalization capability.

Long-horizon tasks: VLA models struggle with tasks requiring many sequential steps and complex state tracking. A single pick-and-place action is tractable, but a 20-step assembly task with error recovery remains very challenging.

The road ahead

The field is moving fast. In 2023, RT-2 was a research demonstration on a single robot in one lab. By 2025, Physical Intelligence demonstrated multi-step household tasks, NVIDIA released open-weight models for humanoids, and open-source alternatives like Octo made the technology accessible to any lab with a robot arm and a GPU.

The convergence of larger pre-trained VLMs, more robot training data (through efforts like Open X-Embodiment and DROID), better action generation architectures (flow matching, diffusion), and cheaper edge compute (Jetson Orin Nano at $249) suggests that VLA models will become the default control architecture for general-purpose robots within the next two to three years.

The companies that master VLA training at scale (Physical Intelligence, Google DeepMind, NVIDIA) will likely define how robots learn and generalize in the same way that OpenAI and Anthropic defined how language models work. The race is on.

FAQ

What is the difference between a VLA model and a regular robot policy?

A regular robot policy maps observations to actions using a model trained only on robot data. A VLA model additionally incorporates a pre-trained vision-language model, which means it inherits knowledge from internet-scale image and text data. This gives it the ability to understand novel objects and instructions without additional robot training. The “language” component is what distinguishes VLA models from earlier visuomotor policies.

Can I run a VLA model on my own robot?

Yes, with Octo, OpenVLA, SmolVLA, openpi or GR00T, subject to licenses and interfaces. Public weights do not remove the need for action-space adaptation, calibration, safety limits and hardware validation.

How much training data does a VLA model need?

It depends on whether you are training from scratch or fine-tuning. Octo was pre-trained on 800,000 trajectories but can be fine-tuned to new tasks with as few as 50 demonstrations. RT-2 used about 130,000 robot episodes for its manipulation fine-tuning. If you are fine-tuning an existing VLA model to a new task on a familiar robot, 50 to 200 demonstrations is often sufficient for basic capabilities.

Are VLA models better than reinforcement learning for robotics?

They address different problems. Reinforcement learning can achieve superhuman performance on narrow tasks through trial and error. VLA models provide broad generalization and language understanding. In practice, the best systems combine both: VLA models provide a foundation of broad capabilities, and RL fine-tuning optimizes specific behaviors. Pi-zero uses this hybrid approach, and several research groups are exploring RL-based post-training for VLA models.

What hardware do I need to deploy a VLA model on an edge device?

Hardware requirements depend on precision, image count, action horizon, control frequency and runtime optimization. Do not infer real-time performance from parameter count or TOPS alone. Benchmark the complete observation-to-command loop.

How do VLA models handle failure and error recovery?

Most current VLA models do not have explicit error recovery mechanisms. They simply predict the next action given the current observation. If a grasp fails, the model sees the new state (object still on table, gripper empty) and predicts a new action. More sophisticated approaches use closed-loop replanning, where the model generates short action sequences and re-observes between each sequence. This is an active research area, and long-horizon error recovery remains one of the hardest open problems in VLA research.

Sources

• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control - Google DeepMind, July 2023
• What is RT-2? - official overview of the VLA concept
• Octo: An Open-Source Generalist Robot Policy - architecture and results
• Octo paper - 93M parameter model, 800k trajectories from Open X-Embodiment
• Octo GitHub repository - open-source weights and training code
• Pi-zero: A Vision-Language-Action Flow Model for General Robot Control - Physical Intelligence, October 2024
• Physical Intelligence pi-zero blog post - model announcement and demonstrations
• GR00T N1: An Open Foundation Model for Generalist Humanoid Robots - NVIDIA, March 2025
• NVIDIA Isaac GR00T GitHub - public code and model documentation
• GR00T N1-2B on Hugging Face - model weights and documentation
• Google DeepMind Gemini Robotics 2 - reasoning and on-device model roles
• Hugging Face SmolVLA - public VLA workflow
• Physical Intelligence openpi - public checkpoints