The Physical AI Technology Stack Explained

Physical AI is not a single technology. It is a stack of interconnected layers, each solving a different part of the problem of making machines interact intelligently with the physical world. Understanding this stack helps you evaluate companies, choose tools, and see where the field is heading.

This guide breaks down the physical AI technology stack into seven layers, from raw hardware at the bottom to deployed robot systems at the top. For each layer, we explain what it does, why it matters, and which companies and tools operate there.

The operational path through those layers is covered in the robot foundation models guide, robotics dataset directory, and sim-to-real workflow.

The seven-layer model

Think of the physical AI stack like the OSI model for networking, but for robots. Each layer builds on the one below it:

┌─────────────────────────────────────┐
│  7. DEPLOYMENT                      │
│     Fleet management, monitoring    │
├─────────────────────────────────────┤
│  6. POLICY                          │
│     Robot actions and decisions     │
├─────────────────────────────────────┤
│  5. WORLD MODELS                    │
│     Physics prediction, planning    │
├─────────────────────────────────────┤
│  4. PERCEPTION                      │
│     Sensing and understanding       │
├─────────────────────────────────────┤
│  3. SIMULATION                      │
│     Virtual training environments   │
├─────────────────────────────────────┤
│  2. OPERATING SYSTEM                │
│     Middleware and communication    │
├─────────────────────────────────────┤
│  1. HARDWARE                        │
│     Robots, sensors, compute        │
└─────────────────────────────────────┘

Each layer can be addressed independently, but the real challenge (and opportunity) lies in the interfaces between layers. A world model is only useful if it connects to perception below and policy above. Simulation only accelerates development if it transfers to real hardware.

Layer 1: Hardware

What it does

The hardware layer encompasses the physical components: robot bodies (arms, legs, wheels), actuators (motors, hydraulics), sensors (cameras, LiDAR, force/torque, tactile), and onboard compute modules.

Why it matters

Software cannot compensate for hardware limitations. A robot with poor actuators cannot perform precise manipulation regardless of how good its AI is. Sensor quality directly bounds perception accuracy. Compute power limits the complexity of models that can run in real time.

Key components

Actuators and robot bodies:

• Collaborative arms: Universal Robots UR series, UFACTORY xArm, Franka FR3
• Humanoids: Unitree G1/H1, Figure 02, Tesla Optimus
• Quadrupeds: Unitree Go2/B2, Boston Dynamics Spot
• Custom actuators: proprietary designs from Figure, Tesla, Apptronik

Sensors:

• Depth cameras: Intel RealSense, Orbbec Femto
• LiDAR: Ouster OS series
• Tactile: XELA uSkin, GelSight
• Force/torque: Robotiq FT, ATI Industrial Automation
• Event cameras: Prophesee Metavision

Edge compute:

• NVIDIA Jetson AGX Orin (275 TOPS, $1,999)
• NVIDIA Jetson Orin Nano Super (67 TOPS, $249)
• NVIDIA Jetson Orin Nano 2 (78 TOPS announced; expected H1 2027, not currently shipping)
• Qualcomm RB5/RB6
• Intel NUC series (for less compute-intensive applications)

For the entry-level Jetson roadmap, see Jetson Orin Nano 2 versus Orin Nano Super. The announced Nano 2 specifications and NVIDIA’s performance claims should not be treated as shipping-product benchmarks.

Current state

Hardware is becoming commoditized at the low end (SO-101 arms for $110, Go2 quadrupeds for $1,600) while the high end pushes boundaries in dexterity and force density. The key bottleneck is dexterous hands: no commercially available robot hand matches human manipulation capability.

Layer 2: Operating system and middleware

What it does

The OS layer provides communication infrastructure between hardware components, abstracts sensor interfaces, manages real-time control loops, and offers tools for configuration and debugging.

Why it matters

Without middleware, every robotics team would write custom communication protocols, driver interfaces, and timing systems from scratch. The OS layer allows components from different vendors to interoperate and enables code reuse across robot platforms.

Key technologies

ROS 2 (Robot Operating System 2): The dominant open-source middleware for robotics. ROS 2 provides:

• Publish/subscribe messaging (DDS-based)
• Service calls for request/response patterns
• Action servers for long-running tasks
• tf2 for coordinate frame management
• Launch systems for multi-process orchestration

ROS 2 is maintained by Open Robotics (now partially under Intrinsic/Alphabet). The current LTS release is Jazzy Jalisco (May 2024). Nearly every non-proprietary robot uses ROS 2 as its communication backbone.

NVIDIA Isaac ROS: A set of GPU-accelerated ROS 2 packages optimized for Jetson hardware. Isaac ROS provides hardware-accelerated perception (stereo depth, visual SLAM, object detection) that runs significantly faster than CPU-based equivalents.

Real-time control frameworks:

• ros2_control: standardized interface for hardware abstraction and control loops
• Orocos RTT: hard real-time framework for control applications
• EtherCAT: industrial communication protocol for actuator control at 1 kHz+

Proprietary stacks: Some companies (Tesla, Boston Dynamics) run entirely proprietary OS stacks optimized for their specific hardware. These offer tighter integration at the cost of ecosystem compatibility.

Current state

ROS 2 has won the middleware war for open robotics. The remaining gaps are in real-time performance (improving with rmw implementations) and ease of use (still requires significant expertise to configure correctly).

Layer 3: Simulation

What it does

Simulation provides virtual environments where robots can train, test, and validate behaviors without physical risk. Modern simulators combine physics engines (for accurate contact, friction, and dynamics), renderers (for camera-realistic imagery), and domain randomization (for robust sim-to-real transfer).

Why it matters

Training robot policies in the real world is slow (real-time only), expensive (hardware wears out), and dangerous (failed experiments can break things). Simulation enables millions of training episodes in parallel at thousands of times real-time speed. The challenge is the “sim-to-real gap”: behaviors learned in simulation must transfer to the messier real world.

Key technologies

NVIDIA Isaac Sim: The most comprehensive robot simulator available. Built on Omniverse, Isaac Sim provides:

• PhysX 5 physics (GPU-accelerated rigid body, soft body, fluid)
• RTX ray-traced rendering for photorealistic synthetic images
• Domain randomization for visual and physical properties
• ROS 2 bridge for direct integration with robot code
• Programmatic scene generation via Python APIs

Isaac Sim runs on NVIDIA GPUs and is free for individual developers.

MuJoCo: The fastest physics engine for articulated body simulation. MuJoCo excels at:

• Contact-rich manipulation (pushing, grasping, assembly)
• Reinforcement learning training (millions of steps per hour)
• Accurate tendon and actuator modeling
• Minimal setup complexity

MuJoCo is free and open source (Apache 2.0). It is the standard for RL research but lacks photorealistic rendering.

Genesis: A newer generative simulation platform that uses AI to automatically create diverse training environments. Rather than hand-designing simulation scenarios, Genesis procedurally generates environments with varied objects, layouts, and physical properties.

Gazebo: The traditional ROS-integrated simulator. Less powerful than Isaac Sim but tightly coupled with the ROS 2 ecosystem. Useful for navigation, planning validation, and sensor simulation without requiring NVIDIA hardware.

Current state

The sim-to-real gap is narrowing. Domain randomization, system identification, and high-fidelity physics have made it possible to train locomotion policies entirely in simulation and deploy directly on real robots (demonstrated by Unitree, ETH Zurich, and others). Manipulation sim-to-real remains harder due to contact complexity.

The Unitree robots and physical AI ecosystem guide shows how one vendor’s hardware, sensors, SDKs, simulation resources, models, and applications map across these layers.

Layer 4: Perception

What it does

Perception transforms raw sensor data into structured understanding of the environment: where are objects, what are they, what is their state, where are the obstacles, where is the robot relative to the world.

Why it matters

A robot cannot manipulate what it cannot see or understand. Perception is the bridge between raw sensor streams and actionable world state. Poor perception means the robot operates blind or with incorrect information.

Key capabilities

3D scene understanding:

• Point cloud segmentation and object detection
• 6D pose estimation (position + orientation of objects)
• Scene reconstruction from depth and color cameras
• Semantic mapping (assigning labels to spatial regions)

Open-vocabulary perception: Modern vision-language models (VLMs) enable robots to find and identify objects they have never seen before using natural language descriptions. Models like GroundingDINO, SAM 2 (Segment Anything), and OWLv2 allow perception without pre-defined object categories.

Visual SLAM and localization: Simultaneous Localization and Mapping allows mobile robots to build maps of unknown environments while tracking their own position. ORB-SLAM3, RTAB-Map, and NVIDIA Isaac Visual SLAM are widely used implementations.

Tactile perception: Extracting information from touch sensors: slip detection, object hardness, surface texture, grasp stability. GelSight and XELA sensors feed into learned models that interpret tactile signals.

Key tools and models

Tool	Function	Provider
GroundingDINO	Open-vocabulary object detection	IDEA Research
SAM 2	Segment anything from prompts	Meta
FoundationPose	6D pose estimation without CAD models	NVIDIA
AnyGrasp	Grasp detection from point clouds	Shanghai Qi Zhi
NVIDIA Isaac Perceptor	Hardware-accelerated stereo and SLAM	NVIDIA
ORB-SLAM3	Visual-inertial SLAM	Universidad de Zaragoza

Current state

Open-vocabulary perception has transformed what is possible. Robots can now find arbitrary objects described in natural language, without requiring object-specific training data. The remaining challenges are speed (running large VLMs on edge hardware), accuracy in cluttered scenes, and handling transparent or reflective objects.

Layer 5: World models

What it does

World models predict how the physical world will change in response to actions. They answer questions like: “If I push this object to the left, where will it end up?” or “If I pour water into this cup at this angle, will it overflow?” World models provide the predictive understanding that enables planning.

Why it matters

Reactive policies (see state, output action) work for simple tasks but fail at anything requiring multi-step reasoning. World models let robots mentally simulate the consequences of actions before executing them, enabling planning, risk assessment, and more intelligent behavior.

Types of world models

Learned physics models: Neural networks trained to predict physical dynamics from observation. These models learn implicit physics from video or interaction data without explicit equations. Examples include video prediction models that forecast future frames given current state and action.

NVIDIA Cosmos: A family of world foundation models from NVIDIA designed to generate physically plausible future world states. Cosmos models are trained on massive video datasets and simulation data, learning to predict how scenes evolve over time. They can be used to generate synthetic training data or to provide planning signals for robot policies.

Structured world models: Models that maintain explicit object representations (position, velocity, shape) and predict interactions between them. More interpretable than pixel-level prediction but require structured input.

Foundation model approaches: Companies like Physical Intelligence and Google DeepMind are building world understanding directly into their robot foundation models, blurring the line between world models and policies. Rather than a separate prediction module, physical understanding emerges from large-scale training on diverse interaction data.

Current state

World models represent the most active research frontier in physical AI. Pure video prediction models can generate plausible futures but struggle with physical accuracy over long horizons. Combining learned models with structured physics knowledge is an active area. No production robot system relies primarily on a learned world model yet, but this is expected to change within 2 to 3 years.

Layer 6: Policy

What it does

The policy layer produces the actual robot actions: joint angles, velocities, gripper commands, and whole-body motions. It takes perception outputs (and optionally world model predictions) and converts them into executable motor commands.

Why it matters

Policy is where all other layers converge into physical action. The quality of a robot’s behavior is ultimately determined by its policy, even if that policy depends on good perception, accurate world models, and proper hardware.

Types of policies

Classical control: PID controllers, impedance control, model predictive control (MPC). Well-understood, reliable for known tasks, but brittle to environmental variation. Still dominant in production manufacturing robots.

Reinforcement learning (RL): Policies trained through trial-and-error in simulation. RL excels at locomotion (walking, running, obstacle traversal) and has been demonstrated for dexterous manipulation in specific domains. Companies like Unitree use RL-trained locomotion policies on their quadrupeds and humanoids.

Imitation learning / Behavior cloning: Learn policies from human demonstrations. The human teleoperates the robot, and a neural network learns to map observations to actions from the recorded data. This is the approach behind ACT, Diffusion Policy, and most robot learning papers from 2023 onward.

Vision-Language-Action (VLA) models: The newest approach: large multimodal models (similar to GPT-4 or Gemini) that take images and language instructions as input and output robot actions directly. Examples include RT-2, Octo, OpenVLA, and Pi-0. These models promise generalization across tasks and robots but currently trade off speed for generality.

Key policy models (2026)

Model	Type	Developer	Key property
Pi-0	VLA foundation model	Physical Intelligence	Cross-robot, cross-task generalization
RT-2-X	VLA	Google DeepMind	Trained on data from 22 labs
OpenVLA	VLA (open source)	Stanford / Berkeley	Open weights, reproducible
Octo	Generalist policy	UC Berkeley	Open source, multi-task
Diffusion Policy	Imitation learning	Columbia / MIT	High-precision manipulation
ACT	Imitation learning	Stanford (Tony Zhao)	Bimanual manipulation
PPO / SAC	RL algorithms	Standard	Locomotion, sim-to-real

Current state

VLA models are the hottest area of policy research. They demonstrate surprising generalization (following novel instructions, adapting to new objects) but remain too slow and unreliable for production deployment in most cases. Production systems still rely on classical control or narrow RL policies. The gap between research demos and deployed systems is closing but remains significant.

Layer 7: Deployment

What it does

The deployment layer handles everything needed to run robots reliably in the real world: fleet management, remote monitoring, over-the-air updates, safety systems, failure recovery, and human oversight.

Why it matters

A robot that works in the lab but cannot be managed at scale is not a product. Deployment infrastructure determines whether robotics companies can go from 10 robots to 10,000 without proportionally scaling their engineering teams.

Key capabilities

Fleet management:

• Remote monitoring of robot health (battery, joint temperature, error rates)
• Centralized task assignment and scheduling
• Software and model deployment across fleets
• Performance analytics and utilization tracking

Safety systems:

• Emergency stop mechanisms (hardware and software)
• Collision detection and avoidance
• Speed and force limiting in human-occupied areas
• Compliance with ISO 10218 and ISO/TS 15066 for collaborative robots

Over-the-air (OTA) updates:

• Deploying new software, models, and configurations without physical access
• Staged rollouts (canary deployments for robots)
• Rollback capability when updates cause issues

Data pipelines:

• Collecting robot operation data for model improvement
• Uploading interaction logs for offline analysis
• Privacy-compliant data handling in customer environments

Off-robot infrastructure: where NVIDIA’s “Scale-In” fits

Not every deployment service runs on the robot. Centralized training, simulation, fleet analytics, shared model serving, and large telemetry pipelines can depend on GPU clusters and storage systems outside the machine. NVIDIA describes one part of that data-center architecture as Scale-In.

These terms cover different paths through NVIDIA’s AI factory design:

Term	Infrastructure role
Scale-up	Connect accelerators inside a tightly coupled compute system, primarily through NVLink
Scale-out	Connect GPU servers across the compute fabric through Spectrum-X Ethernet or Quantum InfiniBand
Scale-in	Connect, secure, provision, and observe the north-south access path around the compute domain

In NVIDIA’s implementation, BlueField-4 DPUs, DOCA, and Spectrum-X Ethernet form the Scale-In layer. BlueField-4 provides a host-independent processing domain for network, storage, security, policy, and telemetry services. DOCA supplies the software and orchestration model, while Spectrum-X carries traffic between compute systems and users, applications, data sources, services, and external storage.

The practical point of the DPU is separation and offload. Infrastructure services can enforce tenant isolation and access policy, virtualize storage access, collect telemetry, and process traffic without doing all of that work on the server’s host CPU. NVIDIA specifies an 800 Gb/s network interface and inline processing of up to 800 Gb/s for the described BlueField-4 architecture; these are platform capabilities, not measured performance for a particular robotics workload.

For physical AI teams, this layer matters only when off-robot infrastructure becomes substantial. A lab running one policy on one Jetson does not need an AI-factory networking architecture. A company centrally post-training models, serving shared models, ingesting data from a large robot fleet, or isolating multiple teams on shared GPU infrastructure may need the security, storage, observability, and control-plane functions that NVIDIA groups under Scale-In. Even then, Scale-In is NVIDIA’s architecture and terminology, not an industry-wide infrastructure standard.

This is separate from inference that runs directly on a robot. The Cosmos 3 Edge deployment guide shows that an on-device policy can remove a data-center GPU from the demonstrated inference loop while training, telemetry, model distribution, and fleet operations may still depend on centralized systems.

Key tools

Tool	Function	Provider
FetchCore	Fleet management for warehouse AMRs	Zebra/Fetch
InOrbit	Cloud-based robot fleet management	InOrbit
Formant	Robot data and fleet management platform	Formant
Freedom Robotics	Remote monitoring and management	Freedom Robotics
AWS IoT RoboRunner	Cloud orchestration for robot fleets	Amazon Web Services
Foxglove	Data visualization and debugging	Foxglove

Current state

Deployment infrastructure is less glamorous than AI research but is critical for commercialization. Companies that solve fleet management, reliable OTA updates, and remote diagnostics at scale have a significant competitive advantage. Most humanoid companies are still in pilot phase and have not yet faced true deployment scale challenges.

How the layers interact

The real complexity in physical AI is not within any single layer but at the interfaces:

Hardware to OS: Driver interfaces, real-time communication protocols, calibration routines OS to Simulation: Digital twin synchronization, sim-to-real parameter matching Simulation to Perception: Synthetic data generation for perception model training Perception to World Models: Structured state estimation feeding prediction models World Models to Policy: Planning with predicted futures, model-predictive approaches Policy to Deployment: Action safety checking, graceful degradation, human intervention triggers

Companies that own multiple layers (NVIDIA with hardware + simulation + perception + world models) have an integration advantage. Startups typically focus on one or two layers and integrate with others via open standards (ROS 2, ONNX, standard sensor interfaces).

Where value is concentrating

In 2026, investment and competitive intensity are highest in three layers:

1. World models / foundation models: The most VC funding is flowing here (Physical Intelligence, Skild, NVIDIA Cosmos). The belief is that the model layer will be the most defensible.
2. Hardware (humanoids): The humanoid hardware race has attracted billions in funding despite uncertain near-term returns.
3. Simulation: NVIDIA’s dominance in simulation (via Isaac Sim + Cosmos) is drawing competitive responses and significant investment.

The less-funded but equally critical layers (deployment, middleware, sensors) represent opportunities for focused companies to build essential infrastructure.

Frequently asked questions

What is the physical AI technology stack? It is the set of layered technologies required to build intelligent robots: hardware, operating systems/middleware, simulation, perception, world models, policies, and deployment infrastructure.

Which layer is most important? No single layer dominates. A robot with perfect AI but bad hardware is useless, and a robot with perfect hardware but no intelligence is just an expensive sculpture. The value comes from integration across layers.

Do I need all seven layers to build a robot? Not necessarily. A simple pick-and-place robot might only need hardware, a basic OS (ROS 2), and classical control (policy). World models and simulation become important for more complex, generalizable behaviors.

What is the difference between a world model and a policy? A world model predicts what will happen (if I do X, the world becomes Y). A policy decides what to do (given the current state, I should do X). World models inform policies but do not directly produce actions.

Which companies span the most layers? NVIDIA spans hardware (Jetson), simulation (Isaac Sim), perception (Isaac Perceptor), and world models (Cosmos). Google DeepMind spans simulation (MuJoCo), perception, world models, and policy (RT-2). No single company dominates all seven layers.

Is ROS 2 required for physical AI? Not required, but it is the most common middleware choice for non-proprietary systems. Tesla, Boston Dynamics, and some Chinese companies use proprietary stacks. For startups and research teams, ROS 2 is nearly universal.

How does the physical AI stack differ from autonomous vehicle stacks? The layers are similar, but physical AI is more general (manipulation, not just navigation) and handles contact-rich interaction. AV stacks focus heavily on perception and planning for driving; physical AI must also solve grasping, tool use, and object manipulation.

Where should a new robotics startup focus? Focus on one or two layers where you have a unique advantage. Trying to build the full stack from scratch is capital-intensive and rarely successful. Use open-source tools (ROS 2, MuJoCo) for layers outside your core focus and build defensibility in your chosen area.

Sources

• ROS 2 Jazzy Jalisco Released – Open Robotics — current LTS release (supported through 2029)
• NVIDIA Isaac Sim GitHub — open-source simulation platform (Apache 2.0)
• NVIDIA Isaac Lab GitHub — GPU-accelerated robot learning framework
• NVIDIA Cosmos 3 Launch Press Release — world foundation models for physical AI
• MuJoCo Documentation — physics simulation engine overview
• Open-sourcing MuJoCo – Google DeepMind — acquisition and open-source timeline
• Physical Intelligence – π₀ announcement — robot foundation model
• RT-2: Vision-Language-Action Models – arXiv — Google DeepMind’s VLA paper (July 2023)
• Octo: An Open-Source Generalist Robot Policy – arXiv — UC Berkeley, 800k trajectories
• NVIDIA Jetson Orin product page — edge compute lineup and pricing
• NVIDIA Jetson Orin Nano 2 announcement — announced specifications and expected H1 2027 availability
• NVIDIA BlueField-4 Scale-In technical overview — NVIDIA’s Scale-In definition, architecture, and deployment use cases
• NVIDIA BlueField platform — official DPU product overview and BlueField-4 positioning