Robotics Datasets and Benchmarks Directory
Robot-learning results are only meaningful when the dataset, embodiment, task split, and evaluation procedure are visible. This directory maps commonly used datasets and benchmarks to the workflows they actually support.
It is not a leaderboard. A model’s LIBERO result cannot be compared directly with a DROID real-world evaluation or an Open X-Embodiment pretraining claim. Dataset size can mean episodes, trajectories, hours, steps, or frames. We preserve the publisher’s unit and link to the source.
Last reviewed: August 18, 2026.
Directory
| Resource | Embodiment | Task type | Published size | License or terms | Main purpose | Access |
|---|---|---|---|---|---|---|
| Open X-Embodiment | 20+ robot embodiments | Manipulation | 1M+ trajectories across component datasets | Per-dataset terms | Cross-embodiment pretraining | Project |
| DROID | Franka arms in diverse real environments | Tabletop manipulation | 76k trajectories, 350 hours | Check dataset terms | Real-world diversity and transfer | Project |
| BridgeData V2 | WidowX arms | Kitchen and tabletop manipulation | 60k trajectories | Check project terms | Policy training and transfer | Project |
| LeRobot community datasets | SO-100, SO-101 and other contributed robots | Manipulation and mobile tasks | Continuously growing | Dataset-card specific | Affordable data collection and VLA fine-tuning | Hub |
| ALOHA 2 datasets | Bimanual ALOHA | Bimanual manipulation | Dataset specific | Repository specific | Imitation learning and bimanual control | Project |
| RoboNet | Multiple robot arms | Object interaction | 15M video frames from 7 platforms | Check repository | Multi-robot visual dynamics | Project |
| LIBERO | Simulated manipulation arm | Long-horizon manipulation suites | 130 tasks across suites | Check repository | Lifelong learning and VLA evaluation | Project |
| Meta-World | Simulated Sawyer arm | Goal-conditioned manipulation | 50 task environments | MIT repository | Multi-task and meta-RL evaluation | Repository |
| RoboCasa | Simulated mobile manipulation | Household and kitchen tasks | Large task and scene collection, version specific | Check repository | Generalist household policy evaluation | Project |
| ManiSkill | Simulated manipulation | Rigid and contact-rich tasks | Version and task specific | Apache 2.0 code, asset terms vary | Reproducible simulation benchmarks | Project |
Always open the current dataset card. Component licenses, image rights, robot logs, annotations, and commercial-use permissions can differ inside one mixture.
Pretraining datasets versus benchmarks
A pretraining dataset supplies demonstrations used to fit a model. A benchmark defines tasks, splits, metrics, resets, and evaluation rules. One resource can support both, but the roles must remain separate.
Open X-Embodiment is primarily a dataset mixture for cross-embodiment learning. LIBERO is primarily an evaluation suite with controlled task families. DROID is real-world demonstration data and is also used to test transfer, but a paper’s private evaluation protocol may not be reproducible from the download alone.
When reading a model card, ask four questions:
- Was the evaluation task present in pretraining?
- Were objects, scenes, camera poses, or instructions held out?
- How many resets or human interventions were allowed?
- Is the reported number simulation-only or measured on physical hardware?
Dataset format and embodiment mapping
Robot data combines observations, actions, timing, task descriptions, calibration, and episode boundaries. Two datasets can both contain RGB images and joint positions but use different coordinate frames, action horizons, gripper conventions, and sampling rates.
RLDS is common in Open X-Embodiment-era pipelines. LeRobot provides its own standardized dataset tooling and Hub integration. Conversion is not purely mechanical. Verify action normalization, timestamp alignment, dropped frames, camera names, joint order, and units after conversion.
The open robot foundation-model comparison identifies which model workflow expects which data conventions. The robot SDK directory maps the surrounding software stack.
Selecting data for a project
Use public data to initialize representations and test the pipeline. Use your own demonstrations to adapt the policy to the actual robot, tooling, camera positions, objects, and environment. Public data rarely removes the need for embodiment-specific collection.
For a low-cost manipulation project, a LeRobot-compatible dataset and SO-101 hardware provide the shortest reproducible path. For broad multi-embodiment research, Open X-Embodiment is the central mixture. For real-world environmental variation, DROID is a key reference. For controlled simulated comparisons, LIBERO, Meta-World, RoboCasa, and ManiSkill provide different task regimes.
Do not choose by row count alone. Ten thousand consistent episodes for the target action space can be more useful than a much larger mixture with incompatible sensors and controls.
Benchmark reporting template
Every result should include:
- model checkpoint and code commit;
- robot or simulator version;
- task suite and exact split;
- number of trials and random seeds;
- success definition;
- resets, interventions and timeouts;
- camera and state inputs;
- control frequency and action horizon;
- inference hardware and latency;
- whether evaluation tasks overlap pretraining;
- safety stops and hardware failures.
A single average success rate hides deployment risk. Report per-task success, variance, intervention rate, cycle time, and failure category. A policy that succeeds 80 percent of the time but occasionally produces a high-force command is not ready for unsupervised hardware.
Simulation data and the reality gap
Simulation can generate perfect labels, rare events, and large task variation. It can also encode incorrect friction, contact, lighting, sensor response, and object behavior. Synthetic volume is not a substitute for real validation.
Use simulation for policy iteration, controlled ablations, perception data, and fault injection. Reserve a real-world dataset that was not used for tuning. The sim-to-real workflow guide covers domain randomization, noise, acceptance criteria, and rollback.
NVIDIA Isaac Sim supports RGB, depth, segmentation and other synthetic outputs. MuJoCo emphasizes fast dynamics and control research. Gazebo is valuable for ROS 2 integration tests. The right data source follows the failure being tested.
Maintaining this directory
Future updates should add structured fields, not prose entries. Useful additions include sensor modalities, file format, update date, data checksum, citation, known robot adapters, benchmark leakage warnings, and license changes.
The directory should eventually become filterable by embodiment, task, real versus simulated data, and allowed use. Until then, the table remains the canonical inventory and each material status change should be sourced to a project page, dataset card, or repository.
Frequently asked questions
What is the largest robotics dataset?
There is no single useful answer because mixtures use different units and may overlap component data. Compare trajectories, hours, frames, embodiments, and licensing separately.
Can I train a robot foundation model only on public data?
You can pretrain or reproduce research, but most real deployments still need data from the target robot and environment for action mapping and adaptation.
Which benchmark is best for VLA models?
LIBERO is widely used for simulated manipulation, but it does not replace real-robot evaluation. Choose a suite matching the intended task and report its limitations.
Are all Open X-Embodiment datasets commercially usable?
Do not assume so. It is a mixture of component datasets with their own terms. Audit each component used in a commercial training run.
How do I prevent benchmark leakage?
Document training mixtures, hold out tasks and scenes, check whether evaluation demonstrations occur in pretraining, and publish the split and checkpoint.