How to Fine-Tune SmolVLA on an SO-101 Robot Arm

SmolVLA is a 450 million parameter vision-language-action model distributed through Hugging Face LeRobot. The SO-101 is a low-cost leader-follower robot-arm design supported by the same project. Together they offer a documented route from human demonstrations to a language-conditioned manipulation policy.

This is a documentation-based implementation guide, not a claim that we ran the exact setup. LeRobot changes quickly, so use the commands below as a map and verify flags against the installed release with --help. The official repository and model documentation remain the source of truth.

For model concepts, read Vision-Language-Action Models Explained. For deciding whether SO-101 fits your project, see the robotics development-kit comparison and affordable robot-arm guide.

What the workflow produces

You will assemble and calibrate a leader and follower SO-101, attach cameras, teleoperate a defined task, record a LeRobot dataset, fine-tune lerobot/smolvla_base, evaluate repeated rollouts, and deploy only after setting physical safety limits.

Fine-tuning is embodiment-specific. The base model does not know your exact motor calibration, camera positions, gripper geometry, objects, or workspace. A checkpoint trained on someone else’s SO-101 dataset is useful for learning the format, but it is not a drop-in policy for a different installation.

Hardware and compute checklist

At minimum, plan for:

• one SO-101 follower arm;
• a compatible leader arm or another supported teleoperation controller;
• separate, correctly rated power supplies;
• USB serial adapters and reliable cables;
• one or more fixed USB cameras;
• a computer for robot control and recording;
• storage for video episodes and checkpoints;
• a CUDA-capable training system or compatible cloud GPU.

The SO-101 ecosystem includes kits from several vendors. Motor models, adapter boards, printed parts, firmware, and assembly quality can vary. Follow the LeRobot SO-101 hardware guide for the exact bill of materials tied to your software release. Do not infer electrical limits from a visually similar kit.

Hugging Face documents roughly four hours for 20,000 SmolVLA training steps on a single NVIDIA A100 with its example configuration. That is a reference workload, not a guarantee. Dataset size, batch size, storage speed, precision, GPU, logging, and software version all change runtime and memory use.

1. Pin and verify the software environment

Create an isolated environment and install LeRobot using the official installation instructions. For a source checkout, the SmolVLA documentation currently identifies the smolvla optional dependencies:

git clone https://github.com/huggingface/lerobot.git
cd lerobot
pip install -e ".[smolvla]"

Record the Git commit or package version, Python version, PyTorch build, CUDA version, and camera backend. Save lerobot-train --help with the experiment. A tutorial copied months later may target a different dataset schema or command interface.

Test cameras, serial devices, and model loading before recording. If Linux device names change after reconnecting, use stable device paths where supported. Do not run the follower at full speed merely to confirm communication.

2. Configure and calibrate the SO-101

Follow the current LeRobot SO-101 documentation for motor IDs, ports, homing, calibration, leader configuration, and follower configuration. Calibration maps raw servo positions into a consistent joint representation. A mismatch can command a joint toward a mechanical limit.

Before teleoperation:

1. secure the base to a stable surface;
2. clear the workspace and define an emergency stop action;
3. verify each joint direction at low speed;
4. test the gripper separately;
5. set conservative range and torque limits;
6. save leader and follower calibration files;
7. label which calibration belongs to which physical arm.

Recalibration can change the action distribution. Treat it as a dataset boundary. Do not merge episodes from materially different calibrations without testing that the state and action conventions remain compatible.

3. Design one narrow task

Start with a task that has an unambiguous reset and success condition, such as moving one cube from a marked set of starting zones into a tray. Avoid beginning with open-ended tidying or many object classes.

Write down:

• the natural-language instruction;
• allowed object positions and orientations;
• robot starting pose;
• camera positions;
• maximum episode duration;
• success tolerance;
• collision and abort conditions;
• reset procedure.

The robot teleoperation guide explains synchronization, variation planning, and failure labels. The datasets and benchmarks directory provides examples of broader robot-data structures.

4. Record demonstrations

Use the recording command documented for your installed LeRobot release. A current workflow uses a robot configuration, teleoperator configuration, camera definitions, task text, and Hub repository ID. Exact port names and camera indices are machine-specific.

lerobot-record \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --teleop.type=so101_leader \
  --teleop.port=/dev/ttyACM1 \
  --teleop.id=my_leader_arm \
  --dataset.repo_id=YOUR_USER/so101_pick_cube \
  --dataset.num_episodes=50 \
  --dataset.single_task="Pick up the cube and place it in the tray"

Camera options are omitted because their syntax and device identifiers must match the installed release. Add the documented camera configuration and verify the preview before motion.

Hugging Face recommends around 50 episodes as an initial SmolVLA baseline. Its example uses five cube positions with ten demonstrations per position and reports poor results with a similar 25-episode dataset. Do not turn that observation into a universal rule. Collect enough repeated examples of each planned variation, then measure held-out task success.

After every short collection block, replay samples. Check camera framing, action alignment, gripper state, task text, episode boundaries, and reset consistency. It is cheaper to discard five broken episodes than to discover a configuration error after 100.

5. Audit and split the dataset

Keep validation and test episodes separate by collection session, not by randomly selecting nearly identical frames from the same recording. A useful test set changes object position, lighting, distractors, or starting pose within the intended operating envelope.

Review:

• episode duration and frame count;
• missing or duplicated images;
• joint positions near limits;
• sudden action jumps;
• task success and failure reason;
• coverage per variation;
• camera or calibration changes.

Training loss cannot tell you whether a policy is safe or whether the validation set is independent. Preserve failed rollouts for evaluation, but do not automatically train behavior cloning on operator mistakes.

6. Fine-tune SmolVLA

The official SmolVLA guide currently shows this pattern:

lerobot-train \
  --policy.path=lerobot/smolvla_base \
  --dataset.repo_id=YOUR_USER/so101_pick_cube \
  --batch_size=64 \
  --steps=20000 \
  --output_dir=outputs/train/my_smolvla \
  --job_name=so101_pick_cube_smolvla \
  --policy.device=cuda \
  --wandb.enable=true

Start with a batch size that fits memory. Save checkpoints and the complete configuration. If using Weights & Biases, confirm that images, instructions, or repository metadata do not expose private environments. Disable external logging when policy requires it.

Do not choose the best checkpoint from training loss alone. Low imitation loss can coexist with zero task success if observations, normalization, action conventions, or evaluation differ. Evaluate checkpoints on repeated physical rollouts under controlled conditions.

7. Evaluate before deployment

Define a fixed evaluation matrix with seen and held-out object positions. Report numerator and denominator, such as 17 successful placements in 20 attempts, rather than “works reliably.” Record partial failures separately: missed grasp, dropped object, wrong target, timeout, collision, and failure to recover.

Measure:

• end-to-end task success;
• success by variation;
• completion time;
• repeatability across resets;
• latency and action rate;
• behavior after occlusion or a displaced object;
• unsafe joint-limit or collision events;
• operator interventions.

The sim-to-real workflow offers a broader validation and rollback structure. For compute placement, see edge computers for robotics.

8. Deploy with physical safeguards

Run initial policy rollouts at reduced speed with a person ready to stop motion. Keep the workspace clear and exclude people, pets, sharp tools, hot objects, and fragile items. Software confidence is not a safety rating.

Verify that inference uses the same camera ordering, calibration, state normalization, action convention, task text, and control frequency as training. Log each deployed checkpoint and dataset version. If the system changes, return to validation rather than assuming the policy transfers.

An SO-101 is a development platform, not a certified collaborative industrial robot. Its low price does not remove pinch points, unexpected motion, electrical risk, or the need for supervision.

Common failure modes

The arm barely moves or oscillates: Check calibration, action normalization, checkpoint compatibility, and whether the policy produces the action convention expected by the follower.

Training loss falls but rollouts fail: Inspect dataset alignment, validation independence, camera order, language instruction, and action scaling. Add task-relevant variation rather than merely training longer.

The policy succeeds only at one position: The dataset likely correlates the task with a narrow visual setup. Record repeated demonstrations across deliberately planned positions.

Grasps fail despite good approach motion: Check gripper calibration, object variation, camera occlusion near contact, and whether demonstrations consistently close at the same stage.

Performance changes after restart: Confirm device identifiers, calibration files, camera settings, lighting, model checkpoint, and control rate.

Practical verdict

SO-101 and SmolVLA form a credible learning stack because hardware support, dataset tooling, training code, and model weights are available in one ecosystem. The limiting factor is not typing the training command. It is collecting consistent demonstrations and proving the resulting policy works outside the exact examples it saw.

Use 50 episodes as an official starting reference, not a promise. Version everything, test multiple checkpoints, report failures, and keep a safe teleoperation fallback.

FAQ

Can SmolVLA run directly on an SO-101 without fine-tuning?

The base model should be adapted to your task, cameras, calibration, and embodiment. A public checkpoint may demonstrate compatibility, but it is not evidence that an unmodified policy will control your setup correctly.

Do I need an NVIDIA A100?

No specific GPU is mandatory, but compute and memory affect batch size and training time. Hugging Face cites about four hours for 20,000 steps on one A100 as a reference configuration.

Is one SO-101 arm enough?

You need a follower plus a way to produce demonstrations. The common setup uses a second SO-101 as a leader, although another LeRobot-supported teleoperator may work.

How many cameras should I use?

Use the minimum views that make the task observable. One fixed camera may work for a simple tabletop task. A wrist or second view can reduce occlusion but adds synchronization and calibration work.

Can I publish the recorded dataset to the Hugging Face Hub?

Yes, if you have the rights to all recorded content and have removed private information. Use a private repository when the workspace, people, labels, or objects are sensitive.

Sources

• Hugging Face LeRobot repository
• LeRobot SO-101 documentation
• LeRobot SmolVLA documentation
• SmolVLA paper and project resources
• NVIDIA SO-101 sim-to-real course