pi0.5 fine-tuned on four xArm6 pushing tasks (diverse400, simulation)
Full fine-tune of openpi's pi0.5 base (PyTorch) with its native conditional flow-matching loss, on all 400
training episodes of bx6d/r2s2r-pushing-diverse400:
push a 65 mm tennis ball to one side of a reference object, 100 demonstrations per task. One model serves all four
tasks, and the instruction selects the task:
| task | instruction |
|---|---|
| left of cube | Push the tennis ball to the left of the cube. |
| right of cube | Push the tennis ball to the right of the cube. |
| left of bowl | Push the tennis ball to the left of the bowl. |
| right of bowl | Push the tennis ball to the right of the bowl. |
Use relative_step_010000: 10,000 steps, trained to predict the arm targets as offsets from the observed arm
joints (the gripper command stays absolute). It succeeds in 188 of 200 closed-loop trials. Three earlier
checkpoints with absolute targets (1k, 1.5k and 2.5k steps) are kept for comparison. Observations are the table
and wrist cameras at 224x224 plus eight actual joint positions. For every checkpoint, inference_example.py
returns the next 20 absolute joint/gripper targets at 20 Hz (one second), as in
bx6d/r2s2r-pi05-blue-cube-to-bowl-stage1.
Training and evaluation are simulation only: MuJoCo actuator-executed demonstrations rendered with Blender EEVEE. Physical camera extrinsics and real robot dynamics are unvalidated.
Checkpoints and closed-loop results
Suite totals out of 100 (25 trials per task) with Wilson 95% confidence intervals, and per-task successes out of
50 (seen + shifted). Per-suite, per-task counts are in evaluation/.
| checkpoint | arm targets | seen | shifted | left of cube | right of cube | left of bowl | right of bowl |
|---|---|---|---|---|---|---|---|
step_001000 |
absolute | 40 (31-50) | 32 (24-42) | 27 | 21 | 21 | 3 |
step_001500 |
absolute | 46 (37-56) | 39 (30-49) | 42 | 7 | 33 | 3 |
step_002500 |
absolute | 83 (74-89) | 73 (64-81) | 50 | 43 | 39 | 24 |
relative_step_010000 |
relative | 97 (92-99) | 91 (84-95) | 49 | 50 | 40 | 49 |
- seen: 25 training layouts per task, with their recorded camera, lighting and arm start. The release has no held-out episodes.
- shifted: 25 other training layouts per task with the ball and the reference object each moved 2-3 cm in a random direction, inside the task's training span. Not verified by a scripted expert.
Why relative targets. With absolute targets, right pushes often stall. The hand reaches the point where the
demonstrations start pushing, 7.7 cm beside the ball at table height, and then holds still or drifts away instead of
creeping in to touch the ball. The demonstrations start right pushes very slowly: the hand typically moves 0.7 mm in
the first 2 s and 5 mm in the first 4 s (left pushes: 2 mm and 15 mm; medians over 100 demonstrations per task).
pi0.5 reads the joint state as 256 bins, about 3 mrad for most arm joints here. During those first seconds the arm moves
less than one bin, so an absolute target cannot reliably sit just past the current position, and the result varies
a lot between checkpoints (right of cube: 21, 7, 43 and 17 of 50 at 1k, 1.5k, 2.5k and 3.5k steps; 3.5k is not
included). With offsets, a small steady offset gives a steady creep, because the observed joints are added back
exactly. On the same trials and noise seeds, 41 trials succeed only with relative_step_010000 and 9 only with
step_002500 (exact McNemar p = 6e-6). The relative run had already reached 187 of 200 at 3.5k and at 5k steps
(summaries in evaluation/).
Remaining failures of relative_step_010000. 10 of its 12 failures are left-of-bowl near misses: the ball ends
12-14 cm to the left of the bowl's centre, but only 0.2-1.9 cm clear of its rim (2 cm required). One left-of-cube
push stops at a 7.7 cm centre gap (8 cm required), and one right-of-bowl trial never touches the ball. There were
no safety aborts.
Evaluation protocol
The release's simulation scenes are not published; they were reconstructed from the shared base scene for evaluation (tennis ball 65 mm, 60 g, solid-sphere inertia; cube or bowl as a free reference object; the release's state layouts). Checked against the release itself: replaying recorded demonstrations open loop reproduces the recorded ball trajectories within about 1 mm (median maximum error 0.5-1.2 mm per task, anchors exact), and re-rendered recorded frames match the recordings to JPEG noise (about 46 dB PSNR).
- Rollout: before every chunk, both cameras are re-rendered with Blender EEVEE using the training renderer; actions run through the MuJoCo position actuators with a 90 s simulated timeout.
- Success, following the release's
relation_definition: the ball was pushed by the hand, then rests to the instructed side of the reference along the fixed table camera's image axis (left = negative world x) with a centre gap of at least 8 cm and an edge clearance of at least 2 cm. The reference moved at most 3 mm, the ball never rose more than 6 mm (a push, not a pick), and the ball is still, below 3 cm/s, with no hand contact for 0.5 s. This test accepts all 400 demonstrations. - Safety: aborts count as failures (simulator instability, joint limits, leaving the workspace, arm-link collisions).
Files
relative_step_010000/ recommended; arm targets as offsets from the observed joints
step_001000/, step_001500/, step_002500/ absolute targets
model.safetensors FP32 weights (14.5 GB; lm_head tied to the embeddings)
norm_stats.json train-only q01/q99 normalization over the 400 episodes, required for inference;
"action_representation" says whether the arm targets are absolute or offsets
config.json training configuration of the run
inference_example.py standalone policy wrapper and command-line demo (handles both representations)
evaluation/ suites and per-checkpoint summaries
Optimizer states are not included.
Inference
Same environment and contract as
bx6d/r2s2r-pi05-six-tasks-diverse600: Python 3.11,
torch 2.7.1, rlinf-openpi==0.1.1 with openpi's patched Transformers modules, the PaliGemma tokenizer, and
JAX_PLATFORMS=cpu.
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("bx6d/r2s2r-pi05-pushing-diverse400",
allow_patterns=["relative_step_010000/*", "inference_example.py"])
sys.path.insert(0, path)
from inference_example import INSTRUCTIONS, Policy
policy = Policy(f"{path}/relative_step_010000", device="cuda") # about 1 min to build on CPU
actions = policy(table_rgb, wrist_rgb, joint_state, "Push the tennis ball to the left of the cube.", seed=0)
# actions: float (20, 7) -> per 50 ms step: joint1..joint6 absolute targets (rad), gripper 0..255
Command-line check against a dataset frame (the instruction defaults to the episode's own):
JAX_PLATFORMS=cpu python inference_example.py --checkpoint relative_step_010000 \
--episode <dataset>/tasks/push_tennis_ball_right_of_cube/train/episode_000005 --frame 300
For a relative checkpoint the model outputs offsets, and Policy adds the observed arm joints (the joint_state
you pass) back before returning, so the output is absolute targets either way. The arm targets match the
evaluation server bit for bit on the same input and seed, for both representations; the gripper also matches after
the [0, 255] clip both apply. Use the instructions verbatim (INSTRUCTIONS). Inputs are uint8 RGB
(224, 224, 3) table and wrist images (center-square crop of the native 640x480 image, OpenCV INTER_AREA
resize) and the 8 actual joint positions (six arm joints and two gripper driver joints, radians). The output is
(20, 7): six absolute arm joint targets (rad) and a gripper command in [0, 255]. In evaluation, arm targets
were clipped to the actuator ranges, commanded joint speed was limited to 1 rad/s, and all 20 targets were
executed through a C1 cubic spline to 100 Hz servo targets before re-observing.
Training
- Data: all 400 training episodes (400,832 action steps): per task, five location strata crossed with five robot-start regions, ball/reference positions, push approach, distance and withdrawal varied, and camera/lighting variation. The prompt carries each episode's own instruction.
- Common recipe: global batch 64, AdamW (betas 0.9/0.95, weight decay 0.01), BF16 compute with FP32 master weights; all parameters trained; FSDP with replicated parameters, no gradient checkpointing, NVIDIA B300s.
relative_step_010000: each sample's 20 arm targets minus the observed arm joints at that frame, gripper absolute; the q01/q99 normalization of the offsets covers every valid (frame, horizon step) pair of the 400 episodes. 10,000 steps frompi05_base, peak LR 1e-5 with 500 warmup steps and cosine decay to 10% at step 10,000, on 2 GPUs (32 samples each, about 0.82 s per step). Training loss over the last 100 steps averaged 0.0018 (not comparable with the absolute runs' losses, which use a different normalization).step_001000,step_001500: absolute targets, peak LR 1e-5 with 500 warmup steps and cosine decay to 10% at step 1,500, on 4 GPUs (16 samples each, about 0.75 s per step). Final training loss 0.0011.step_002500: the 1.5k run resumed (weights and AdamW state) with its cosine schedule retargeted to 3,500 steps, so the LR restarted at 6.5e-6 and was 2.7e-6 at step 2,500; on 2 GPUs (32 samples each). The batches continue the same seeded sequence. Training loss over steps 2,401-2,500 averaged 0.0024.config.jsonrecordsresumed_fromandmax_steps: 3500.- Base: openpi
pi05_base(gs://openpi-assets/checkpoints/pi05_base), converted to PyTorch with openpi's converter.
Limitations
- Simulation only, on a single robot rig; evaluated in reconstructed scenes (see above) on seen and slightly shifted layouts, since the release has no held-out episodes.
- The absolute checkpoints are early and their right pushes are fragile (see above).
- 25-trial cells give roughly ±15-20-point confidence intervals.
License
No license has been assigned to these weights. They are derived from openpi's pi05_base (Physical
Intelligence) and PaliGemma, so the upstream terms (including the Gemma Terms of Use) apply.