--- title: Visualize Dataset (v2.0+ latest dataset format) emoji: 💻 colorFrom: blue colorTo: green sdk: docker app_port: 7860 pinned: false license: apache-2.0 hf_oauth: true hf_oauth_scopes: - read-repos hf_oauth_expiration_minutes: 480 --- # LeRobot Dataset Visualizer LeRobot Dataset Tool and Visualizer is a web application for interactive exploration and visualization of robotics datasets, particularly those in the LeRobot format. It enables users to browse, view, and analyze episodes from large-scale robotics datasets, combining synchronized video playback with rich, interactive data graphs. ## Project Overview This tool is designed to help robotics researchers and practitioners quickly inspect and understand large, complex datasets. It fetches dataset metadata and episode data (including video and sensor/telemetry data), and provides a unified interface for: - Navigating between organizations, datasets, and episodes - Watching episode videos - Exploring synchronized time-series data with interactive charts - Analyzing action quality and identifying problematic episodes - Visualizing robot poses in 3D using USD models - Paginating through large datasets efficiently ## Key Features - **Dataset & Episode Navigation:** Quickly jump between organizations, datasets, and episodes using a sidebar and navigation controls. - **Synchronized Video & Data:** Video playback is synchronized with interactive data graphs for detailed inspection of sensor and control signals. - **Overview Panel:** At-a-glance summary of dataset metadata, camera info, and episode details. - **Statistics Panel:** Dataset-level statistics including episode count, total recording time, frames-per-second, and an episode-length histogram. - **Action Insights Panel:** Data-driven analysis tools to guide training configuration — includes autocorrelation, state-action alignment, speed distribution, and cross-episode variance heatmap. - **Filtering Panel:** Identify and flag problematic episodes (low movement, jerky motion, outlier length) for removal. Exports flagged episode IDs as a ready-to-run LeRobot CLI command. - **3D Replay:** Visualize robot joint poses frame-by-frame in an interactive 3D scene, with end-effector trail rendering. Supports SO-100, SO-101, OpenArm bimanual, and Unitree G1 (Dex3, Dex1, Inspire and BrainCo hands) robots. - **3D Hands:** Automatically detects native SOMA156 hand annotations. Shows wrists, 21-joint hand skeletons, wrist trails and the saved camera trajectory alongside a synchronized camera overlay, with a Follow camera control. - **Annotations Panel:** Hand-edit the v3.1 language schema (`language_persistent` + `language_events`) — subtask, plan, memory, interjection + paired speech, and VQA atoms with bounding-box / keypoint / count / attribute / spatial answers. VQA bboxes and keypoints render as overlays on the video player; drag or click on a camera to draw new ones. Backed by an optional FastAPI service (in `backend/`) for parquet rewrites and HF Hub push. - **Efficient Data Loading:** Uses parquet and JSON loading for large dataset support, with pagination, chunking, and lazy-loaded panels for fast initial load. - **Responsive UI:** Built with React, Next.js, and Tailwind CSS for a fast, modern user experience. ## Hand annotations Choose **Advanced → 3D Hands** (below **Annotations**) or open an annotated episode with `?tab=hands`. Detection requires the 156D `action` plus `annotation.hand_joints_camera` and `annotation.hand_joints_world` (126 values each), `annotation.hand_action_valid`, `annotation.hand_observed` (left/right) and `annotation.hand_reference_frame`. No model weights or meshes are required. Missing or masked poses remain gaps in both views. When `annotation.camera_world_from_optical` contains saved rigid transforms, the viewer reconstructs recording-wide hand positions from camera-space joints, anchored to the first saved camera pose. The camera path also breaks at missing transforms. Without them, it displays the stored segment-start FLU coordinates and breaks wrist trails at reference resets. Positions are in meters. The overlay uses the source camera and per-source-episode calibration saved in `annotation_studio/run.json`, resolving exported episodes through `source_episode_map`; a video feature's `info["camera.intrinsics"]` is also supported. Pinhole and OpenCV fisheye distortion are projected onto the original video at its saved resolution. Unknown cameras, missing calibration and mismatched video dimensions disable the overlay. Estimated calibration is labeled. ## Technologies Used - **Next.js** (App Router) - **React** - **Recharts** (for data visualization) - **Three.js** + **@react-three/fiber** + **@react-three/drei** (for the 3D scene) - **@huggingface/usd-web-viewer** (loads and poses the robot USD models in [lerobot/robot-usds](https://huggingface.co/datasets/lerobot/robot-usds)) - **hyparquet** (for reading Parquet files) - **Tailwind CSS** (styling) ## Getting Started ### Prerequisites This project uses [Bun](https://bun.sh) as its package manager. If you don't have it installed: ```bash # Install Bun curl -fsSL https://bun.sh/install | bash ``` ### Installation Install dependencies: ```bash bun install ``` ### Development Run the development server: ```bash bun dev ``` Open [http://localhost:3000](http://localhost:3000) with your browser to see the result. You can start editing the page by modifying `src/app/page.tsx` or other files in the `src/` directory. The app supports hot-reloading for rapid development. ### Other Commands ```bash # Build for production bun run build # Start production server bun start # Run linter bun run lint # Format code bun run format ``` ### Environment Variables - `DATASET_URL`: (optional) Base URL for dataset hosting (defaults to HuggingFace Datasets). - `NEXT_PUBLIC_ANNOTATE_BACKEND_URL`: (optional) URL of the FastAPI annotation backend (`backend/app.py`). When set, the Annotations tab can save edits and rewrite parquet shards / push to the Hub. When unset the tab is read/edit only with sessionStorage persistence. ## Annotations backend (optional) The Annotations tab edits LeRobot v3.1 language atoms — `language_persistent` (broadcast subtask/plan/memory) and `language_events` (per-frame interjection / vqa / speech) — and renders existing bbox/keypoint atoms over the video player. Edits live in `sessionStorage` by default; to write the new columns into `data/chunk-*/file-*.parquet` (matching the writer in [lerobot#3471](https://github.com/huggingface/lerobot/pull/3471)) and push the result to the Hub, run the bundled FastAPI service: ```bash # 1. install + start the backend (port 7861 by default) cd backend python -m venv .venv && source .venv/bin/activate pip install -r requirements.txt uvicorn app:app --port 7861 --reload # 2. start the visualizer with the backend URL configured cd .. NEXT_PUBLIC_ANNOTATE_BACKEND_URL=http://127.0.0.1:7861 bun run dev ``` The backend exposes: - `POST /api/dataset/load` — load a dataset by `repo_id` or `local_path` - `GET /api/episodes/{ep}/atoms` — list atoms for an episode - `POST /api/episodes/{ep}/atoms` — replace atoms (event timestamps are snapped to exact source-frame timestamps before persisting) - `GET /api/episodes/{ep}/frame_timestamps` — used client-side for snapping - `POST /api/export` — rewrite parquet with the new language columns plus the dataset-level `tools` column (drops legacy `subtask_index`) - `POST /api/push_to_hub` — export and push to a target repo ## Docker Deployment This application can be deployed using Docker with bun for optimal performance and self-contained builds. ### Build the Docker image ```bash docker build -t lerobot-visualizer . ``` ### Run the container ```bash docker run -p 7860:7860 lerobot-visualizer ``` The application will be available at [http://localhost:7860](http://localhost:7860). ### Run with custom environment variables ```bash docker run -p 7860:7860 -e DATASET_URL=your-url lerobot-visualizer ``` ## Contributing Contributions, bug reports, and feature requests are welcome! Please open an issue or submit a pull request. ### Acknowledgement The app was orignally created by [@Mishig25](https://github.com/mishig25) and taken from this PR [#1055](https://github.com/huggingface/lerobot/pull/1055)