# 50 Embodiments, 497 Episodes, One Indexed Dataset: Curating the LeRobot Community Dataset with FiftyOne

*A single prompt to an AI agent running* [*FiftyOne Skills*](https://github.com/voxel51/fiftyone-skills/tree/main) *pulled 10 episodes from every robot embodiment in the LeRobot community dataset, computed* [*video embeddings*](https://voxel51.com/video-embeddings) *with Qwen3-VL, and built a queryable similarity index. Here's the code and the thinking behind it.*

The LeRobot community dataset is one of the largest community-built pools of [cross-embodiment robot data](https://voxel51.com/cross-embodiment): 235 contributors, 50 robot types, ~900 GB. Hugging Face's dataset card lists 46 robot types across 791 cleaned datasets. The 50 here come from scanning `robot_type` in all 1,755 contributor folders, so the two counts cover different sets of datasets. It's organized as 1,755 contributor datasets, one per folder, so picking a balanced slice starts with reading each one's metadata. If you're training a policy that has to generalize across robots, you need episodes from every embodiment and a way to see which clips are actually worth keeping.

Now that [FiftyOne natively supports LeRobot](https://voxel51.com/blog/lerobot-dataset-support-voxel51), I figured I’d use [the FiftyOne skill to import LeRobot datasets](https://github.com/voxel51/fiftyone-skills/blob/main/skills/fiftyone-dataset-import/LEROBOT-IMPORT.md) and check it out. All it took was a one-sentence prompt:

*"i want to have a little bit from every embodiment, i have enough disk space for the whole dataset... but let's take 10 episodes from each embodiment to start"*

That sentence kicked off a session that read every contributor dataset's metadata, found 50 unique robot types, downloaded the right shard files for each, imported 497 episodes into FiftyOne, embedded every clip with Qwen3-VL, computed a similarity index, and saved 75 queryable views. All without writing a data loader, a Parquet parser, or a video decode loop.

One sentence was enough because the agent had [FiftyOne Skills](https://github.com/voxel51/fiftyone-skills/tree/main) installed. Skills are [packaged workflows that teach AI assistants](https://voxel51.com/blog/fiftyone-agent-skills-mcp-tools) how to do complex FiftyOne tasks: [importing LeRobot episodes](https://voxel51.com/blog/lerobot-import-fiftyone-agent-skills), running zoo models, computing embeddings, finding duplicates. The agent didn't have to guess at the LeRobot v3 layout or the FiftyOne API. It followed the skill. Install them in Cursor, Claude Code, Codex, or Gemini CLI with one command:

```python
curl -sL skil.sh | sh -s -- voxel51/fiftyone-skills
```

This is what FiftyOne is for. Not visualization. Dataset curation at the scale that [robot learning](https://voxel51.com/robotics) actually requires.

**New here?** [FiftyOne](https://github.com/voxel51/fiftyone) is an open-source toolkit for loading, querying, visualizing, and curating ML datasets — images, videos, and multimodal data. [LeRobot](https://github.com/huggingface/lerobot) is Hugging Face's framework for robot learning: it defines a [standard v3 dataset format](https://voxel51.com/lerobot) (Parquet + MP4 shards with episode metadata) that the community uses to share teleoperation recordings.

## Key takeaways

- `lerobot/community_dataset_v3` is a collection of 1,755 contributor datasets across 50 robot embodiments, each a [complete LeRobot v3 dataset](https://voxel51.com/blog/lerobot-v3-datasets-vla-training-fiftyone) with its own `meta/info.json`.
- Using the [FiftyOne Skills dataset-import workflow](https://github.com/voxel51/fiftyone-skills/tree/main/skills/fiftyone-dataset-import), the agent scanned every `*/*/meta/info.json`, grouped by `robot_type`, picked the smallest source dataset per embodiment with ≥10 episodes, and downloaded only the needed shard-0 files.
- Each FiftyOne sample is one episode — not a frame, not a video file — with fields `robot_type`, `task`, `duration`, `fps`, `dataset_name`, and a `media_reference` pointer into the local LeRobot directory tree.
- `Qwen3-VL-Embedding-2B` turns each episode (up to 32 frames from its time window) into one 2048-dimensional video embedding. The similarity index, Uniform Manifold Approximation and Projection (UMAP), uniqueness, and representativeness all run from that one field in under a minute.
- The same model embeds text into the same space, so the index answers text queries: type "folding a cloth" and get the `same bimanual` cloth-folding episodes back, with no labels involved.
- `uniqueness` scores range from 0.013 to 1.0 across the 497 episodes. 292 clips score below 0.1 because each embodiment's 10 clips come from one recording session. The two highest scores, the only ones above 0.7, both belong to single-frame episodes (0.02–0.03 s long): uniqueness flags [broken recordings](https://voxel51.com/blog/robot-episode-quality-triage) as readily as rare behavior.
- 75 saved views — 50 by robot type plus analytical cuts by frames per second (FPS), duration, robot category, task keyword, and curation intent — are stored in the dataset and load from the App's Views dropdown with no code.

## What you need

**To load and query the pre-built dataset (no GPU required):**

```python
pip install "fiftyone>=1.22.0" "huggingface_hub[cli]" transformers qwen-vl-utils torch torchvision
```

**To re-run UMAP visualization yourself (Step 3):**

```python
pip install umap-learn
```

FiftyOne 1.22.0 is the first release with `fo.types.LeRobotDataset`. The dataset loads from the Hub with embeddings, similarity index, UMAP layout, and curation scores already computed. Steps 1–2, Step 4, and episode-to-episode search in Step 5 run entirely on the pre-built data, with no GPU or model weights. Text search in Step 5 downloads `Qwen3-VL-Embedding-2B` (about 4 GB) to embed your query; it runs on CPU and is faster on a GPU. Step 3 is for readers building their own dataset who want to replicate the embedding pipeline.

## How the LeRobot community dataset is organized

![FiftyOne App grid of LeRobot community dataset episodes with bar charts of robot type and an episode duration histogram, then one episode opened with its camera views and state and action values](https://image.mux.com/eI01iFQM5P7BZVGgeu3BcVmPSaSwGwEJPNml7bj3BD00Q/animated.gif?width=640)

*FiftyOne App grid of LeRobot community dataset episodes with bar charts of robot type and an episode duration histogram, then one episode opened with its camera views and state and action values*

`lerobot/community_dataset_v3` is a collection of contributor datasets. Each one is a complete LeRobot v3 dataset in its own `<contributor>/<dataset>/` folder:

```python
community_dataset_v3/
  kuehnrobin/
    g1_stack_cube_left/
      meta/info.json
      data/
      videos/
  villekuosmanen/
    pick_coffee_capsule_under_dome/
      meta/info.json
      ...
```

1,755 directories. 17,617 files. 897.5 GB total. Each sub-dataset carries its own `meta/info.json` with its robot type, episode count, and FPS.

`fo.types.LeRobotDataset` reads one LeRobot v3 dataset at a time, so the agent started by downloading all `*/*/meta/info.json` files — a 56-second fetch — and then scanning them to build the plan:

```python
# Counts across all 1,755 sub-datasets
Robot types (50 unique):
  so100                   899 datasets   34801 total_episodes
  so101                   312 datasets    7490 total_episodes
  so101_follower          121 datasets    4414 total_episodes
  so100_follower          120 datasets    3065 total_episodes
  arx5                     43 datasets    2108 total_episodes
  Unitree_G1_inspire        1 dataset        7 total_episodes  # only 7 available
  so100MovellaDot          75 datasets      157 total_episodes  # max 5 per source
  ...
```

All 1,755 sub-datasets were already at `codebase_version: v3.0`. No conversion needed.

![Diagram of the contributor and dataset folder layout, the four-step plan to scan, group by robot type, pick one source per embodiment, and download, and the shard-0 Parquet and MP4 files fetched from one source](https://cdn.sanity.io/images/h6toihm1/production/7d22b86502de41d8a77cf9f67bd13947e3c951b7-1999x908.png)

*Left: the contributor/dataset folder layout, with the per-dataset meta/info.json files the agent scans first. Middle: the four-step plan — scan, group by robot_type, pick one source dataset per embodiment, download. Right: inside one source dataset, only file-000.parquet and file-000.mp4 (the first ~10 episodes) are fetched; the other shards are skipped.*

**Good to know. What's a shard-0 set?** LeRobot v3 packs many episodes into shared MP4 and Parquet shards. `file-000.parquet` and `file-000.mp4` hold the first batch of episodes for each dataset. Downloading only those files makes a 10-episode subset cheap, even from a 50-GB source dataset.

## Step 1: load the dataset

The agent session did the heavy lifting — reading every contributor dataset's metadata, picking the right shard files, importing 497 episodes, computing embeddings, and pushing the result to the Hub. The import step is covered in [How to Import LeRobot Datasets into FiftyOne Using Agent Skills](https://voxel51.com/blog/lerobot-import-fiftyone-agent-skills).

The dataset is at [`Voxel51/community_v3_10per_embodiment`](https://huggingface.co/datasets/Voxel51/community_v3_10per_embodiment). Load it:

```python
import fiftyone as fo
import fiftyone.utils.huggingface as fouh

ds = fouh.load_from_hub("Voxel51/community_v3_10per_embodiment")

print(len(ds))                    # 497
print(ds.distinct("robot_type"))  # 50 embodiments
print(ds.get_field_schema())      # episode_index, robot_type, task, duration, fps,
                                  # dataset_name, media_reference,
                                  # qwen3vl_embedding, qwen3vl_embedding_camera,
                                  # uniqueness, representativeness, ...
```

From here on, the source videos live in the Hub repo alongside the metadata. FiftyOne reads through the `media_reference` pointer — nothing to download separately. And because the unit is an episode rather than a file, every query you run is a dataset-level question, not a file-browser search.

## What "one sample = one episode" actually means

![A single LeRobot episode opened in the FiftyOne App, showing synced top and wrist camera views, a joint state and action table, a joint position plot, and the sample's fields, including media_reference](https://image.mux.com/mLAub2AoBay9lWFeU00Lz2I13a8IQ7kMq6e2j19dUDTg/animated.gif?width=640)

*A single LeRobot episode opened in the FiftyOne App, showing synced top and wrist camera views, a joint state and action table, a joint position plot, and the sample's fields, including media_reference*

Most tools treat robot data as video files or frame sequences. FiftyOne takes a different unit: [the episode](https://voxel51.com/episode).

Every sample in this dataset carries:

```python
sample.episode_index   # int — 0..9 within the source dataset
sample.robot_type      # str — e.g. "arx5", "so100", "Unitree_G1_Dex3"
sample.dataset_name    # str — e.g. "villekuosmanen/pick_coffee_capsule_under_dome"
sample.task            # str — e.g. "Pick up the coffee capsule"
sample.duration        # float — seconds (0.02 to 148.6 in this dataset)
sample.fps             # float — 10, 15, 20, 25, 30, or 50
sample.media_reference # LeRobotEpisodeReference — a FiftyOne type with two fields:
                       #   .data   = [chunk_index, file_index, first_row, last_row]  (Parquet)
                       #   .videos = {camera: [chunk_index, file_index, from_ts, to_ts]}  (MP4)
```

Think of a FiftyOne sample as a row in a table: the left side has filterable scalar fields (`robot_type`, `task`, `duration`, `fps`) and the right side has the `media_reference` pointer. The pointer's two values — `data[2:4]` and `videos[top][2:4]` — map to the actual Parquet rows and MP4 time window in the LeRobot folder on disk.

![Diagram of a FiftyOne sample with filterable scalar fields above a media_reference pointer whose data and videos values map to Parquet rows and an MP4 time window on disk](https://cdn.sanity.io/images/h6toihm1/production/8402f3457feeb61226788b88d420798ece94c6a4-1787x914.png)

*A FiftyOne sample (left) with filterable scalars above the line and the media_reference pointer below it. Arrows connect the pointer's two values — data[2:4] and videos[top][2:4] — to the actual Parquet rows and MP4 time window in the LeRobot folder on disk.*

Three roads lead from a FiftyOne view back to frame data:

![Diagram of three ways to get frames from a FiftyOne view: read in place and Torch dataset, which copy no data, and export to a standalone LeRobot v3 dataset, which copies data](https://cdn.sanity.io/images/h6toihm1/production/e444b7797bcf6db1098f3c8808533f915492913e-1752x993.png)

1. **Read in place** via `sample.media_reference.data` — no copy, your code, one episode at a time.
1. **Torch dataset** via `view.values("episode_index")` passed to `LeRobotDataset(..., episodes=...)` — no copy, for training loops. `episode_index` restarts at 0 in every source dataset, so filter the view to one `dataset_name` first; for a mix of sources, export instead
1. **Export** via `view.export(dataset_type=fo.types.LeRobotDataset)` — copies data, produces a standalone v3 repo. Each export must come from a single source dataset, so filter by `dataset_name` first.

The rule: in-process → don't copy. Out-of-process → copy.

| **Path** | **Call** | **Copies data?** | **Use it when** |
| --- | --- | --- | --- |
| Read in place | `sample.media_reference.data` | No | Your own code needs one episode's rows, such as for a plot, a metric, or a policy |
| Torch dataset | `LeRobotDataset(..., episodes=view.values("episode_index"))` | No | A training loop reads batches from a curated subset filtered to one `dataset_name` |
| Export | `view.export(dataset_type=fo.types.LeRobotDataset)` | Yes | Something outside the Python process needs a standalone LeRobot v3 repo, one source per export |

*Three ways to get frames from a FiftyOne view of LeRobot episodes, and when each one copies data*

## From inspection to curation

![Filtering LeRobot episodes by task in the FiftyOne App sidebar and lasso-selecting points in the Embeddings panel to show the matching episodes in the grid](https://image.mux.com/G7JNF68bXWMIA021fWsoTJ4gUTCneoCMklNXLDyyYofQ/animated.gif?width=640)

*Filtering LeRobot episodes by task in the FiftyOne App sidebar and lasso-selecting points in the Embeddings panel to show the matching episodes in the grid*

Timeline viewers are built for inspecting one recording at a time: open a log, scrub the timeline, and watch camera, inertial measurement unit (IMU), and joint streams in sync. That's the right tool when you're debugging one specific run.

FiftyOne starts where that ends. Once a run is recorded and you have hundreds of episodes, the question isn't "what happened in this episode?" It's "across all 497 episodes from 50 different robots, which clips are visually unique, which are near-duplicates, which ones actually belong in my training set?"

A timeline scrubber can't answer that. A queryable, indexed dataset can.

The rest of this post shows exactly what that looks like — every step from here on is something you can't do in a timeline scrubber.

## Step 2: what's already on the dataset

The Hub dataset ships with a video embedding for every episode, computed with `Qwen/Qwen3-VL-Embedding-2B` from the [`qwen3vl_embeddings`](https://github.com/harpreetsahota204/qwen3vl_embeddings) remote zoo source. Each episode's time window is decoded with PyAV; up to 32 evenly spaced frames go to the model as one video clip, producing a 2048-dimensional `qwen3vl_embedding` field. `qwen3vl_embedding_camera` records which camera stream the frames came from, since camera names and layouts differ across the 50 embodiments.

### What Qwen3-VL-Embedding actually produces, and why it matters

[Qwen3-VL](https://huggingface.co/collections/Qwen/qwen3-vl) is an open vision-language model family from Qwen/Alibaba. It comes in two variants: `Qwen3-VL-Instruct` (generates text — captions, bounding boxes, answers) and `Qwen3-VL-Embedding` (trained for retrieval, so it produces dense vectors directly). The embedding model reads the clip, and the hidden state of the last token becomes the vector, L2-normalized:

```python
# what produced qwen3vl_embedding (inside the qwen3vl_embeddings zoo model)
outputs = model(**inputs)                                        # frames + instruction
embedding = outputs.last_hidden_state[row, last_token_position]  # one 2048-d vector per clip
embedding = F.normalize(embedding, p=2, dim=-1)
```

![Two-lane diagram in which frames decoded with PyAV pass through Qwen3-VL-Embedding into one 2048-dimensional vector, and a text query passes through the same model into the same vector space](https://cdn.sanity.io/images/h6toihm1/production/e6035051f36efbec6353b0d06db21e7ef5f73ded-1901x967.png)

*Top lane: an episode's time window is decoded with PyAV, up to 32 evenly spaced frames are tokenized by the vision encoder and run through the language model, and the last token's hidden state becomes one 2048-d vector stored as qwen3vl_embedding. Bottom lane: at query time, a text query goes through the same model and lands in the same space, which is what makes text-to-video search work.*

This model is useful for robotics data because **video and text map into the same vector space.** The similarity index `qwen3vl_embedding_sim` keeps the model name, so it accepts text queries as well as episodes: `ds.sort_by_similarity("folding a cloth", k=10, brain_key="qwen3vl_embedding_sim")` — no labels, no metadata matching, just the model's understanding of what was in the clip.

For [Vision-Language-Action (VLA) model](https://voxel51.com/vision-language-action-model-vla) development, this is specifically useful at the [**data curation stage**](https://voxel51.com/blog/fiftyone-supports-multimodal-data). A VLA policy needs to generalize across scenes, embodiments, and task variations. But you can only curate what you can measure. Video embeddings give you a numeric handle on task semantics and visual appearance simultaneously, which makes it possible to compute:

- **Similarity** — which episodes, across all 50 embodiments, are doing visually similar things?
- **Uniqueness** — which demonstrations cover behavior your dataset hasn't seen much of?
- **Representativeness** — which episodes best stand in for entire clusters of similar behavior?

You can't extract these signals from episode metadata alone. They require a model that actually watched the clips.

You also get `uniqueness` and `representativeness` scores out of the box. More on those below.

If you want to try a different model, such as `Qwen/Qwen3-VL-Embedding-8B`, write its vectors to a new field; Step 5 shows the full embedding loop for LeRobot episodes.

## Step 3: Compute similarity, uniqueness, and representativeness

![FiftyOne App grid of LeRobot episodes tagged with uniqueness and representativeness scores, next to the Embeddings panel and a score histogram](https://image.mux.com/rpjHY1Jdxy5I1aZeQsUtyQxjZTxcEgvlNqOwztrtyis/animated.gif?width=640)

*FiftyOne App grid of LeRobot episodes tagged with uniqueness and representativeness scores, next to the Embeddings panel and a score histogram*

**Skip the code if you loaded from the Hub.** The pre-built dataset already includes `qwen3vl_embedding_sim`, `qwen3vl_embedding_umap`, `uniqueness`, and `representativeness`. The code block below is for readers building their own LeRobot dataset — it requires umap-learn and an existing embedding field. The explanations after it apply either way.

If you're building your own dataset, compute the `qwen3vl_embedding` field first (the loop is in Step 5), then run the four calls:

```python
import fiftyone.brain as fob

EMBED = "qwen3vl_embedding"

# Similarity index — powers Sort by Similarity in the App. Passing the model name
# lets the index embed text queries (register the qwen3vl_embeddings source first)
fob.compute_similarity(
    ds, embeddings=EMBED, model="Qwen/Qwen3-VL-Embedding-2B",
    brain_key="qwen3vl_embedding_sim",
)

# UMAP — 2D layout for the Embeddings panel
fob.compute_visualization(
    ds, embeddings=EMBED, method="umap",
    brain_key="qwen3vl_embedding_umap", num_dims=2, seed=51,
)

# Uniqueness — how different each episode is from its nearest neighbors
fob.compute_uniqueness(
    ds, embeddings=EMBED,
    uniqueness_field="uniqueness",
    similarity_index="qwen3vl_embedding_sim",
)

# Representativeness — how close each episode is to a cluster center
fob.compute_representativeness(
    ds, embeddings=EMBED,
    representativeness_field="representativeness",
    similarity_index="qwen3vl_embedding_sim",
)
```

### Why UMAP for video embeddings?

Each episode is a 2048-dimensional vector. You can't look at that. Dimensionality reduction maps it to 2D so you can see structure in the Embeddings panel.

The three common choices are principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE), and UMAP:

- **PCA** is linear. It finds the axes of maximum variance, which is fast but can't capture the curved manifold structure that emerges when you embed semantically diverse video. A pick-and-place cluster and a folding cluster may sit on a low-variance direction PCA ignores entirely.
- **t-SNE** is non-linear and good at revealing local clusters, but it destroys global structure — the *distance between* clusters in the 2D plot is meaningless, and it's slow on datasets larger than a few thousand samples.
- **UMAP** is non-linear, keeps local cluster structure, and usually keeps more of the arrangement *between* clusters than t-SNE does. Don't over-read it, though: distances between far-apart clusters are still only roughly meaningful. It runs on 500 episodes in seconds. FiftyOne's UMAP backend uses Euclidean distance with `min_dist=0.1` by default, which gives tight clusters.

![Three scatter plots of the same 497 Qwen3-VL-Embedding vectors reduced with PCA, t-SNE, and UMAP, colored by robot category](https://cdn.sanity.io/images/h6toihm1/production/1bb3dba307a6e86ab87e24e760b6e88eaa2b6385-1999x830.png)

*The same 497 Qwen3-VL-Embedding vectors reduced three ways, colored by robot category. PCA's two axes hold 20% of the variance, and the categories overlap; t-SNE separates local knots but inter-knot distances are arbitrary; UMAP is the layout the Embeddings panel shows. In every map, the 10 clips from one source dataset collapse into a single knot—same scene, same camera, same task.*

For a first look at a cross-embodiment dataset, UMAP is the practical choice: you get clusters you can click into, and their rough arrangement is meaningful.

The map also shows what the embeddings respond to most: the recording session. For 495 of the 497 episodes, the nearest neighbor comes from the same source dataset. Step past a clip's own siblings, though, and the next matches in the full 2048-d space often share the task on a different robot. The nearest non-sibling to the `so100` box-in-bin episodes is a `so100MovellaDot` ball-in-box episode (cosine distance 0.157), and the nearest non-sibling to the `arx5` dome-lifting episodes is a `trossen_ai_stationary` cube-in-box episode (0.170).

### What uniqueness and representativeness actually compute

Both metrics are grounded in the embedding space. The FiftyOne Brain docs cover [`uniqueness`](https://docs.voxel51.com/brain/index.html#image-uniqueness) and [`representativeness`](https://docs.voxel51.com/brain/index.html#image-representativeness); here's what each one computes:

**Uniqueness** — K-nearest-neighbor distance, K=3. For each episode, FiftyOne finds its 3 nearest neighbors in embedding space and computes a weighted mean of those distances:

```
uniqueness = weighted_mean(distances_to_3_nearest_neighbors)
             weights = [0.6, 0.3, 0.1]   # closer neighbors count more
             normalized to [0, 1] by dividing by the dataset maximum
```

Interpretation: `uniqueness=1.0` means this episode is farther from its nearest neighbors than any other episode in the set — it occupies a region of embedding space that no other clip is near. `uniqueness=0.013` means the episode is almost on top of another one — effectively a near-duplicate.

![Toy embedding space where an isolated episode scores 1.0 and a clustered one scores about 0.01 on uniqueness, beside a log-scale histogram of uniqueness across 497 LeRobot episodes](https://cdn.sanity.io/images/h6toihm1/production/ae48056d027013ff6bdc1e346c22177978c8fd0d-1999x861.png)

*Left: a toy embedding space. Episode A is far from its three nearest neighbors (line width = weight), so it scores 1.0; episode B sits in a knot of near-identical clips and scores ≈ 0.01. Right: the real distribution over all 497 episodes on a log scale — 292 score below 0.1, two score above 0.7, and one arx5 clip (a single-frame episode) sets the maximum at 1.0.*

**Representativeness** — K-Means distance to cluster center, K=20 clusters. For each episode, FiftyOne finds the nearest cluster center and scores it as:

```
representativeness = 1 / (1 + distance_to_nearest_cluster_center)
                     normalized locally within each cluster
```

Interpretation: `representativeness=1.0` means this episode is at the heart of its cluster — it's the most "typical" example of that group of behaviors. Low representativeness means the episode sits at the edge of a cluster, more idiosyncratic than central.

![K-means cluster centers with equal-distance rings, where a central episode scores near 1.0 and a peripheral one near 0.3, beside the representativeness distribution across 497 episodes](https://cdn.sanity.io/images/h6toihm1/production/a41e89583b647fe8c7340b819c92d66b9b59f35f-1999x861.png)

*Left: K-Means centers (×) with rings of equal distance. Episode P sits on its center and scores ≈ 1.0; episode Q is assigned to the blue cluster but sits far out and scores ≈ 0.3. Right: the real distribution — 198 of 497 episodes score above 0.8, and the floor of 0.33 means no episode is a pure outlier.*

### Why these two metrics are the right pair for VLA data curation

A Vision-Language-Action model needs two things from its training data:

1. **Coverage** — enough behavioral diversity for the policy to generalize to new scenes and task variations. Low-uniqueness episodes (near-duplicates) don't add coverage; they just add training compute and risk overfitting to that one scene configuration.
1. **Signal density** — enough repetition of each important behavior for the policy to learn it. Keeping only high-uniqueness episodes would discard the repetitions that make a skill learnable.

Uniqueness and representativeness together let you navigate that tradeoff explicitly:

| Goal | Query |
| --- | --- |
| Seed set: diverse AND learnable | High uniqueness + high representativeness |
| Redundancy audit: what can be cut? | Low uniqueness floor (near-duplicates) |
| Canonical demos per task | Sort by representativeness within each task view |
| Rare demonstrations (or broken ones) | Uniqueness > 0.7, then check duration |

*Four VLA data curation goals and the uniqueness and representativeness queries that answer each one.*

![Scatter plot of uniqueness against representativeness for every LeRobot episode, colored by robot category, with shaded query regions and 50 ringed points marking curate_diverse_seeds](https://cdn.sanity.io/images/h6toihm1/production/a30a824556bd3fb478e6b2d21e3700878836c3cd-1999x1054.png)

*Every episode is placed by its two scores (uniqueness on a symlog axis, colored by robot category). The shaded regions are the queries from the table: the redundancy audit on the left, canonical demos along the top, high-uniqueness outliers on the right, and the 50 orange-ringed points are curate_diverse_seeds — the top-50 by uniqueness with representativeness above 0.5.*

In this dataset, `uniqueness` ranges from 0.013 to 1.0. The floor shows near-identical clips in the set — 292 of 497 score below 0.1, and the bottom six are four `moss` cube-stacking episodes and two `sam_bimanual` cloth-folding episodes, each from a single recording session. At the top, only two episodes score above 0.7: an `arx5` clip at 1.0 and an `xarm6` clip at 0.86. Both are single-frame episodes, 0.02–0.03 seconds long. `representativeness` ranges from 0.33 → 1.0 — no episode is a pure outlier; even the most peripheral clips are loosely connected to a cluster.

**Keep it honest.** Both ends of the uniqueness ranking come from how we built this subset, not from rare behavior or wasted data. Each embodiment contributes 10 consecutive episodes from one recording session, so every clip has nine siblings sitting right next to it in embedding space. On this dataset, every one of the 50 episodes in `curate_redundant_candidates` has a sibling as its nearest neighbor, sampled on purpose. The top of the ranking has the opposite story: the five single-frame episodes look unlike everything else because there's almost nothing in them. They take the top two spots, and all five land in `curate_diverse_seeds`. Uniqueness did its job by surfacing recordings worth a second look. Those clips belong in the cut pile, though, not the seed set, so pair uniqueness with a duration filter.

These two numbers let you make dataset decisions — what to keep, what to cut, what's overrepresented — without watching a single video. A log viewer shows you one episode at a time. FiftyOne gives you a signal across all 497 at once.

## Step 4: Save views that answer real questions

![Switching between the category_single_arm and category_bimanual saved views from the Views dropdown in the FiftyOne App](https://image.mux.com/1KsDYAzEH102JfWRMq5cplQC02MpivSrEZ1lfv7P502WT4/animated.gif?width=640)

*Switching between the category_single_arm and category_bimanual saved views from the Views dropdown in the FiftyOne App*

Views in FiftyOne are lazy queries — no data is copied, they resolve on load. Once saved, they appear in the App's Views dropdown.

The pre-built dataset already has 75 views saved. Check what's there:

```python
import fiftyone as fo

print(ds.list_saved_views())
# ['so100', 'so101', ..., 'curate_diverse_seeds', 'curate_redundant_candidates',
#  'category_humanoid', 'task_pick_place', 'fps_30', ...]
```

To build them yourself on your own dataset:

```python
import fiftyone as fo

# Curation: 50 most visually distinct AND cluster-representative episodes
ds.save_view(
    "curate_diverse_seeds",
    ds.sort_by("uniqueness", reverse=True)
      .match(fo.ViewField("representativeness") > 0.5)
      .limit(50),
)

# Redundancy candidates: bottom 50 by uniqueness
ds.save_view(
    "curate_redundant_candidates",
    ds.sort_by("uniqueness").limit(50),
)

# Robot category views
HUMANOID = ["Unitree_G1_Dex3", "Unitree_G1_Gripper",
            "Unitree_G1_Inspire", "Unitree_G1_inspire", "g1"]
ds.save_view("category_humanoid",
    ds.match(fo.ViewField("robot_type").is_in(HUMANOID)))

# Task keyword views
ds.save_view("task_pick_place",
    ds.match(fo.ViewField("task").re_match("pick|place|grasp|grab", options="i")))

# FPS views — reveal recording-condition variation across embodiments
for fps_val in [10.0, 15.0, 20.0, 25.0, 30.0, 50.0]:
    view = ds.match(fo.ViewField("fps") == fps_val)
    if len(view) > 0:
        ds.save_view(f"fps_{int(fps_val)}", view)
```

75 views total. fps_30 covers 387 of the 497 episodes. `fps_10` covers 30 — all from three stationary arms: `widowx`, `ur5e_gello`, and `mycobot320`. `task_pick_place` returns 367 episodes; `task_fold_pour` returns 70.

Each is a reusable, shareable slice of the dataset that resolves in milliseconds. No script, no re-download, no custom data loader — just a name in a dropdown.

## Step 5: similarity search by episode and by text

![Running a text similarity search for "robot arm picking up lego pieces" against the qwen3vl_embedding_sim index in the FiftyOne App's Similarity Search panel](https://image.mux.com/JlCOm00OS6Cz4KipzCrlNuv02lg01RLDJ7sqCEWeNdURxQ/animated.gif?width=640)

*Running a text similarity search for "robot arm picking up lego pieces" against the qwen3vl_embedding_sim index in the FiftyOne App's Similarity Search panel*

### Search by episode

The `qwen3vl_embedding_sim` index powers episode-to-episode search directly from the App or from Python. Pass any sample's ID, and FiftyOne returns the nearest neighbors by video embedding distance. Because every clip's closest matches are its own recording session, filter out the query's `dataset_name` to see what other robots are doing something similar:

```python
from fiftyone import ViewField as F

# one sam_bimanual cloth-folding episode as the query
query = ds.match(
    (F("dataset_name") == "girardijp/sam_fold_cloth_single")
    & (F("episode_index") == 0)
).first()

# unfiltered: the query and its 9 siblings from the same session
same_session = ds.sort_by_similarity(query.id, k=10, brain_key="qwen3vl_embedding_sim")

# filtered: the 10 nearest episodes from other source datasets
others = (
    ds.sort_by_similarity(query.id, k=100, brain_key="qwen3vl_embedding_sim")
    .match(F("dataset_name") != query.dataset_name)
    .limit(10)
)
for s in others:
    print(f"[{s.robot_type}] {s.task}")
# [so100_bimanual] playing with folding      (all 10 hits)
```

![A sam_bimanual cloth-folding query episode marked on the UMAP, beside the 10 nearest episodes from other source datasets ranked by cosine distance, all so100_bimanual folding clips](https://cdn.sanity.io/images/h6toihm1/production/e565d04ae7cba216591455e66d12b0921441f61e-1999x863.png)

*Left: the query (star) on the UMAP, with its session's knot and the cross-source neighbors. Right: the 10 nearest episodes from other source datasets, ranked by cosine distance. The query's 9 siblings all sit within 0.015; past them, every hit is a so100_bimanual "playing with folding" episode, 0.19–0.31 away.*

![Selecting LeRobot episodes in the FiftyOne App and running an image similarity search from the Similarity Search panel](https://image.mux.com/xqVi8dFer9jSn801701OqAwU4YgZE8nZOua9eowBGzGlM/animated.gif?width=640)

*Selecting LeRobot episodes in the FiftyOne App and running an image similarity search from the Similarity Search panel*

A `sam_bimanual` rig folding a cloth and a pair of SO-100 arms folding cloth look nothing alike as robots, and the model still puts them next to each other. That's the cross-embodiment signal. Some sessions pull in a neighboring robot type instead: the 10 `Unitree_G1_Dex3` cube-stacking clips sit within 0.030 of each other, and the first other episode, at rank 10, is a `Unitree_Z1_Dual` red-cup pick (0.142).

### Search by text

![Reopening saved similarity searches in the FiftyOne App, such as "robot arm picking up lego pieces" and "blue robotic arm," to reload their matching LeRobot episodes](https://image.mux.com/SGFzV6AlZ01rQ00R8ZPDD5uDXe8dYqFVSjT24zSQ1KnPY/animated.gif?width=640)

*Reopening saved similarity searches in the FiftyOne App, such as "robot arm picking up lego pieces" and "blue robotic arm," to reload their matching LeRobot episodes*

The same index takes text. The dataset stores the episode vectors, but FiftyOne still needs the model to embed your query, so register the [`qwen3vl_embeddings`](https://github.com/harpreetsahota204/qwen3vl_embeddings) remote zoo source first:

```python
import fiftyone.zoo as foz

# declare which models the remote source provides
foz.register_zoo_model_source(
    "https://github.com/harpreetsahota204/qwen3vl_embeddings",
    overwrite=True,
)

# optional: the first text query downloads the weights automatically
foz.download_zoo_model("Qwen/Qwen3-VL-Embedding-2B")

for query in ["folding a cloth", "pouring water into a cup", "humanoid robot picking up a can"]:
    view = ds.sort_by_similarity(query, k=3, brain_key="qwen3vl_embedding_sim")
    print(query, "->", [(s.robot_type, s.task) for s in view])
# summarised output:
# folding a cloth -> sam_bimanual, "Fold a cloth" (x3)
# pouring water into a cup -> bi_so100_follower, "bi soarm100 pour water" (x3)
# humanoid robot picking up a can -> Unitree_G1_Inspire, "Pick up the cans into the box on the table." (x3)
```

The same text box works in the App's similarity panel once `qwen3vl_embedding_sim` is selected. Not every query lands: "robot stacking a cube" returns a `Unitree_Z1_Single` red-cup pick rather than the `Unitree_G1_Dex3` cube-stacking clips. Treat [text search](https://voxel51.com/blog/fiftyone-search-multimodal-data) as a fast way to find candidates, then check them.

### Build the embedding and index on your own dataset

LeRobot samples have no `filepath`, so `compute_embeddings` can't read the clips for you: decode each episode's time window with PyAV and pass the frames to the model as one video. This is the code behind `qwen3vl_embedding` (set ROOT to the folder that holds your `<contributor>/<dataset>/ directories`):

```python
from pathlib import Path

import av
import fiftyone.brain as fob
import fiftyone.zoo as foz

ROOT = Path("/path/to/community_dataset_v3")


def episode_frames(video_path, from_ts, to_ts, max_frames=32):
    frames = []
    with av.open(str(video_path)) as container:
        stream = container.streams.video[0]
        fps = float(stream.average_rate or 30)
        step = max(1, int((to_ts - from_ts) * fps) // max_frames)
        container.seek(int(from_ts * 1_000_000), any_frame=False, backward=True)
        i = 0
        for frame in container.decode(stream):
            ts = float(frame.pts * stream.time_base)
            if ts < from_ts:
                continue
            if ts > to_ts or len(frames) >= max_frames:
                break
            if i % step == 0:
                frames.append(frame.to_image())
            i += 1
    return frames


foz.register_zoo_model_source(
    "https://github.com/harpreetsahota204/qwen3vl_embeddings", overwrite=True
)
model = foz.load_zoo_model("Qwen/Qwen3-VL-Embedding-2B")  # or Qwen/Qwen3-VL-Embedding-8B
if model._embedder is None:
    model._load_model()

for sample in ds.iter_samples(autosave=True, progress=True):
    cam, (chunk, file, from_ts, to_ts) = next(iter(sample.media_reference.videos.items()))
    path = ROOT / sample.dataset_name / "videos" / cam / f"chunk-{chunk:03d}" / f"file-{file:03d}.mp4"
    frames = episode_frames(path, from_ts, to_ts)
    # model.embed() expects a file path; the underlying embedder accepts a frame list as a video
    sample["qwen3vl_embedding"] = model._embedder.process([{"video": frames}])[0].cpu().float().numpy()
    sample["qwen3vl_embedding_camera"] = cam

fob.compute_similarity(
    ds,
    embeddings="qwen3vl_embedding",
    model="Qwen/Qwen3-VL-Embedding-2B",
    brain_key="qwen3vl_embedding_sim",
)
```

Passing the model name to `compute_similarity` lets the index embed text at query time; an index built without it accepts only episodes as queries. On a GPU, the 497 episodes take about 20 minutes. Then run the UMAP, uniqueness, and representativeness calls from Step 3.

## Putting it together: export a curated LeRobot v3 training split

Once loaded, querying across 50 embodiments looks like this:

```python
# 50 most unique AND representative episodes, minus the single-frame recordings
seed_view = ds.load_saved_view("curate_diverse_seeds").match(fo.ViewField("duration") > 1)

# Episodes from other robots that look like a given clip
query = ds.match(fo.ViewField("robot_type") == "sam_bimanual").first()
similar = (
    ds.sort_by_similarity(query.id, k=100, brain_key="qwen3vl_embedding_sim")
    .match(fo.ViewField("dataset_name") != query.dataset_name)
    .limit(10)
)

# Episodes matching a description (register the qwen3vl_embeddings source first)
cloth = ds.sort_by_similarity("folding a cloth", k=10, brain_key="qwen3vl_embedding_sim")

# Export the curated subset as standalone LeRobot v3 repos, one per source dataset
for name in seed_view.distinct("dataset_name"):
    seed_view.match(fo.ViewField("dataset_name") == name).export(
        export_dir=f"./my_training_subset/{name.replace('/', '__')}",
        dataset_type=fo.types.LeRobotDataset,
    )
```

The export loops over `dataset_name` because a LeRobot v3 repo has one feature schema, and the 50 embodiments here have different cameras and action dimensions. The 45 seed episodes come from 16 source datasets, so you get 16 repos.

That last step is the point. You went from a 900 GB community dataset to a curated, re-indexed LeRobot v3 training split — filtered by visual uniqueness, robot category, task type, or any combination — without writing a parser, a data loader, or a custom video decode loop. A viewer shows you what the robot did. A curated dataset decides what goes into the model.

## Frequently Asked Questions

### What does fo.types.LeRobotDataset actually store in FiftyOne's database?

Episode-level metadata only: `episode_index`, `task`, `duration`, `fps`, `robot_type`, and a `media_reference` that points into the local LeRobot directory. No pixel data, no Parquet rows. The source files stay where they are; FiftyOne reads through the pointer.

### Can I use apply_model on a LeRobot dataset in FiftyOne?

Not for a standard zoo model — those expect `sample.filepath` pointing to a video or image file, and LeRobot samples use `media_reference` instead. The embedding in this post was computed by working around that: PyAV extracts frames from the episode's time window, and the frames go to the model as a list of PIL images. Step 5 shows the full loop.

### Why does Qwen3-VL-Embedding use last-token pooling?

That's how `Qwen3-VL-Embedding` was trained. It's a decoder-only model with no CLS token, and its retrieval training teaches the final token's hidden state to summarize the whole input, whether it's a video clip or a sentence. Using the same pooling for both is what puts episodes and text queries in one space. Mean pooling over an instruction-tuned model's hidden states gives usable vectors for comparing clips, but not for matching them against text.

### How do saved views work after load_from_hub?

They're stored in the FiftyOne dataset document alongside the samples. `ds.list_saved_views()` returns all 75 names; `ds.load_saved_view("curate_diverse_seeds")` returns the 50 episodes matching that query, resolved against whatever embeddings and fields exist locally.

### How do I export a curated FiftyOne view to LeRobot v3 for policy training?

Export a curated view as standalone LeRobot v3 repos, one per source dataset:

```python
for name in view.distinct("dataset_name"):
    view.match(fo.ViewField("dataset_name") == name).export(
        export_dir=f"./my_curated_subset/{name.replace('/', '__')}",
        dataset_type=fo.types.LeRobotDataset,
    )
```

The exporter refuses to mix episodes from different sources in one repo, since each embodiment has its own cameras and action layout. For each source, it re-indexes episodes from 0, recomputes stats, and writes a self-contained v3 structure that any LeRobot-compatible trainer can read directly.

---

*This post first appeared as [the original article](https://voxel51.com/blog/lerobot-community-dataset-curation-fiftyone). I'm Harpreet Sahota, Hacker-in-Residence at [The Employable Data Scientist](https://employabledatascientist.com), where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the [Artists of Intelligence](https://employabledatascientist.com/podcast/) podcast. For new field notes by email, [subscribe on Substack](https://harpreetsahota.substack.com).*