HapTile: 1,699 UR5e Episodes With Tactile Video and Haptic Levels, in FiftyOne

In HapTile, the person teleoperating the robot feels the contact, and the dataset logs what they felt. A UR5e arm is teleoperated through contact-rich tabletop tasks, and the feedback level the operator received through the rig is logged as firm, mild or soft, next to the robot state and the video.

What is in it

HapTile comes from the authors listed on the source release and the paper "HapTile: A Haptic-Informed Vision-Tactile-Language-Action Dataset for Contact-Rich Imitation Learning" (arXiv:2606.04825). The Voxel51 version converts it to multimodal MCAP episodes. It has 1,699 episodes across 38 tasks, 664,045 frames and 12.36 hours.

Each gripper finger carries a vision-based tactile sensor. It films a gel pad printed with a marker grid, so contact shows up as the markers moving. Two RGB-D cameras watch the scene, one facing the table and one on the wrist.

Each episode has these channels:

The colour and tactile streams are 320x240, except in move_mobile_box, whose 47 episodes record them at 640x480. Episodes range from 12 to 1,838 frames and from 0.8 to 122.8 seconds, and the average frame rate per episode runs from 11.6 to 18.7. The recording clock is the source's own wall-clock stamps, so the frame rate varies the way the recording did.

Scrub tactile against haptic

Load it and launch the App:

import fiftyone as fo
import fiftyone.utils.huggingface as fouh

dataset = fouh.load_from_hub(
    "Voxel51/HapTile",
    name="HapTile",
    persistent=True,
)
session = fo.launch_app(dataset)

Open an episode and play it. Both tactile streams, both cameras and the plots share one timeline, so you can find the frame where the markers start to move and check which haptic level the operator was receiving at that moment.

The dataset has no FiftyOne label fields. Everything that varies inside an episode lives in the MCAP channels. What you can filter on are the per-episode fields: task, instruction, num_frames, end_effector_path_m, peak_marker_motion_left, peak_marker_motion_right and haptic_feedback. To pull the firm episodes of one task and rank them by the strongest marker motion on the left fingertip:

from fiftyone import ViewField as F

view = (
    dataset
    .match(F("task") == "insert_peg")
    .match(F("haptic_feedback").contains("firm"))
    .sort_by("peak_marker_motion_left", reverse=True)
)
session.view = view

haptic_feedback is a list, so an episode can carry more than one level. Across episodes, firm appears in 1,481, mild in 414 and soft in 575, and those counts overlap for that reason.

Tasks and instructions

Episodes are grouped by task under data/<task>/. The tasks are things like fold_tshirt, stack_glass_cups, press_coffee_machine and remove_sticky_note, and each episode carries an instruction such as "insert the peg in the red hole". There are 81 distinct instruction strings. 19 tasks have several variants, which differ in the target object, colour or location, so the language signal is more than a task label.

The size spread is wide. insert_peg has 50 episodes but the most frames (40,656, or 45.2 minutes). move_plush_toy has the most episodes at 70. move_toy_car has 30 episodes and 7,806 frames, or 8.7 minutes.

Known limits

Depth is not metric. The depth streams are 8-bit renderings carried as video, and the card marks metric depth estimation as out of scope.

Haptic coverage is uneven. Of the 107 episodes with an empty haptic_feedback, the 47 move_mobile_box episodes have no haptic channel at all and the other 60 have haptic_feedback_frames of 0. If you train on haptic levels, keep those two groups apart: one has no signal, the other has a signal that stayed silent. The release also records the haptic channel in three ways. Most episodes carry the feedback level with the summed marker motion of both fingertips, some carry the larger of the two fingertips instead, and move_mobile_box carries none. The conversion maps them onto /tactile.plot as sum or max, plus normalized.

The conversion makes some changes you should know about. Every stream is cut to the shortest where the source's lengths disagree, which drops one trailing frame in 40 episodes. One folder with an empty trajectory is left out. The source's activated flag is set on every frame and its 30-channel touch array is zero throughout, so neither is carried. Task names are lower-cased.

The card leaves the curation rationale, the teleoperator details and how the instruction texts were written as "More Information Needed", so this note cannot say.

For how gel sensors, force vectors and taxel grids differ across tactile data, see Robots Can See. They Still Can't Feel. For another gel-fingertip arm dataset in FiftyOne, see DreamTac.

The source release is CC BY 4.0, and the conversion uses the same license. The dataset is on Hugging Face at Voxel51/HapTile. For more of my FiftyOne dataset notes, see my Voxel51 author page, and to curate your own robot data in the same tool, try Voxel51.

Frequently asked questions

What is in the HapTile dataset?

1,699 teleoperated episodes across 38 tasks, 664,045 frames and 12.36 hours, recorded on a UR5e arm with a Robotiq 2F-85 gripper. Each episode has two RGB views, two fingertip tactile videos, two depth renderings, robot state, a haptic feedback stream and a task instruction.

What is haptic feedback in HapTile?

It is the feedback level the operator felt through the teleoperation rig, logged as firm, mild or soft. In 1,592 episodes the operator felt it at some point, and the level is written to the episode each time it changes.

How do I load HapTile in FiftyOne?

Call load_from_hub with Voxel51/HapTile and persistent=True, then launch the App. Each sample is one episode stored as a multimodal MCAP file, and the App's multimodal viewer plays every channel on one timeline.

Can I use HapTile for metric depth estimation?

No. The depth streams are 8-bit renderings rather than metric depth, and the card lists metric depth estimation as out of scope. The dataset is meant for imitation learning from vision, tactile, haptic and language signals.

Do all HapTile episodes have haptic feedback?

No. 107 episodes have an empty haptic_feedback field. The 47 move_mobile_box episodes have no haptic channel at all, and the other 60 have a haptic frame count of 0.

This post first appeared as the original article. I'm Harpreet Sahota, Hacker-in-Residence at The Employable Data Scientist, where we help AI and developer-tool teams ship prototypes, evaluations, and technical content. I also host the Artists of Intelligence podcast. For new field notes by email, subscribe on Substack.