Sparsh: Self-Supervised Touch Representations for Vision-Based Tactile Sensing
Carolina Higuera · Akash Sharma · Chaithanya Krishna Bodduluri · Taosha Fan · Patrick Lancaster · Mrinal Kalakrishnan · Michael Kaess · Byron Boots · Mustafa Mukadam · Mike Lambeta · Roberto Calandra
Most tactile models today are trained from scratch for one sensor and one task, which needs a lot of hand-labeled touch data that barely exists. Sparsh flips this: it learns what touch generally looks like by studying hundreds of thousands of unlabeled sensor frames, the same way large language models learn from raw text. The result is a reusable 'touch backbone', plug it into a new task like slip detection or force estimation and it already understands contact, so you need far fewer labels to get strong performance.
Labeled tactile data is the bottleneck for touch-driven robots. A pretrained backbone that transfers across sensors and tasks is the same leap that ImageNet and language pretraining were for their fields, it makes tactile perception practical for teams without giant in-house datasets.
- 01A self-supervised pretraining recipe (masked autoencoding and DINO-style objectives) tailored to tactile images.
- 02Pretrained encoders that transfer across DIGIT, GelSight, and other vision-based sensors.
- 03TacBench, a benchmark of six downstream tasks spanning force, slip, pose, grasp stability, and manipulation.
- 04Strong label efficiency, competitive results with as little as 1–5% of the labels end-to-end training needs.
- Scoped to vision-based (optical) tactile sensors; taxel/force-array sensors are not directly covered.
- Pretraining corpus skews toward the sensors and objects seen during collection, so exotic materials may still need adaptation.
- Downstream gains depend on a modest amount of task-specific labeled data for the readout head.
Sparsh introduces a family of general-purpose tactile representations pretrained with self-supervision on more than 460,000 unlabeled tactile images, so downstream tasks can be learned with a fraction of the labeled data.