Designed and built the multi-camera capture rig: DSLRs, RealSense, webcams and head-mounted cameras
Integrated ESP32 sensors and a uniform lighting setup for reproducible recordings
Built the video processing pipeline and helped speed up the AI pipeline
Coursera for Robots: Capturing multi-modal skill datasets for robotic foundation models
Building a dynamic catalogue of skills that robots can pull and do test-time training on during deployment. We capture student-teacher interaction datasets to teach robots how to learn skills and interact with humans, unlike existing data companies that focus on in-the-wild egocentric footage.
As a co-founder, I designed and built the hardware and video processing pipeline for multi-modal data capture. Created a reproducible recording setup using DSLRs, Intel RealSense, webcams, ESP32, head mounts, and uniform lighting systems. Also contributed to AI pipeline setup for faster processing.
7modalitiesUnique data types competitors miss
3labsPartner gait labs in India
MulticameraDSLRs, RealSense, Optitrack
HaWoR3D poseHand pose reconstruction
Problem Statement
Need: Robotic foundation models require high-quality, multi-modal training data to learn dexterous manipulation skills, but physical AI lacks internet-scale datasets
Prior art: Existing robotics data companies (Micro1, Scale, Surge, MicroAGI, BuildAI) focus primarily on in-the-wild egocentric footage, missing critical modalities like gripping force, temperature sensing, and structured student-teacher interactions
Our bet: cheap custom hardware that matches what a robot actually senses, paired with structured skill demonstrations, gives better post-training data for physical AI.
Methodology
The project follows a hardware-capture-process workflow for building scalable robotics training datasets.
Hardware Design & Integration
Designed a reproducible multi-camera capture system integrating professional motion capture with consumer-grade sensors. The setup is engineered to be portable and deployable across our partner gait labs in India.
StackComplete tech stack
Motion Capture: Optitrack for precise full-body pose tracking
Egocentric: Head-mounted dashcams for first-person perspective
Depth: Intel RealSense for RGB-D capture
Video: DSLRs and webcams for multi-angle coverage
Sensors: ESP32 for gripping force and temperature data
Processing: Claude Code for pipeline automation, HaWoR for 3D hand pose reconstruction
SensorsESP32 sensor integration
Gripping Force: Force-sensitive resistors integrated into tool handles for measuring grip pressure during manipulation tasks
Temperature: Thermocouples for capturing thermal data relevant to soldering and heat-sensitive tasks
Synchronization: ESP32 timestamps all sensor readings for alignment with video streams
Wireless: Data transmitted via WiFi to central recording system
Data Capture System
Capturing student-teacher interaction datasets to teach robots how to learn skills and interact with humans. Our unique approach focuses on structured instruction rather than in-the-wild egocentric footage.
Key7 unique data modalities
Gripping Force: Measuring grip pressure during manipulation tasks
Temperature Sensing: Thermal data for heat-sensitive tasks like soldering
Chain-of-thought Audio: Self-annotation through verbal explanation during tasks
Full Body Pose: Complete skeletal tracking via Optitrack
Exception Cases: Captured task failures for robust training
Multi-modal Instructions: PDFs, assembly videos paired with demonstrations
Egocentric Capture
Head-mounted dashcams capture first-person perspective essential for training robots on human visual perception during task execution. Uniform lighting systems ensure consistent image quality across all recording sessions.
Processing Pipeline
Established the video recording and processing pipeline for handling multi-view synchronized footage. Integrated HaWoR for 3D hand pose reconstruction and contributed to AI pipeline optimization for faster processing throughput.
PipelineProcessing workflow
Synchronization: Multi-view video streams aligned using audio sync and timestamp matching
Pose Estimation: HaWoR integration for accurate 3D hand pose reconstruction
Annotation: Chain-of-thought audio enables self-annotation of skill demonstrations
Automation: Claude Code for pipeline automation and quality checks
Results
ProgressCurrent achievements
Captured sample dataset covering soldering and dexterous skills
Operating with 3 gait labs in India for scalable data collection
Product ready: in conversation with potential customers
Reaching out to robotics companies for licensing contracts
DataMito Dex vs. Competitors
Mito Dex captures 7 unique modalities that competitors typically miss, focusing on the higher-value post-training data market.
Data Modality
Mito Dex
Competitors
Gripping Force
Yes
No
Temperature Sensing
Yes
No
Chain-of-thought Audio
Yes
No
Full Body Pose
Yes
Partial
Student-Teacher Interactions
Yes
No
Exception Cases (Failures)
Yes
No
Multi-modal Instructions
Yes
No
Egocentric Video
Yes
Yes
Depth Data
Yes
Yes
MarketCompetitive landscape
Larger competitors: Micro1, Scale, Surge
Smaller competitors: MicroAGI, BuildAI
YC companies in space: HumanArchive, Sensei, Cortex, Spatial
FutureFuture plans
Secure data licensing contracts with robotics companies
Scale data collection across more gait labs
Build custom decentralized wearable data capture device
Create standardized data capture protocol for franchise model
Retrain humans on newer skills using captured data
SkillsTechnical skills demonstrated
Hardware: Multi-camera system design, ESP32 embedded systems, sensor integration
I designed and built the hardware and video processing pipeline for multi-modal data capture. Created a reproducible recording setup using DSLRs, Intel RealSense, webcams, ESP32, head mounts, and uniform lighting systems. Contributed to AI pipeline setup for faster processing.
Co-founders
Vrishank Bhardwaj and Sanchit Jhunjhunwala handle business development, customer outreach, gait lab partnerships, and partial hardware/software pipeline work.
Founder relationships
Sanchit & Avva: 4 years, worked together at Sanchit's startup
Avva & Vrishank: 3 years, both consulted for Sanchit's startup
Additional Information
Demo video of the Mito Dex multi-modal data capture system.
WhyMotivation
Inspired by Westworld; we want to make robots in homes and industry common. Experience building Data Acquisition systems to create custom hardware for worthwhile datasets. Robotics researchers at leading labs are enthusiastic about such data.