¶ EgoUse — Hand–Object Interaction Timelines
EgoUse detects when a wearer is interacting with each object in egocentric Aria / HOT3D RGB video, and outputs a per-object interaction timeline (swimlane view). The project uses privileged teacher distillation: a rule-based teacher built from HOT3D 3D ground truth produces training labels; a deploy-time student runs only on RGB plus off-the-shelf detectors.
Dataset: HOT3D Aria recordings — 122 train sequences, 10 held-out prove sequences, plus mentor clip P0003_c701bd11 (only sequence with human-annotated interaction segments).
Final student checkpoint: phase_d_r7 (checkpoints/phase_d_r7/best.pt).
¶ System Flow
At a high level:
- Video — egocentric RGB from Project Aria (HOT3D) or a new MP4 at deploy time.
- Detectors — 100DOH (
handobj_100K+ego) for hands; YOLOE (open-vocabulary) for objects, with text prompts per HOT3D object name. - Pairwise spatial & motion features — per (hand, object) frame: IoU, normalized center distance, area ratio, detector scores, contact/offset, relative speed, rolling distance statistics (k≈15), and object presence fraction. No ground-truth boxes or poses enter student features.
- GRU — 2-layer GRU over pooled left/right hand streams; multi-label head over ~34 object classes. Trained on teacher
y*_cleanonly. - Object use timeline — binary ON/OFF per object, visualized as horizontal swimlanes (merged L∪R hands).
The image_2d baseline skips the GRU and applies the same dual-dwell scoring rules as the teacher directly on detector boxes — a fair detector-only comparison.
¶ Teacher (Label Generator)
The teacher is not deployed at runtime. It is a frozen pseudo-label generator:
| Item | Detail |
|---|---|
| Preset | teacher_aabb_stationary_v22 |
| Motion frame | HOT3D world-frame 3D velocities (oracle ego compensation) |
| Rule | Dual dwell: spatial IoU gate + co-motion gate, with independent gap-tolerant clocks |
| Cleanup | Lone-short removal + morphological smoothing → pair_scores_clean.csv |
| P0003 vs human GT | Frame F1 0.79, event seg_f1 0.67 (reference quality) |
Stationary interactions (keyboard, mouse) remain a known weakness of the co-motion rule; the teacher is presented as a validated label source, not perfect ground truth.
Output:
 |
|---|
| Human-made groundtruth (Filled color) vs. Teacher (Dashed boxes) on a training sequence |
¶ Student Model
| Item | Detail |
|---|---|
| Architecture | Slot encoder → 2-layer causal GRU (hidden 128) → per-timestep multi-class head |
| Input | 14-D features per (hand, object); left and right streams merged (L∪R) at inference |
| Training | ~122 HOT3D sequences; labels = teacher y*_clean; no GT poses/boxes in features |
| Inference | src/run_student_pipeline.py — supports HOT3D sequences or --video-path MP4 |
Output:
 |
|---|
| Student (Filled color) vs. Teacher (Dashed boxes) on a training sequence |
 |
| Student (Filled color) vs. Teacher (Dashed boxes) on a validation sequence |
¶ Evaluation
All reported numbers compare methods against frozen teacher labels on held-out data, unless noted as vs human GT on P0003_c701bd11.
| Metric | Definition |
|---|---|
| Frame F1 | Pair-level ON/OFF agreement per frame |
| Event F1 (seg_f1) | Segment-level F1 on interaction intervals at tIoU ≥ 0.5 |
| Event precision (seg_prec) | Segment precision at tIoU ≥ 0.5 |
Fair comparison note: the image_2d baseline uses detector boxes (100DOH + YOLOE), not HOT3D GT box2d_*.
¶ Results
¶ Held-out prove set (n = 10) — vs teacher
| Method | Frame F1 | Event F1 (tIoU ≥ 0.5) |
|---|---|---|
| 2D baseline (detector dual-dwell) | 0.146 | 0.061 |
GRU student (phase_d_r7) |
0.163 | 0.061 |
On held-out clips, the GRU student shows a modest frame-level improvement over the 2D baseline (0.146 → 0.163). Event F1 stays low (0.061) for both methods — temporal segmentation and boundary accuracy remain the main bottleneck.
¶ Mentor clip P0003_c701bd11 — additional context
This is the only sequence with human-annotated interaction segments.
vs teacher:
| Method | Frame F1 | Event F1 | Event prec |
|---|---|---|---|
| 2D baseline | 0.393 | 0.350 | 0.359 |
| GRU student | 0.311 | 0.111 | 0.074 |
vs human GT:
| Method | Frame F1 | Event F1 | Event prec |
|---|---|---|---|
| 2D baseline | 0.566 | 0.282 | 0.308 |
| GRU student | 0.783 | 0.108 | 0.074 |
| Teacher (reference) | 0.798 | 0.667 | 0.707 |
On P0003, the GRU improves frame agreement with human GT over the 2D baseline, but event-level scores stay weak relative to the teacher — the student is learning coarse ON/OFF structure more than precise event boundaries.
Honest scope: with ~122 training sequences and teacher-only supervision, results show emerging HOI structure, not state-of-the-art precision. Prove-set teacher agreement remains low (~seg_f1 0.05–0.06); qualitative clips look like “starting to learn.”
¶ Speed
End-to-end timing is dominated by detectors, not the GRU student. Measured on an RTX 4060 (40 s demo clip, 600 frames @ 15 fps):
| Stage | Time (40 s clip) |
|---|---|
| Frame extraction | ~7 s |
| 100DOH hand detection | ~72 s |
| YOLOE object detection | ~12 s |
| Feature build + GRU | < 1 s |
| Total → predictions | ~1.5 min |
Rough guide: about 2–3× slower than real time for 15 fps MP4 on GPU; a 2-minute clip is on the order of 5–10 minutes wall-to-wall (longer at 30 Hz / more frames).
The GRU alone is very fast: < 1 second for a ~2-minute clip on CPU once features exist (~150× faster than real time). It is not the bottleneck.
Not real-time today — 100DOH hand detection accounts for most of the cost. Re-running inference on cached detections (--skip-detect) drops total time to under a second.
¶ Main Findings and Future Work
Findings
- A lightweight GRU over detector features can distill teacher timelines and slightly beat a detector-only dual-dwell baseline on prove-set frame F1.
- Hand-agnostic merging (L∪R) is more reliable than per-hand attribution for “is the user interacting with this object?”
- Event timing is the weak point: segment F1 and boundary quality lag frame-level agreement.
- Detector quality strongly caps both baseline and student; the pipeline is sensitive to missed boxes and identity switches.
- The teacher itself misses some stationary interactions; student errors compound label and detector noise.
Limitations
- Single dataset and device family (HOT3D / Aria).
- Teacher trained with privileged 3D poses — not available at deploy time.
- Object vocabulary fixed to HOT3D instance names; unknown objects at deploy time do not activate.
- No claim of production-ready live deployment; ship gates on prove metrics were not met.
Future work
- More training data and stronger / faster detectors.
- Better temporal modeling (event decoding, boundary refinement, hysteresis).
- Stationary-interaction path in the teacher and student targets.
- Optional evaluation on non-HOT3D Aria clips; latency profiling for near-live use.
¶ Repository
Code, pipelines, and reproduction steps: GitLab README.
Key entry points:
src/run_teacher_pipeline.py— teacher labelssrc/run_student_pipeline.py— GRU inference (HOT3D or MP4)src/run_image_2d_pipeline.py— detector baselinesrc/render_demo_video.py— RGB + boxes + timeline demo MP4