Sudeep Babasaheb Kakade(GR), Ryan Naini(HS), Aleeza Khan(HS), Andrew Chan(HS), Samhita Shriram(HS)
Advisor: Dr. Yingying (Jennifer) Chen
Mentors: Changming Li, Honglu Li, Zejun Xu, Dirk Catpo Risco
Virtual Reality (VR) headsets contain built-in IMU sensors (accelerometer, gyroscope) that capture rich motion data at 72Hz. Accessing these motion sensors does not require special permissions, making them ideal for activity recognition. This project builds a complete pipeline to recognize 7 physical activities using only the Meta Quest HMD's built-in sensors — no extra hardware needed.
- Extract IMU sensor data from Meta Quest VR headset and controllers for Human Activity Recognition (HAR)
- Implement signal preprocessing following Zhao et al. and FitCoach methodologies
- Compare traditional ML classifiers (Random Forest, KNN, SVM) and two-stream CNN
- Use Large Language Model (Gemini) with prompt engineering to classify activities with zero training data — following Penetrative AI concept
Progress
- Read research papers on VR activity recognition [1][2][3]
- Studied VR hardware and IMU sensors — accelerometer, gyroscope
- Learned sensor outputs: Linear Acceleration, Angular Velocity, Linear Velocity, Angular Acceleration
- Set up Android Studio with Meta VrApi SDK on Windows PC
- Connected Meta Quest 1 (device ID: 1PASH9AK670176) via ADB
Next Week Goals
- Build C++ data collection app using VrApi SDK
- Start extracting raw sensor data from the headset
Progress
- Built custom C++ app using Android Studio + Meta VrApi SDK to extract sensor data
- Extracted raw sensor data at 72Hz via ADB logcat from HMD and both controllers
- Collected initial dataset for 3 activities: lunge, bicep curl, head rotation
- Wrote Python scripts to parse and filter raw logcat data
Visualized sensor data as 3-axis line plots (x, y, z over time)
Next Week Goals
- Build first classifier on raw sensor data
- Implement signal preprocessing from research papers
Progress
- Implemented Short Time Energy (STE) rep segmentation from FitCoach paper [2]
- Segmented continuous captures into individual reps with energy plot visualization
- Built first Random Forest classifier on raw data — 96.85% accuracy across 3 activities
- Achieved 100% accuracy after applying preprocessing on 3-activity dataset
Next Week Goals
- Implement full preprocessing pipeline from Zhao et al. [1] and FitCoach [2]
- Expand dataset with more activities
Progress
- Implemented full preprocessing pipeline:
- FitCoach: Gravity removal — subtracted mean acceleration as gravity estimate
- FitCoach: Low-pass filter — 5-frame moving average for noise smoothing
- Zhao et al.: Idle frame trimming — removed standing-still frames (15% threshold)
- Zhao et al.: Sliding window segmentation — 40 frames, stride=1 (~0.55 sec per window)
- Two-stream split following Zhao et al.
- Body stream: Linear Velocity (x,y,z)
- Head stream: Angular Velocity + Angular Acceleration (x,y,z each)
- Accuracy improved from 96.85% to 100% after preprocessing
High School Intern Progress
- Read 2 papers to better understand and learn more about the project [1][2]
- Gained background information on data sensing pipeline, gyroscopes, accelerometers, etc.
Next Week Goals
- Implement PCA feature extraction
- Collect data for additional activities
- Compare multiple classifiers
- High School Interns: Learn about machine learning model
Progress
- Expanded dataset to 7 activities (added squat, nodding, side raise, arm swing)
- Extracted 60 statistical features per window (5 stats × 12 channels)
- Applied two-stream PCA: Body → 2 components, Head → 3 components = 5 PCA total
- Compared classifiers on 28,040 windows:
KNN k=3: 98.82% (best)
Random Forest: 98.70%
SVM RBF: 80.49%
- Identified top 5 most discriminative features — all Angular Acceleration min/max (95-96% individual accuracy)
- Read Penetrative AI paper [3]
High School Intern Progress
- Studied KNN, Random Forest, and SVM machine learning networks
- Helped expand the main project's data set
- Began brainstorming actions to classify for the intern classification project
Next Week Goals
- Collect 50 reps per activity to expand dataset
- Implement two-stream CNN following Zhao et al. architecture
- High School Interns: Begin data collection and build a preprocessing pipeline
Progress
- Collected 50 reps per activity — expanded dataset to 162,047 windows across 7 activities
- Implemented two-stream CNN/MLP following Zhao et al. architecture:
Body stream: Conv1D(64) → BatchNorm → Conv1D(64) → BatchNorm → GlobalAvgPool → Dense(64)
Head stream: Conv1D(64) → BatchNorm → Conv1D(64) → BatchNorm → GlobalAvgPool → Dense(64)
Merged: Concatenate(128) → Dense(128) → Softmax(7)
- Results on 162,047 windows:
Two-Stream CNN/MLP: 99.61%
KNN k=3 (5 PCA): 95.43%
Random Forest (5 PCA): 93.91%
SVM RBF (5 PCA): 67.07%
High School Intern Progress
- Began collecting data for tree pose, forward fold, and neck stretch activities.
- Implemented preprocessing pipeline:
- Gravity Removal: Removing gravity from raw linear acceleration data ensures the true linear acceleration is captured.
- Idle Trimming: Removing segments of inactivity at the beginning and end of an activity.
- Denoising: Utilized Butterworth low pass filter and Savitzky-Golay filter to smooth out the data around the peaks and at the 3 second rest intervals.
- Segmenting: Split up data into segments of reps that will be classified by machine learning models.
Next Week Goals
- Study prompt engineering techniques
- Design prompts for LLM-based activity classification
- High School Interns: Build machine learning models (e.g. random forest, svm, knn, and cnn) to classify reps.
Progress
- Studied Prompt Engineering Guide (promptingguide.ai)
- Designed 3 prompting strategies for Gemini LLM:
Zero-shot: background info + sensor data, no examples
Few-shot: 1 labeled example per activity before unknown window
Chain of thought: step-by-step reasoning through sensor physics
- Tested 35 samples (5 per activity) across all 3 strategies
Result: 100% accuracy with zero training data
- Validates Penetrative AI concept — LLMs comprehend physical sensor data using built-in world knowledge
High School Intern Progress
- Expanded dataset to 5 activities with 50 reps each (250 reps total)
- Conducted feature extraction and classification.
- Determined 5 most discriminative overall and individual features to use in ML classifiers
- Applied 2-stream PCA (2 body + 3 head components) to the ML classifiers
- Compared accuracy of 5 different ML models (training dataset: 200 reps, testing dataset: 50 reps), as well as two-stream CNN:
- KNN (k=3): 96.08%
- KNN (k=5): 96.08%
- SVM (linear kernel): 94.12%
- SVM (RBF kernel): 94.12%
- Random Forest: 94.12%
- CNN: 96.08%
Next Week Goals
- Prepare final presentation and poster
- Complete project wiki documentation
Achieved 99.61% accuracy using Two-Stream CNN/MLP trained on 162,047 labeled windows across 7 activities (lunge, bicep curl, head rotation, squat, nodding, side raise, arm swing). KNN k=3 with 5 PCA components achieved 95.43% accuracy. Using Gemini LLM with prompt engineering (zero-shot, few-shot, chain of thought), achieved 100% classification accuracy with zero training data — validating the Penetrative AI concept that LLMs can comprehend physical sensor data using built-in world knowledge.
High school interns expanded the dataset to include 5 additional activities which are typically at home yoga exercises (forward fold, neck stretch, tree pose, lunge twist, jumping jacks). Individual repetitions of these activities were successfully classified using 5 different ML models as well as a CNN with an average accuracy of 95.1%.
162,047 preprocessed sliding windows collected from Meta Quest HMD and controllers across 7 activities, each with 50 repetitions captured at 72Hz.
High School Interns: 104,607 preprocessed sliding windows collected from Meta Quest HMD and controllers across 5 activities, each with 50 repetitions captured at 72 Hz.
[Week 1]
[https://docs.google.com/presentation/d/1wUjsd-sG3P-VXgxIJu-z9nkT1aEpIzdF/edit?slide=id.p1#slide=id.p1
[Week 2]
[https://docs.google.com/presentation/d/1JgqKGWHXbb1dJp2_NhDU_YfBjaDFQlQx/edit?slide=id.p8#slide=id.p8
[Week 3]
[https://docs.google.com/presentation/d/11Kl1zEhsmCWiBBZAKw-PN-2Z9wn5ksVN/edit?slide=id.p1#slide=id.p1
[Week 4-5]
[https://docs.google.com/presentation/d/1IixPCsmJ_5guVXTz7RDaUpIDGyMdH3q_/edit?slide=id.g3f04344e761_0_122#slide=id.g3f04344e761_0_122
[Week 6]
[https://docs.google.com/presentation/d/1MO4D_2LN15n5Tqx1XiJQHjMBiKChv3X2/edit?slide=id.p1#slide=id.p1
[Week 7]
[https://docs.google.com/presentation/d/1z76p3I-6iXtIR2JRiFryhLUL3vJNJSJW/edit?slide=id.p1#slide=id.p1
[Week 8]
[https://docs.google.com/presentation/d/1J-i4bfN_HUH1iUnhXx2nFiIdbITc9RyC/edit?slide=id.p1#slide=id.p1
[Final Presentation]
https://docs.google.com/presentation/d/1htMlGfXu3UUHsYMQ46fIJU3KvUuDVGlc/edit?slide=id.g3f427365eec_0_154#slide=id.g3f427365eec_0_154
[1] Zhao et al., "Real-time Recognition of In-place Body Actions and Head Gestures using Only Head-tracking in Virtual Reality," IEEE VR 2023
[2] Guo et al., "FitCoach: Virtual Fitness Coach Empowered by Wearable Mobile Devices," IEEE INFOCOM 2017
[3] Xu et al., "Penetrative AI: Making LLMs Comprehend the Physical World," HotMobile 2024