¶ Privacy Leakage Study and Protection for Virtual Reality Devices
Advisor: Dr. Yingying (Jennifer) Chen
Mentors: Changming LiGR, Honglu LiGR, Tianfang ZhangGR
Team: Dirk Catpo RiscoGR, Brody VallierHS, Emily YaoHS
¶ Project Overview
Augmented reality/virtual reality (AR/VR) is used for many applications and have been used for many purposes ranging from communicating and tourism, all the way to healthcare. Accessing the built-in motion sensors does not require user permissions, as most VR applications need to access this information in order to function. However, this introduces the possibility of privacy vulnerabilities: data from zero-permission motion sensors encodes various types of the user’s private information, such as activity information and preferences.
¶ Project Goal
- Investigate privacy leakage in AR/VR devices
- Extract data from the IMU on AR/VR headset and controllers for Human Activity Recognition (HAR)
- Use Support Vector Machine (SVM) and Large Language Model (LLM) to show how IMU data maliciously exposes activities of victim users
¶ Weekly Updates
¶ Week 1
Progress
- Read research paper [1] regarding an eavesdropping attack called Face-Mic
Next Week Goals
- We plan to meet with our mentors and get more information on the duties and expectations of our project
¶ Week 2
Progress
- Read research paper [2] regarding LLMs comprehending the physical world
- Build a connection between research paper and also privacy concerns of AR/VR devices
Next Week Goals
- Get familiar with AR/VR device:
- Meta Quest
- How to use device
- Configure settings on host computer
- Extract motion data from IMU
- Connecting motion sensor application program interface (API) to access data
- Data processing method
¶ Week 3
Progress
- Set up host computer and android studio environment
- Started extracting data from the inertial measurement unit (IMU)
- Recorded and ran trials of varying head motions
Next Week Goals
- Run more tests to collect more data
- Design more motions for data collection
*Different head motions
- Rotational
- Linear
- Combinations of head motions
- Looking left and then right
- Looking up and then down
¶ Week 4
Progress
- Designed and collected more motion data
- Looking up then back middle
- Looking right then back middle
- Moving head around in a shape
- Moving head back and forward
- Used MATLAB noise removing functions to clean graphs
- Original
- Smooth
- Lowpass
- Findpeaks
- Original
- 3D visual of acceleration to show time and position
Next Week Goals
- Find a way to get hand motion data using Android Studio
- Work on fixed prompts to get accurate LLM results using ChatGPT 4o and ChatGPT 4
¶ Week 5
Progress
- Enabling hand motion data collection using VR device
- Utilize Android Studio and VrApi to access VR controller interface and extract motion data
- Conducted additional motion experiments to gather comprehensive data sets
- Motion with both head and hand activities
- Motion with both head and hand activities
- Implemented 3D plots to visualize and analyze hand motion data for accuracy
Next Week Goals
- Utilize motion research paper [3] to model more motion activities
- Start building a CNN model that can recognize activity based on motion data
¶ Week 6
Progress
- Made a list of different motion data to capture and train a convolutional neural network (CNN)
- Do research on previous work based on raw motion data and CNN’s
- Specifics of the motion data
- Samples: 250 per motion
- Users: Dirk, Brody, and Emily
- Motions: front raise, side raise, head right, head left, head up, and head down
- Design prompt for LLM and see output results
Next Week Goals
- Start getting more motion data
- Start using LLM to analyze the collected data
- Use the designed prompts
- Design and try more prompt structures and compare LLM responses
¶ Week 7
Progress
- Collected 250 samples from six different motions to enlarge datasets for the CNN task
- Designed prompts with specific parts for LLMs to establish activity recognition tasks
- Tested prompt with different LLMs
- Inaccurate results from ChatGPT 4o using side raise motion
- Inaccurate results from Gemini Advanced using side raise motion
- Inaccurate results from ChatGPT 4o using side raise motion
Next Week Goals
- Improve the prompt design to get more accurate prediction results from LLMs
- Begin developing a CNN using samples collected from the six motion patterns
¶ Week 8
Progress
- Built a 1-dimensional (1D) convolution neural network (CNN) for activity recognition
- 1D CNN Architecture
- Training and Validation Graph
- Confusion Matrix
- 1D CNN Architecture
- Improving the prompt design to get more accurate predictions results from LLM
- Previous prompt
- New Prompt
- Previous prompt
- LLM’s accuracy of classifying six different motions with new prompt
Next Week Goals
- Use MATLAB to convert time domain motion data into frequency domain for more data representation
- Improve LLM results (78.18%) to be more similar to CNN results (98.22%) by improving fixed prompt
¶ Week 9
Progress
- Built threat models for this AR/VR human activity recognition project (HAR)
- Conducted feature extraction methods and used a support vector machine (SVM) model to select effective features
- Improved LLM fixed prompt by adding more data understanding in expert knowledge
- LLM’s accuracy of classifying six different motions with new week 9 improved prompt from a previous 78.18% to a 90.6%
Next Week Goals
- Improve LLM fixed prompt to get better results than 90.6% by adding statistical features into the prompt derived from SVM results (99.33%)
¶ Week 10
Progress
- Looked into adding statistical features into LLM prompt to improve accuracy
- Gemini Advanced was unable to correctly compute the statistical features based on the current prompt
- Gemini Advanced was unable to correctly compute the statistical features based on the current prompt
- Worked on open house presentation slides and poster
Next Week Goals
- Complete open house poster
- Complete open house presentation slides
¶ Links to Presentations
<a
href="https://docs.google.com/presentation/d/1VZrjZfJtpEVUlbCmo1g7Ra4WfqlPO7QHdEdDePhxg54/edit#slide=id.g20dd9abe089_0_0"style="color:#BB0000; margin-right: 20px;">Week 1
<a
href="https://docs.google.com/presentation/d/1ks7pjpWpulA2y2GsImNmkVBnvmLo06NcEL-JPvpfuwg/edit#slide=id.g20de48f9cd7_0_70"style="color:#BB0000; margin-right: 20px;">Week 2
<a
href="https://docs.google.com/presentation/d/1HyIgimaEBOFLwYhSPUpR2JkvApu04FP9c8duMg6zeYc/edit#slide=id.g20de48f9cd7_0_70"style="color:#BB0000; margin-right: 20px;">Week 3
<a
href="https://docs.google.com/presentation/d/1KRUhHDRpHhzC8x8RDTdJcpnMpVKybfsiIKIJ-DGMdBU/edit#slide=id.g20de48f9cd7_0_70"style="color:#BB0000; margin-right: 20px;">Week 4
<a
href="https://docs.google.com/presentation/d/1Ub-aKYOBnlRYxKw9RfWrc-aVimDxZ3YlwLOPu0zaYlc/edit#slide=id.g2e889759ea4_0_30"style="color:#BB0000; margin-right: 20px;">Week 5
<a
href="https://docs.google.com/presentation/d/1RKGL3x1_bact6RlaXQpRFMf6e8vnYp15zUTWYiKm1UY/edit#slide=id.g2e889759ea4_0_30"style="color:#BB0000; margin-right: 20px;">Week 6