This project investigates how Embodied Virtual Reality (VR) can provide more immersive and measurable environments for neurodivergent learning, therapy, and behavioral research. The project combines the Virtuix Omni One, Meta Quest, and a modular Unity 6 framework to transform functional skills into configurable VR scenarios and synchronized behavioral data.
Harpita Pandian
Rutgers University Honors College
Harpith Pandian
Rutgers University Honors College
Dr. Yingying Chen
Department of Electrical and Computer Engineering
Rutgers University–New Brunswick
Virtual reality can provide safe, controlled, and repeatable environments for neurodivergent children to practice functional skills. However, many existing VR systems remain seated or stationary, limiting both real-world movement practice and the behavioral information that can be captured. Embodied XR makes the participant's whole body part of the activity, allowing researchers to examine how movement, attention, communication, decision-making, and interaction work together during a task.
Can a modular embodied XR framework provide the foundation for measuring and training functional gross and fine motor skills in neurodivergent children?
The primary contribution of this project is not a single VR game, but a reusable framework that can:
Turn a skill into a scenario, a scenario into controlled variations, and every interaction into data.
May 28, 2026
View Week 1 Slides
We reviewed existing VR systems for neurodivergent therapy and identified full-body sensorimotor interaction as an underexplored area. Based on this gap, we defined the project around movement-centered VR experiences that could capture multimodal behavioral data. We also explored Unity XR and Meta XR development tools, planned the initial measurement pipeline, and created an early conversational-avatar prototype using Reallusion characters and the Convai SDK.
June 4, 2026
View Week 2 Slides
We investigated how the Virtuix Omni exposes movement as a three-dimensional velocity vector and how that input is translated into movement inside Unity. We began designing a walking-calibration scene, created a head-directed attention tracker using the Meta Quest CenterEyeAnchor, and logged which virtual objects the participant was facing over time. We also started integrating Meta hand tracking so users could naturally reach for and grab objects within the scene.
June 11, 2026
View Week 3 Slides
We transitioned the project to a Meta Quest deployment workflow and connected the Quest application to the Omni One through Omni Connect. A custom movement script was developed to read treadmill input, rotate it into the correct Unity world direction, and apply it to the Meta XR player rig. We also designed a controlled eight-direction walking experiment to evaluate movement responsiveness, direction mapping, headset-orientation effects, and treadmill-to-rig latency.
June 17, 2026
View Week 4 Slides
We implemented the controlled walking experiment using eight targets and repeated trials while logging Omni input, player trajectory, movement vectors, headset orientation, frame timing, and speed. Locomotion conflicts within the Unity player hierarchy were resolved by creating a simplified Omni player rig, and movement scaling was added to compensate for small input magnitudes. Testing showed no perceptible treadmill-to-rig delay, and turning the participant's head did not produce unintended movement drift.
June 24, 2026
View Week 5 Slides
We replaced the original black-box conversational prototype with a custom speech pipeline connecting microphone input, Whisper speech recognition, a Groq-hosted Llama model, text-to-speech, spatial audio, and NPC animation. An activation boundary was added so conversations only began when the participant approached the character. We also strengthened the social-attention system using raycasting, distance and angle checks, and adaptive visual cues that redirected the participant toward the NPC when attention was lost.
July 2, 2026
View Week 6 Slides
We organized locomotion, attention, speech, hand interaction, and task logic into reusable framework modules. These components were then applied to an embodied peer-interaction experience in which participants would navigate to virtual peers, learn their names and interests, and later recall and approach the correct person. The scenario was designed to combine gross motor navigation with social attention, memory, communication, and object interaction while producing synchronized behavioral measures.
July 8, 2026
We integrated the foundational modules within a single social-interaction scene and resolved issues affecting simultaneous locomotion and hand tracking. Reusable NPC profiles were developed so one avatar prefab could support different names, voices, roles, personalities, and task knowledge. A central NPC controller connected each profile to the speech-recognition, language-model, and text-to-speech components, making it easier to create multiple personalized virtual peers without rebuilding the interaction pipeline.
July 15, 2026
We developed an embodied VR tag experience in which the participant physically pursued a virtual character using the Omni One and tagged the character using hand tracking. The NPC followed a NavMesh-based route, adjusted its behavior based on the participant's distance, and provided AI-generated verbal encouragement. We also created speech-activated power objects, including a Freeze Crystal that temporarily stopped the NPC and a Speed Boost that increased the participant's movement speed.
July 22, 2026
We expanded the tag game from one NPC to multiple peers traveling along separate paths. At each checkpoint, the participant received a visual or spoken instruction identifying the correct target, such as chasing a specific person or shirt color. The system measured whether the participant attended to and pursued the correct NPC, recorded correct and incorrect tags, and provided corresponding rewards or penalties. Head-directed attention was logged throughout each prompt window to measure target focus and switching behavior.
We consolidated the technical components into a unified modular framework and demonstrated how the same foundation could support several skill-targeted experiences. We also finalized the behavioral data structure, research poster, and presentation communicating how embodied XR can make functional mind–body coordination more measurable.
A gamified experience combining selective attention, instruction following, gross motor navigation, hand-based tagging, reward collection, and speech interaction. Participants receive a target prompt, identify and pursue the correct virtual peer, tag them, collect rewards, and use speech-activated power-ups.
A social-learning environment in which participants meet virtual peers, learn information about their names and interests, and later navigate to the correct peer in response to a recall prompt. The experience combines memory, social attention, communication, navigation, and hand interaction.
A structured running experience in which the participant races against virtual peers while regulating speed, following a lane, completing laps, and responding to changing race conditions. Potential measures include lap times, lane adherence, speed regulation, place over time, and response to movement cues.
Future development will focus on expanding the library of skill-targeted experiences, evaluating the system with neurodivergent participants and relevant professionals, and collecting repeated-session data to study individual behavioral patterns and progression. Additional work will explore clinically grounded AI characters, configurable therapist-facing tools, personalized difficulty adjustment, and more advanced analysis of synchronized movement, attention, speech, and interaction data.
We thank Dr. Yingying (Jennifer) Chen, our research advisor, along with Changming Li, Honglu Li, Dirk Catpo Risco, and Zejun Xu for their mentorship and technical guidance. We also thank the teams at Rutgers WINLAB, Virtuix Inc., and the Aiyer Foundation for their resources, collaboration, and support throughout the project.