Members: Crystal Burbano OtalvaroUG, Alan JiangUG
Faculty: Professor Jorge Ortiz
A room-based sensing system that uses camera and their data to create timestamped, repeatable evidence artifacts with metadata, logs, quality checks, and routable event records.
Key Takeways:
During our first week at WINLAB, we familiarized ourselves with the ongoing apps and projects our work is based on, most notably the CityOS paper. We focused on understanding how a sensing node collects, timestamps, organizes, and routes data. In the CityOS paper there are three types of API privacy interfaces, for this project specifically we focus most on API 1. Also known as the "On-Scene" stage, API 1 focuses on how sensors observe the current environment, process data locally, and produce outputs without immediately relying on a cloud backend.
We also began setting up the Raspberry Pi and USB camera to create our first demo, of 6-12 second long timestamped videos. Video footage included either project member being in the frame, as well as overall of camera focus. Since our project is a stack of other components that are built on top of one another, a local website was created for easy access of the videos for different use amongst different steps of the pipeline.
Key Takeways:
During our second week we wanted to enhance our understanding with the projects that expand on us so we read the individual papers: TraceFix, ConcordFS, BenchPress, etc. By knowing more about the projects that use "SmartRoom as an IndoorCityOS Node" as a foundation we are able to understand the influence of our outputs and how they work later on.
We introduced the sensors into our current environment attached to the Raspberry Pi. Sensors such as PIR motion, environmental (temperature, humdity,pressure, gas), color/light, accelerometer, magnetometer, etc. The main goal we completed was creating a full run script that automatically organizes that video, event record, metadata, and logs into a run folder. Tests were ran to make sure all the data captured was real and working, not just interim correctly formatted data. We also worked on adding each version of the newer file system to the Cosmos GitLab for documentation purposes.
*List of Sensors below
Key Takeways:
During our third week we wanted to create a data validating pipeline that would immeadiately create a file, example name: validation_report.json, to check if all the metadata, video, audio, and sensor recordings were working accurately and unanimously. Here unanimously is linked to all expected files existing, timestamps having the expected number of rows, and aligning frames per second. During the previous week we were mannually checking for predetermined vs. real data, working audio, and etc, so the pipeline would work well here.
Since introducing the sensors to our environment we found some issues with the OPS243-C Radar. We noticed that the values kept snapping to 200 cm. or 500 cm. and the data was being put into bins with 8.5 cm. distance apart. After some further research we learned that the radar is picking up on a larger reflector, most likely the wall, because of that higher reflective objects need to be used during testing to combat our issue. We were also informed that the radar's capabilities can be different depending on the wavelength setting its placed to.
A lot of adaptations for the radar are needed depending on the environment. Since we were testing in a semi-packed room with all types of equipment we were guided to keep a complete camera environment instead.
Key Takeways:
After meeting with Professor Ortiz and having a better sense of the updated scope of our part of the whole stack we wanted to implement a more controlled environment with only cameras. We already have one USB Camera and added another on a seperate Raspberry Pi so we can place them on seperate areas of the room.
We ended up discarding the idea of sensors, specifically the radar one because the project goal is not to asnwer how fast a person is moving a room, instead its to pick up on if someone is in the room and what are they doing. After determining the locations of our two cameras we started looking into vision/object detection models that would be able to use our 2 angle camera footage.
During research we came across R-CNN (Regions with Convolutional Neural Networks) it was a model that works by first generated possible regions where objects might be located, then running the neural network on those regions to extract features and classify objects. Although it is a powerful way for object detection its a bit slower and multi-stepped compared to newer models. YOLO (You Only Look Once) works with one neural network to predict class probabilities and bounding boxes directly from the image/video in one pass. This model, specifically YOLO26n would be very useful in our project since it is very strong, fast, one step, has realt-time vision tasks capabilties, and is small enough to be handled on our laptops. Since we eventually want to scale this project using YOLO on the Pis is not recommended.
Key Takeways:
Week 5 focused on improving action classification and exploring new hardware for more reliable sensing. While waiting for a new camera and microphone, we researched their roles in the project and prepared for installation and configuration. We also tested the Intel RealSense D435 depth camera using the RealSense Viewer and its measurement tools, exploring how depth information could strengthen room understanding beyond regular video.
On the AI side, we improved the temporal classifier used with YOLO pose detection. We addressed issues such as skeletal keypoint mislabeling and flickering action labels by changing the camera angle to capture more of a person’s body, rejecting partial skeletons, filtering out low-confidence classifications, and blacklisting irrelevant action classes.
We also evaluated limitations in the NTU RGB+D dataset, including exaggerated actions, niche labels, and its Kinect-based 3D skeleton format. Because these limitations may not match realistic SmartRoom footage, we began considering other datasets such as HMDB51. Finally, we explored stride length settings: lower strides can make action labels more stable, while higher strides reduce processing but may miss or weaken short actions like jumping or running.
Key Takeways:
During our sixth week, we focused on improving the reliability of our action recognition pipeline and preparing our outputs for integration with the larger CityOS ecosystem. One major change was replacing YOLO pose estimation with RTM pose. While YOLO can localize a person and estimate keypoints in a single pass, RTM produced more stable skeleton tracking with less jitter and benefited from broader training data.
We also continued experimenting with our temporal classification pipeline. Actions are classified using a 48-frame sliding window, where a buffer of pose information is accumulated before making a prediction. We tested different stride lengths, where a stride of 1 processes every frame and a stride of 2 processes every other frame. Adujusting the stride and overlap between windows allowed us to explore the tradeoff between computational cost and classification stability. Lower strides provide more information to the model but increase processing time, while higher strides reduce computational load at the risk of missing short actions.
We also continued preparing our outputs for CityOS integration by reintroducing metadata files and data validation, ensuring each recording contained consistent timestamped footage, detections, pose estimates, logs, etc. To support future groups, we generated scenario descriptions and spatial temporal question-and-answer pairs from recorded footge that describe events within a scene. For example, one generated question was:
"What object does the person hold close to the camera near the end of the clip?"
With the answer:
"A light-colored smartphone."
Finally, we began testing our JSON outputs with TraceFix. This highlighted the importance of standardized formatting across projects and revealed current limitations in assigning identities and positions to specific individuals for more detailed scene understanding.
Key Takeways:
During our seventh week, we focused on improving the reliability of our camera and preparing the project for larger scale data collection. We expanded our data validation pipeline to automatically verify that each recording contained valid videos, metadata, timestamps, etc. These validation checks ensure that every recording entering the SmartRoom pipeline is complete and internally consistent before being used by downstream applications. We also continued improving camera and timestamp synchronization while preparing to replace one of the existing cameras for more consistent recordings.
Another major focus was camera calibration. Using a checkerboard calibration target, we estimated each camera's intrinsic parameters, including focal length, principal point, and lens distortion coefficients. We also reviewed extrinsic calibration, which determines the camera's position and orientation relative to the environment, allowing future systems to better understand object locations across multiple cameras.
Finally, we continued planning how SmartRoom outputs will integrate into the CityOS architecture. We explored how processed data should be formatted, where it should enter the CityOS pipeline, and how applications may later retrieve information through a dashboard or retrieval interface. Rather than simply storing video recordings, our goal is to provide structured outputs, including timestamps, detections, tracking information, and action classifications, that can be efficiently consumed by future CityOS applications.
Key Takeways:
During our eight week, we focused on improving the hardware setup for the SmartRoom. We designed a custom mount in OnShape for the Intel RealSense Depth cameras to provide a stable and consistent camera placement. Since the project will eventually estimate where people are within the room, maintaining a fixed camera position is important for obtaining accurate depth measurements and future geometric reasoning.
We also integrated the depth and security cameras into our current pipeline and worked on synchronizing recordings across multiple cameras. Rather than relying only on a common recording start time, we began using the Intel RealSense SDK to retrieve each frame's hardware capture timestamp. To further improve coordination, we developed a calibration method using controlled light flickers as a shared reference point between cameras, allowing timestamp offsets to be measured more accurately.
Another major focus was optimizing performance on the Raspberry pi 4. Running multiple cameras silmultaneously placed a heavy computational load on the device. To reduce this, we changed the pipeline so that raw footage is captured first and encoded after recording is complete, reducing frame drops and improving capture speed. We also lowered the resolution to 640x680, which remains sufficient for pose estimation while decreasing processing requirements.
Finally, we continued planning the final SmartRoom deployment by evaluating camera placement within the room.
BME680 (0x76) – environmental: temp, humidity, pressure, VOC gas:
TCS34725 (0x29) – RGB color/light:
ADXL345 (0x53) – 3-axis accelerometer:
MLX90393 (0x0c) – 3-axis magnetometer:
Analog-side chips (not on I2C — read via the MCP3008 SPI ADC)
MCP3008 (8-ch ADC):
MAX9814 (mic amp):
ZRE200GE (pyroelectric PIR) + NCS36000 (PIR controller IC): the ZRE200GE is from Excelitas; the NCS36000 datasheet is on onsemi's site.
LMV324 (quad op-amp):