Team 3, CityOS Agentic Stack · Rutgers Winlab Summer 2026
Goal: define a structured, verifiable format for what "ground truth" means in a CARLA scene, before any converter code gets written.
- Drafted the ground truth schema from an initial sketch through a fully validated structure: time-series frames[], each holding objects[] (id, type, state, position, visibility) and relations[] (subject-predicate-object triples against named zones).
- Established the core design rule that shapes everything downstream: ground truth is a factual record from the simulator's own internal state, never an AI's description of a scene.
- Began building the CARLA converter, the tool that turns raw simulator output (trajectories, logs) into this schema.
- Settled early architectural decisions: location as a named zone reference rather than raw coordinates; visibility as a required field, not an assumption, since something existing in the simulator isn't the same as something being observable.
Outcome: a schema stable enough to build against, and a converter skeleton ready for its first real test.
¶ Week 5 — First Real Converter Run and Taxonomy v1
Goal: prove the converter works against real data, and establish how difficulty gets measured.
- Ran the converter for the first time against a real CARLA scenario (5 pedestrians crossing a crosswalk, 193 ticks), producing the first genuine, verified ground_truth.json the project produced.
- Finalized the initial difficulty taxonomy: five scenario-complexity factors (actor count, density, location complexity, event type, weather severity) and a six-filter question-difficulty system (presence, count, attribute, relation, temporal, identity).
- Established the governing rule used everywhere since: difficulty is set by the single hardest factor, never averaged, with the driving factor always recorded.
- Went through a teammate code review and merge, incorporating real fixes: schema violations from undeclared fields, per-tick visibility labeling instead of blanket assumptions, and weather severity redefined as a count of active modifiers rather than intensity.
Outcome: first real, checkable ground truth output, and a difficulty framework with real rules instead of placeholders.
¶ Week 6 — Bug Fixes and the Three-Document Architecture
Goal: fix real bugs found during the first run, and design how scenario intent, ground truth, and validation relate to each other.
- Fixed two real converter bugs surfaced by the first live run: an actor_absent sentinel to stop a missing-actor placeholder from ever leaking onto a real tracked object, and an events-per-actor normalization fix so scenes with more routine actors stopped being mislabeled as more complex.
- Designed the three-document architecture that still governs the pipeline: a scenario intent (what was planned), ground truth (what actually happened), and a validation report comparing the two, keeping design-time and measured data permanently separate.
- Defined the strict/tolerant field split for that validation step: things like actor count and map must match exactly; things like exact event timing or spawn position are expected to vary and are logged, not failed.
- Began scoping Checker A (the tool that would eventually perform this comparison) and Checker B (the separate, deterministic checker that verifies TeLLMe's answers against ground truth) as two distinct jobs sharing one ground truth reference.
Outcome: real bugs closed, and the architectural split that prevents scenario design from ever being confused with measured results.
¶ Week 7 — Presentation Groundwork and Location Complexity Redesign
Goal: communicate the project's status to the wider team, and fix a real gap in how location difficulty was being decided.
- Built the first round of status presentations (a then/now overview and a pipeline architecture diagram), establishing a visual language (built vs. designed vs. external-system vs. human-decision) used in every deck since.
- Identified that location_complexity (originally map_complexity) was being authored by feel, not measured, a real gap since every other factor was data-driven.
- Renamed the tier vocabulary from straight/intersection/junction to easy/medium/hard for clarity, and separated "map" (which town) from "location complexity" (a property of one specific spot), two things that had been conflated.
- Designed and built parse_locations.py, a tool that derives location complexity directly from a town's real road-network file, using junction connectivity and clustering, with zero manual judgment involved.
- Began the camera-standardization discussion, since the clustering signal depends on knowing what a camera would actually capture in one frame.
Outcome: location complexity became a measured property instead of an authored one, with a reusable, map-agnostic tool behind it.
¶ Week 8 — Camera Correction and the First Real Location Table
Goal: lock in the real camera parameters, and produce the actual location table for Town10HD.
- Corrected the camera assumption twice: an initial 90° field-of-view guess was replaced with the confirmed real value (75°), and height was set to 25m specifically to protect detection quality: a much higher camera was shown to cut pixel density per object sharply enough to threaten pedestrian detection.
- Ran the location parser against Town10HD's real road file for the first time, producing 10 real locations (4 easy, 5 medium, 1 hard), replacing every earlier estimate with derived, reproducible data.
- Sent the first real zone-polygon request to the CARLA team, including the reasoning behind the camera numbers so the ask was self-contained.
- Found and fixed two real converter bugs while testing against the CARLA team's newest raw data format: a tick-indexing assumption that silently broke on a newer schema version, and a vehicle light-state field that was being read from the wrong data shape entirely.
- Wired real vehicle attributes (color, coarse light state, emergency/taxi special type) into the converter, explicitly excluding "electric" as a claim type since nothing about a vehicle's exterior signals that to a camera.
Outcome: a real, derived location table, a corrected camera standard, and two silent converter bugs caught before they could corrupt further data.
¶ Week 9 — Full Zone Data, Pipeline Packaging, and Checker A
Goal: get complete zone coverage, package the pipeline for the rest of the team, and build the first real scenario-validation checker.
- After an initial zone batch came back incomplete, sent a corrected, fully self-contained prompt to the CARLA team explaining what a zone polygon is, the coordinate-frame mistakes to avoid, and the full set of zone types needed (crosswalk, intersection, sidewalk, road, lane), not just one per location.
- Received and independently verified 146 real zone polygons across all 10 locations: checked centroid accuracy against real reference coordinates and confirmed no degenerate polygons before accepting the data.
- Designed and built scenario_intent.schema.json (the structured scenario-design format) and a pre-flight validator that catches invented locations, mismatched zones, and structurally unreachable difficulty targets before a scenario ever reaches simulation.
- Built and tested Checker A end-to-end against real data, including deliberately broken test cases, confirming it correctly distinguishes PASS, FAIL, and PASS_WITH_NOTES.
- Packaged the full pipeline (schema, converter, location/zone data, both checkers) into one reviewed, renamed deliverable for the team, and produced three tiers of presentation material (public-facing, technical, and a visuals-only deck) for different audiences.
Outcome: complete real zone coverage, a working scenario validator, and a pipeline package the rest of the team could actually build on.
¶ Week 10 — Backlog Generation and the First Real Test Batch
Goal: start generating real scenario backlogs at scale, and get the first batch into actual CARLA execution.
- Defined the concrete rules bulk generation needed: floor/baseline values for every factor, a 30-scenarios-per-cell target grounded in standard sample-size conventions, and a mechanical (not judgment-based) rule for tagging which question families a scenario supports.
- Caught a real structural issue before generation started: since every factor's baseline is its low-tier value, the low tier only has one meaningful case, not five, cutting the originally planned scope from 450 scenarios to roughly 330.
- Wrote staged, tier-specific generation prompts (low, medium, high), with the medium and high prompts explicitly held back until each prior tier is reviewed.
- Rebuilt the taxonomy rubric to reflect every real change made since the original version, and audited the full pipeline package for stale references, catching and fixing several small but real inconsistencies.
- Generated and rigorously evaluated the first real batch: 30 low-tier scenarios, independently checked against the real validator and a from-scratch recomputation of family tagging, with zero mismatches found.
- Sent the batch to the CARLA team for actual execution, the first real test of whether the pipeline works end-to-end outside a validation script.
Outcome: real backlog generation began, and the first batch is now in front of CARLA, the current open item heading into next week.
d
Week 1: Created the data directory schema
smartroom-autolabeling/
├── README.md
├── requirements.txt
├── .gitignore
├── data/
│ └── sample_dataset/
│ └── day_01_2026-06-01/
│ └── rec_20260601_003/
│ ├── metadata.json
│ └── streams/
| ├── camera_main.mp4
| ├── mic_array.wav
| ├── env_sensor_01.csv
| ├── motion_sensor_01.csv