Member: Ryan Purakal, Ryan Baik
Faculty Advisors: Jorge Ortiz
The CARLA slice (P5) is one of eight interconnected project tracks in the WINLAB Summer 2026 CityOS system. The goal is to build one repeatable CARLA workflow with an LLM-style user interaction loop, producing a controlled experiment and a credible transition plan to the lab. CARLA serves as a synthetic sensor: it generates controlled, reproducible simulation traces structured as evidence artifacts, giving downstream components (TeLLMe / P4, AutoLabeling / P3) clean labeled inputs to test against without depending on physical smart room hardware. The interaction loop takes a user intent, runs a CARLA scenario, and returns a grounded, auditable trace in the shared evidence artifact format used across all eight slices.
Details: Environment setup and initial design sketch for the interaction loop.
Goals:
Outcome: Environment access path confirmed and baseline scenario scoped; no simulation run yet.
Details: Got CARLA running headless on the COSMOS node and produced the first structured evidence artifact output.
Goals:
Outcome: First real, structured evidence artifact output produced; node setup unblocked further work.
Details: Overhauled the sensor suite toward a fixed smart-city-infrastructure framing instead of a vehicle dashcam.
Goals:
export_to_repo.py; exported 11 traces to Vikhyat's AutoLabeling repo structureintent_extractor.py (natural language → SimConfig JSON via GPT-4o-mini); untested pending API key from mentorOutcome: Sensor suite now matches the fixed-infrastructure framing; first controlled experiment confirms determinism.
Details: Full scenario controllability overhaul, and the first true end-to-end run of the pipeline.
Goals:
ignore_lights_percentage)intent_extractor.py: GPT-4o-mini via the OpenAI API, an expanded ten-field SimConfig Pydantic model, hour inference from natural language time references, and a CLI entry point writing scenarios/output_config.jsonOutcome: First full NL-to-trace pipeline run, all fields populated correctly; established the config schema the rest of the summer built on.
Details: First real pipeline components producing CityOS-compatible output; first reproducibility check.
Goals:
trace_to_claims.py: converts CARLA traces into structured ground-truth claim filescarla-sim app built, runs in the CityOS kernel, streams detections end to endOutcome: First CityOS-integrated pipeline output, and a confirmed-deterministic reproducibility baseline.
Details: Built and validated a real multi-scenario capture pipeline; moved junction selection from hardcoded to catalog-driven.
Goals:
scenario_capture.py: crosswalk, foggy crash, two group crossingsWalkerAIController fails in 0.9.16 sync mode; replaced with per-tick WalkerControl at 0.83 m/s--junction_id, restored 468 as defaultSimConfig with junction_type + camera_countOutcome: Scenario capture pipeline validated end to end; junction selection is now catalog-driven instead of hardcoded.
Details: Lost node access for part of the week; used the time for refactor review, regression checking, and reference-doc work instead of losing the day.
Goals:
carla_trace.py --junction_id 468 resolves the catalog center (-51.5, 131.4), not the legacy hardcoded value (-41, 124)--maneuver straight fails on junction 468, since it is a T-intersection with no straight-through route on Town10HD_Opt; verified --maneuver right for 5 ticks instead; confirmed camera framing on scp'd PNGsbuild_junction_route doesn't gate on junction_type before attempting a straight maneuver, which will break on other T-intersections in the catalogcarla_trace.py into catalog-driven variables) and transition plan skeleton work insteadHigh School Intern:
Outcome: Junction relocation refactor regression-checked and confirmed correct; scope boundary with Habitat-Lab/Smart Room clarified.
Details: Pivoted from the single hardcoded junction-468 script toward a general, composable scenario system: ScenarioSpec plus behavior primitives, plus a real ground-truth pipeline.
Goals:
ScenarioSpec schema and behavior primitive stubs (SCHEMA_DESIGN.md); added a ScenarioSpec extraction path to intent_extractor.py; scaffolded general executor dispatch logic (scenario_executor.py)verify_scenario.py crash on non-dict property_value in min_fraction; added unit tests against manufactured claimscarla_trace.py; ported EIT output to scenario_capture.pyexport_crosswalk_zone.pyground_truth_v2/ pipeline: type_vocabulary.py, zones_j468.json, visibility.py, build_frames.py, build_ground_truth.py, build_events.pypre_reconcile_bak/, grouped ground-truth-v2 and target-architecture files into subfolders); fixed the resulting cwd-relative path bugsHigh School Intern:
crosswalkLookup.py in order to identify all crosswalks in the CARLA map
CalraSIMcrosswalkTEST.py which identifies a desired crosswalk and spawns five pedestrians of different wearing shirts.
Outcome: Repo pivoted from a single hardcoded scenario to a general, composable ScenarioSpec system with a real ground-truth pipeline.
Details: Live-CARLA verification push; first real live end-to-end demo on junction 468.
Goals:
scenario_executor.py, live-verified a 4-actor sequential spec_build_kwargs() speed-kwarg mapping per primitive (hold_position drops it, brake_at renames to target_speed); added regression testsobject.type "vehicle" into car/truck/bus in type_vocabulary.pyKeyError and widened SUPPORTED_SCENARIOS gating in build_frames.py for traces without a tick field; wired zone lookup into per-actor frame outputcarla_trace.pysensor_consistency_check.py (ground_truth_v2 standalone cross-signal flags); added the Layer 3 vision-based scenario checker (perceptual_check.py)High School Intern:
classify_junction.py to classify different intersections (2 way crosswalks, 3 way, etc.)classify_junction.py was then tested against junction_catalog.json to test the accuracy of the script when compared to human identification of junctions.import_osm.py was used to import the .osm file of New Brunswick into CARLA
Outcome: All 6 behavior primitives live-verified against real CARLA; first live end-to-end demo on junction 468.
Details: Built the full checker chain end to end; found and fixed a real motion bug that had gone undetected all summer; closed Query 2; reorganized the repo by role; exported the first real traces to GroundingEval.
Goals:
config_checker + intent_extractordrive_to() was steering via set_target_velocity() instead of apply_control(), so yaw never tracked direction of travel; switched to real apply_control() with proportional steeringfollow_route(), cutting the junction-468 route from over an hour to 1.6x fewer ticks, live-verifiedfollow_route() silent-timeout-advance bug (added an arrived: bool field)execute_simultaneous_group(), wired into execute_scenario(); live-verifiedfull_stack/, investigations/, testing/, tools/, data/); repointed all cross-module pathsperceptual_check.py/post_run_check.py, not just intent-matchingmulti_sensor_grid.py; added a standalone tool compositing all sensor bundles into one synced grid video per tracebuild_ground_truth.py sensor_rig detection on real trace schemasground_truth.json on every run; exported the first real curated GroundingEval demo traces, with MP4s for both camerascompute_vehicle_infractions() as a plain summary fieldHigh School Intern:
Outcome: Full checker chain built end to end; a real motion bug (drive_to) found via video review and fixed; first real traces shipped to GroundingEval.