Member: Ryan Purakal
Faculty Advisors: Jorge Ortiz
The CARLA slice (P5) is one of eight interconnected project tracks in the WINLAB Summer 2026 CityOS system. The goal is to build one repeatable CARLA workflow with an LLM-style user interaction loop, producing a controlled experiment and a credible transition plan to the lab. CARLA serves as a synthetic sensor: it generates controlled, reproducible simulation traces structured as evidence artifacts, giving downstream components (TeLLMe / P4, AutoLabeling / P3) clean labeled inputs to test against without depending on physical smart room hardware. The interaction loop takes a user intent, runs a CARLA scenario, and returns a grounded, auditable trace in the shared evidence artifact format used across all eight slices.
Began CARLA environment setup: connected to the WINLAB gateway and ORBIT testbed and attempted GPU node access. Produced a first design sketch of the five-stage interaction loop: scenario config → simulation runner → observation collector → intent extractor (LLM) → evidence artifact writer. Identified the baseline scenario for Week 2: a Tesla Model 3 on autopilot approaching a static pedestrian at a crosswalk in Town03, ClearNoon weather, synchronous mode at 20 Hz, fixed seed 42.
Completed COSMOS node setup: Docker, NVIDIA Container Toolkit, and CARLA 0.9.16 running headless on a Tesla V100 with a confirmed Python client connection. Saved a minimal base image for faster session startup. Extended the baseline simulation script with command-line controls for weather, seed, and tick count; each run now saves RGB frames and a structured evidence artifact JSON log to its own subfolder, with per-tick claims recording vehicle position, speed, and weather state. Two experiment runs completed. Documented the P5 slice architecture and clarified the full handoff chain from CARLA through CityOS to TeLLMe.
Overhauled the sensor suite to match a fixed smart city infrastructure framing: replaced the vehicle-mounted dashcam with a fixed overhead RGB camera at junction 468 in Town10HD, added a co-located LiDAR sensor outputting compressed point clouds per tick, and confirmed both working with verified output. Added deterministic NPC traffic (20 vehicles, 10 walkers) with traffic manager seeding for reproducible runs. Extended weather logging to include precipitation, wind intensity, fog density, wetness, and yaw. Added a vehicle-mounted IMU as a V2I data source; confirmed accelerometer and gyroscope values physically plausible. Ran a controlled experiment (ClearNoon vs HardRainNoon, seed 42, 200 ticks) confirming tick-for-tick position matching with weather as the only variable. Built export_to_repo.py to reorganize traces into Vikhyat's AutoLabeling repo structure and exported 11 traces. Built intent_extractor.py for natural language to SimConfig JSON using GPT-4o-mini; untested pending API key from mentor.
Overhauled the simulation for full scenario controllability. Implemented waypoint-based ego vehicle routing through junction 468 with support for straight, left, and right turn maneuvers; fixed CARLA 0.9.16-specific autopilot sequencing (world tick before traffic manager calls) and traffic light override (ignore_lights_percentage). Added a second fixed overhead RGB camera covering all three road arms of the intersection. Expanded the scenario config system with five new CLI controls: traffic density presets (empty through rush hour), weather condition overlays (clear, post-rain, foggy), clock hour inference mapped to sun altitude, ego start mode (moving or stopped at the intersection), and a JSON config file argument that drives the full simulation with CLI override priority. Added semantic segmentation pixel counting with correct CARLA 0.9.16 vehicle tags (14/15/16), radar detection logging, traffic light state logging, and per-tick fields for all new scenario parameters. Rewrote intent_extractor.py to use GPT-4o-mini via the OpenAI API with an expanded ten-field SimConfig Pydantic model covering all new scenario knobs; added hour inference from natural language time references and a CLI entry point that writes scenarios/output_config.json. Ran the full end-to-end pipeline for the first time: typed a natural language prompt, extracted a structured config, and produced a complete labeled simulation trace with all fields populated correctly.