Member: Yiming Liao
Faculty advisor: Prof. Bo Yuan
Week 1 (05/26-05/28)
This week, I focused on building a theoretical foundation for the diffusion inference pipeline. I studied the core mathematics behind Denoising Diffusion Probabilistic Models (DDPMs), Score-based Generative Models (SGMs), and Score SDE bridging. My primary objective was to analyze exactly why sequential evaluations are computationally expensive in modern inference pipelines, establishing the necessary groundwork to identify memory bottlenecks and implement future optimization strategies.
Week 2 (06/02–06/04)
Foundation study: Reviewed Scalable Diffusion Models with Transformers, World Models, and Video Generation Models as World Simulators to connect diffusion architectures with agent-centered world modeling.
DiT inference structure: Latent Encoding → Patch Tokens → Transformer Denoising → Repeated Sampling → Decoding. DiT replaces the U-Net denoiser with a Transformer but retains the iterative diffusion process.
Cost decomposition: Identified the main inference costs as denoising steps × Transformer forward-pass cost, with token count, self-attention, MLP layers, model depth, and model width as major contributors.
World-model distinction: Clarified that a world model must support latent-state compression, temporal dynamics, memory, and action selection, rather than only generating visually plausible videos.
Project relevance: Established the architectural foundation for studying token reduction, attention optimization, feature reuse, and efficient action-conditioned rollout.
Week 3 (06/09–06/11)
Video diffusion study: Reviewed four representative models: Video Diffusion Models, Stable Video Diffusion, Latte, and CogVideoX.
Architecture evolution: 3D U-Net Video Diffusion → Latent Video Diffusion → Video DiT → Large-Scale Expert Transformer.
Representation structure: Studied how video models encode spatial and temporal information using latent tensors or spatiotemporal patch tokens.
Inference bottlenecks: Identified large video representations, long token sequences, expensive attention, multi-step denoising, and long-horizon error accumulation as the main computational challenges.
Project relevance: Connected video diffusion with world-model inference: video diffusion models visual dynamics, while interactive world models additionally require actions, state transitions, and continuous rollout.
Week 4 (06/16–06/18)
Interactive world-model baseline: Studied Genie to understand the repeated inference workload of interactive environment generation.
Interactive workflow: Previous Frames → Video Tokens → Latent Action → Dynamics Model → Next Video Tokens → Generated Frame. This process repeats after every user action.
Three redundancy sources: Identified spatial redundancy, temporal redundancy, and iterative redundancy as complementary optimization targets.
Efficiency methods: Sparse Imagination reduces visual tokens, SKIP reduces expensive generated frames through keyframe interpolation, and WorldCache reuses or predicts intermediate features across denoising steps.
Main trade-off: Efficiency improvements must preserve action-critical information, rapid physical events, temporal consistency, and long-horizon rollout quality.
Week 5 (06/23–06/25)
Alternative generation structures: Studied Visual Autoregressive Modeling, Infinity, and an efficient diffusion survey to explore efficiency beyond sampling-step reduction.
VAR prediction structure: Coarse Scale → Intermediate Scale → Fine Scale, replacing conventional next-token generation with next-scale prediction.
Infinity prediction structure: Uses bitwise token prediction and self-correction to change how discrete visual representations are generated.
Key finding: Efficiency can come from changing what is predicted, how it is represented, and in what order it is generated, rather than only accelerating diffusion denoising.
Project relevance: Motivated further study of visual tokenizers and video VAEs as the representation foundation for efficient world-model generation and rollout.
Week 6 (06/30–07/02)
Benchmark study: Reviewed the VBench series, including VBench, VBench++, and VBench 2.0, to examine whether current video benchmarks are sufficient for world-model evaluation.
Evaluation workflow: Dimension-Specific Prompts → Generated Videos → Specialist Evaluators → Dimension Scores → Capability Profile.
Evaluation coverage: VBench measures visual quality, temporal consistency, motion smoothness, prompt alignment, subject consistency, object classes, actions, and spatial relationships.
Benchmark evolution: VBench: video quality → VBench++: broader tasks and trustworthiness → VBench 2.0: physics and commonsense faithfulness.
Identified gap: VBench remains mainly an offline video-generation benchmark and does not fully evaluate action-conditioned rollout, causal response, persistent state, memory, or controllability.
Week 7 (07/07–07/09)
Benchmark comparison: Compared VBench, EvalCrafter, and WorldJen through four design choices: evaluation targets, prompt construction, evaluator strategy, and score interpretation.
VBench strategy: Uses controlled dimension-specific prompts and specialist evaluators to produce an interpretable capability profile.
EvalCrafter strategy: Uses structured metadata and 17 objective proxy metrics, then applies regression mappings learned from human ratings to produce human-calibrated aspect scores.
WorldJen strategy: Uses complex prompts, 16 evaluation dimensions, prompt-specific Likert questions, and a VLM judge, with rankings validated through human pairwise preferences.
Overall conclusion: Current benchmarks evaluate visual realism, prompt adherence, motion, temporal consistency, and partial physical plausibility, but still lack complete testing of action-conditioned rollout, counterfactual actions, state-transition accuracy, and long-horizon controllability.
Project direction: Future evaluation should jointly measure inference latency, memory consumption, rollout quality, temporal consistency, action responsiveness, and long-horizon state preservation.