Member: Yiming Liao
Faculty advisor: Prof. Bo Yuan
Research Context: Efficient video generation is only meaningful if reduced computation does not damage visual quality, prompt adherence, motion, or temporal consistency. Reliable evaluation is therefore necessary for measuring the quality–efficiency trade-off.
Project Evolution: The project initially studied inference bottlenecks and efficiency methods for diffusion-based video generation. It later shifted toward video-generation evaluation because quality preservation cannot be established without understanding how benchmark scores are produced.
Core Challenge: Existing benchmarks use specialist models, objective metrics, and VLM judges as proxies for video quality. However, these scores may not agree with visible video content and may fail to distinguish different generation failures.
Project Objective: This project investigates how current video-generation benchmarks evaluate generated videos and whether their scores reliably reflect visible quality, prompt fulfillment, motion, and temporal behavior.
Methodology: Compared the evaluation designs of VBench, EvalCrafter, WorldJen, and WorldScore; inspected official benchmark implementations; manually compared same-prompt videos with their scores; and conducted a two-model WorldScore experiment to identify score–content contradictions.
Week 1 (05/26-05/28)
This week, I focused on building a theoretical foundation for the diffusion inference pipeline. I studied the core mathematics behind Denoising Diffusion Probabilistic Models (DDPMs), Score-based Generative Models (SGMs), and Score SDE bridging. My primary objective was to analyze exactly why sequential evaluations are computationally expensive in modern inference pipelines, establishing the necessary groundwork to identify memory bottlenecks and implement future optimization strategies.
Week 2 (06/02–06/04)
Foundation study: Reviewed Scalable Diffusion Models with Transformers, World Models, and Video Generation Models as World Simulators to connect diffusion architectures with agent-centered world modeling.
DiT inference structure: Latent Encoding → Patch Tokens → Transformer Denoising → Repeated Sampling → Decoding. DiT replaces the U-Net denoiser with a Transformer but retains the iterative diffusion process.
Cost decomposition: Identified the main inference costs as denoising steps × Transformer forward-pass cost, with token count, self-attention, MLP layers, model depth, and model width as major contributors.
World-model distinction: Clarified that a world model must support latent-state compression, temporal dynamics, memory, and action selection, rather than only generating visually plausible videos.
Project relevance: Established the architectural foundation for studying token reduction, attention optimization, feature reuse, and efficient action-conditioned rollout.
Week 3 (06/09–06/11)
Video diffusion study: Reviewed four representative models: Video Diffusion Models, Stable Video Diffusion, Latte, and CogVideoX.
Architecture evolution: 3D U-Net Video Diffusion → Latent Video Diffusion → Video DiT → Large-Scale Expert Transformer.
Representation structure: Studied how video models encode spatial and temporal information using latent tensors or spatiotemporal patch tokens.
Inference bottlenecks: Identified large video representations, long token sequences, expensive attention, multi-step denoising, and long-horizon error accumulation as the main computational challenges.
Project relevance: Connected video diffusion with world-model inference: video diffusion models visual dynamics, while interactive world models additionally require actions, state transitions, and continuous rollout.
Week 4 (06/16–06/18)
Interactive world-model baseline: Studied Genie to understand the repeated inference workload of interactive environment generation.
Interactive workflow: Previous Frames → Video Tokens → Latent Action → Dynamics Model → Next Video Tokens → Generated Frame. This process repeats after every user action.
Three redundancy sources: Identified spatial redundancy, temporal redundancy, and iterative redundancy as complementary optimization targets.
Efficiency methods: Sparse Imagination reduces visual tokens, SKIP reduces expensive generated frames through keyframe interpolation, and WorldCache reuses or predicts intermediate features across denoising steps.
Main trade-off: Efficiency improvements must preserve action-critical information, rapid physical events, temporal consistency, and long-horizon rollout quality.
Week 5 (06/23–06/25)
Alternative generation structures: Studied Visual Autoregressive Modeling, Infinity, and an efficient diffusion survey to explore efficiency beyond sampling-step reduction.
VAR prediction structure: Coarse Scale → Intermediate Scale → Fine Scale, replacing conventional next-token generation with next-scale prediction.
Infinity prediction structure: Uses bitwise token prediction and self-correction to change how discrete visual representations are generated.
Key finding: Efficiency can come from changing what is predicted, how it is represented, and in what order it is generated, rather than only accelerating diffusion denoising.
Project relevance: Motivated further study of visual tokenizers and video VAEs as the representation foundation for efficient world-model generation and rollout.
Week 6 (06/30–07/02)
Benchmark study: Reviewed the VBench series, including VBench, VBench++, and VBench 2.0, to examine whether current video benchmarks are sufficient for world-model evaluation.
Evaluation workflow: Dimension-Specific Prompts → Generated Videos → Specialist Evaluators → Dimension Scores → Capability Profile.
Evaluation coverage: VBench measures visual quality, temporal consistency, motion smoothness, prompt alignment, subject consistency, object classes, actions, and spatial relationships.
Benchmark evolution: VBench: video quality → VBench++: broader tasks and trustworthiness → VBench 2.0: physics and commonsense faithfulness.
Identified gap: VBench remains mainly an offline video-generation benchmark and does not fully evaluate action-conditioned rollout, causal response, persistent state, memory, or controllability.
Week 7 (07/07–07/09)
Benchmark comparison: Compared VBench, EvalCrafter, and WorldJen through four design choices: evaluation targets, prompt construction, evaluator strategy, and score interpretation.
VBench strategy: Uses controlled dimension-specific prompts and specialist evaluators to produce an interpretable capability profile.
EvalCrafter strategy: Uses structured metadata and 17 objective proxy metrics, then applies regression mappings learned from human ratings to produce human-calibrated aspect scores.
WorldJen strategy: Uses complex prompts, 16 evaluation dimensions, prompt-specific Likert questions, and a VLM judge, with rankings validated through human pairwise preferences.
Overall conclusion: Current benchmarks evaluate visual realism, prompt adherence, motion, temporal consistency, and partial physical plausibility, but still lack complete testing of action-conditioned rollout, counterfactual actions, state-transition accuracy, and long-horizon controllability.
Project direction: Future evaluation should jointly measure inference latency, memory consumption, rollout quality, temporal consistency, action responsiveness, and long-horizon state preservation.
Week 8 (07/14–07/16)
Benchmark implementation audit: Examined the official source code for all 16 VBench 1.0 dimensions and 18 VBench 2.0 dimensions to verify how their scores are actually calculated.
Evaluator evolution: VBench 1.0 mainly uses specialist models such as DINO, CLIP, RAFT, and GRiT, while VBench 2.0 increasingly uses LLaVA-Video caption/VQA → Qwen2.5 judge pipelines.
Video-level validation: Selected videos generated from the same prompts with large score differences, then manually compared the scores with the visible video content.
Key finding: Found score–content contradictions in dimensions including color, scene, human_action, and background_consistency.
Implementation finding: The 16 VBench 1.0 dimensions use only 11 distinct generated-video pools, meaning that some dimensions reuse the same videos with different evaluators.
Project direction: Benchmark scores should be checked against actual generated videos instead of being trusted solely from metric names or paper descriptions.
Week 9 (07/21–07/23)
WorldScore experiment: Evaluated VideoCrafter1 and VideoCrafter2 using WorldScore to examine whether its metric results agreed with visible video behavior.
Model comparison: VideoCrafter2 achieved better results on 8 of the 9 evaluated metrics, while motion_magnitude was the only major regression.
Camera-motion check: Large camera_error values appeared mainly for orbit and pan cases. Manual inspection confirmed that neither model correctly completed the requested camera movement.
Action check: Both jellyfish videos failed the requested action, but the collapsed VideoCrafter1 video received a motion_accuracy score of −96.39, while the mostly static VideoCrafter2 video received −5.08.
Key finding: The score difference mainly separated generation collapse from static output, even though both videos failed to perform the requested action.
Project direction: Evaluation should distinguish between the requested behavior did not happen, the behavior happened incorrectly, and the generation collapsed.