Skip to content

README.md

Latest commit

 

History

History
905 lines (715 loc) · 54 KB

README.md

File metadata and controls

905 lines (715 loc) · 54 KB

English | 中文

Temporal Consistency

Objective: Model temporal dynamics, object motion, and causal relationships in video sequences.

Historical Significance: Early explorations of the world's "physics engine," capturing regularities in how scenes evolve over time.

Representative Works

PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning
  • Authors: Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang, Philip S. Yu, Mingsheng Long
  • arXiv ID: 2103.09504
  • One-liner: A novel recurrent architecture that decouples memory cells to capture spatio-temporal dynamics in predictive learning of video frames.
  • Published in: TPAMI 2023
  • Links: [Paper] | [PDF] | [Code]

Core Innovation
Introduces decoupled memory cells embedded in an RNN backbone to separately capture spatial appearance and temporal dynamics, then fuses them into a unified complex-scene representation. A “zig-zag” memory flow propagates information across layers, while a curriculum-learning strategy enables modeling of long-term temporal dependencies.

Abstract The predictive learning of spatiotemporal sequences aims to generate future images by learning from the historical context, where the visual dynamics are believed to have modular structures that can be learned with compositional subsystems. This paper models these structures by presenting PredRNN, a new recurrent network, in which a pair of memory cells are explicitly decoupled, operate in nearly independent transition manners, and finally form unified representations of the complex environment. Concretely, besides the original memory cell of LSTM, this network is featured by a zigzag memory flow that propagates in both bottom-up and top-down directions across all layers, enabling the learned visual dynamics at different levels of RNNs to communicate. It also leverages a memory decoupling loss to keep the memory cells from learning redundant features. We further propose a new curriculum learning strategy to force PredRNN to learn long-term dynamics from context frames, which can be generalized to most sequence-to-sequence models. We provide detailed ablation studies to verify the effectiveness of each component. Our approach is shown to obtain highly competitive results on five datasets for both action-free and action-conditioned predictive learning scenarios.
Key points * Introduces a memory‐pair structure: two memory cells decoupled in transition to separately capture spatial appearances and temporal dynamics. * Unified representation by merging the outputs of the two memory streams. * Applied to spatiotemporal predictive learning without explicit geometry or flow guidance. * Demonstrates improved future-frame prediction performance on benchmark datasets.
SimVP: Simpler yet Better Video Prediction
  • Authors: Zhangyang Gao, Cheng Tan, Lirong Wu, Stan Z. Li
  • arXiv ID: 2206.05099
  • One-liner: A purely convolutional video-prediction model trained end-to-end with MSE loss, showing that simplicity can outperform more complex architectures.
  • Published in: CVPR 2022
  • Links: [Paper] | [PDF] | [Code]

Core Innovation
Proposes an ultra-simple end-to-end video-prediction framework that uses only a CNN encoder–decoder trained with plain MSE loss; without RNNs, Transformers, or any sophisticated modules, it matches or surpasses far more complex models on multiple benchmarks.

Abstract From CNN, RNN, to ViT, we have witnessed remarkable advancements in video prediction, incorporating auxiliary inputs, elaborate neural architectures, and sophisticated training strategies. We admire these progresses but are confused about the necessity: is there a simple method that can perform comparably well? This paper proposes SimVP, a simple video prediction model that is completely built upon CNN and trained by MSE loss in an end-to-end fashion. Without introducing any additional tricks and complicated strategies, we can achieve state-of-the-art performance on five benchmark datasets. Through extended experiments, we demonstrate that SimVP has strong generalization and extensibility on real-world datasets. The significant reduction of training cost makes it easier to scale to complex scenarios. We believe SimVP can serve as a solid baseline to stimulate the further development of video prediction.
Key points * Model based only on convolutional encoder-decoder structure (no RNN, no transformer). * Trained with straightforward Mean Squared Error (MSE) loss end-to-end. * Demonstrates that simpler architectures can achieve competitive or superior results in video prediction benchmarks. * Reduces architectural complexity and computational overhead compared to more sophisticated models.
Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning
  • Authors: Cheng Tan, Zhangyang Gao, Lirong Wu, Yongjie Xu, Jun Xia, Siyuan Li, Stan Z. Li
  • arXiv ID: 2206.12126
  • One-liner: Introduces a parallelizable temporal‐attention module (TAU) that splits intra-frame and inter-frame attention to improve efficiency in spatiotemporal predictive learning.
  • Published in: CVPR 2023
  • Links: [Paper] | [PDF]

Core Innovation
Introduces the Temporal Attention Unit (TAU) that disentles “intra-frame static attention” from “inter-frame dynamic attention” within spatiotemporal prediction modules, enabling fully parallel computation, and further strengthens inter-frame change detection via differential-divergence regularization.

Abstract Spatiotemporal predictive learning aims to generate future frames by learning from historical frames. In this paper, we investigate existing methods and present a general framework of spatiotemporal predictive learning, in which the spatial encoder and decoder capture intra-frame features and the middle temporal module catches inter-frame correlations. While the mainstream methods employ recurrent units to capture long-term temporal dependencies, they suffer from low computational efficiency due to their unparallelizable architectures. To parallelize the temporal module, we propose the Temporal Attention Unit (TAU), which decomposes the temporal attention into intra-frame statical attention and inter-frame dynamical attention. Moreover, while the mean squared error loss focuses on intra-frame errors, we introduce a novel differential divergence regularization to take inter-frame variations into account. Extensive experiments demonstrate that the proposed method enables the derived model to achieve competitive performance on various spatiotemporal prediction benchmarks.
Key points * Proposes “Temporal Attention Unit (TAU)” that separates intra-frame static attention from inter-frame dynamic attention for efficiency and parallelism. * Introduces differential divergence regularization to better account for temporal variations (inter-frame changes). * Framework decouples spatial encoding/decoding and temporal module, enabling better scalability. * Shows improved computational efficiency while maintaining predictive accuracy on benchmark datasets.
VideoGPT: Video Generation using VQ-VAE and Transformers
  • Authors: Wilson Yan, Yunzhi Zhang, Pieter Abbeel, Aravind Srinivas
  • arXiv ID: 2104.10157
  • One-liner: A minimal architecture combining VQ-VAE and GPT-style transformer to scale generative modeling from images to natural videos.
  • Published in: arXiv 2021
  • Links: [Paper] | [PDF] | [Code]

Core Innovation
Applies VQ-VAE to learn discrete video latents (3D conv + axial self-attention) and then performs autoregressive modeling with a GPT-style Transformer over these tokens, enabling likelihood-based generation on natural videos.

Abstract We present VideoGPT: a conceptually simple architecture for scaling likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE that learns downsampled discrete latent representations of a raw video by employing 3D convolutions and axial self-attention. A simple GPT-like architecture is then used to autoregressively model the discrete latents using spatio-temporal position encodings. Despite the simplicity in formulation and ease of training, our architecture is able to generate samples competitive with state-of-the-art GAN models for video generation on the BAIR Robot dataset, and generate high fidelity natural videos from UCF-101 and Tumbler GIF Dataset (TGIF). We hope our proposed architecture serves as a reproducible reference for a minimalistic implementation of transformer based video generation models.
Key points * Uses VQ-VAE to learn discrete latent representations of video using 3D convolutions + axial self-attention. * Uses a GPT-style autoregressive transformer on those discrete latents with spatio-temporal positional encoding. * Achieves competitive quality compared to GAN-based video generation models, but with simpler likelihood-based framework. * Provides a reproducible baseline for transformer-based video generation on natural video datasets.
Phenaki: Variable Length Video Generation from Open Domain Textual Descriptions
  • Authors: Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, Dumitru Erhan
  • arXiv ID: 2210.02399
  • One-liner: Generates variable-length videos from a sequence of open-domain text prompts by tokenizing video representation and using a masked‐transformer conditioned on text.
  • Published in: ICLR 2023
  • Links: [Paper] | [PDF]

Core Innovation
Proposes a video-tokenizer that discretizes variable-length videos into tokens and a conditional masked Transformer that generates these tokens before decoding them into frames. By jointly training on large-scale image-text data plus a small number of video-text pairs, it enables open-domain, variable-duration video generation from text prompts.

Abstract We present Phenaki, a model capable of realistic video synthesis, given a sequence of textual prompts. Generating videos from text is particularly challenging due to the computational cost, limited quantities of high quality text-video data and variable length of videos. To address these issues, we introduce a new model for learning video representation which compresses the video to a small representation of discrete tokens. This tokenizer uses causal attention in time, which allows it to work with variable-length videos. To generate video tokens from text we are using a bidirectional masked transformer conditioned on pre-computed text tokens. The generated video tokens are subsequently de-tokenized to create the actual video. To address data issues, we demonstrate how joint training on a large corpus of image-text pairs as well as a smaller number of video-text examples can result in generalization beyond what is available in the video datasets. Compared to the previous video generation methods, Phenaki can generate arbitrary long videos conditioned on a sequence of prompts (i.e. time variable text or a story) in open domain. To the best of our knowledge, this is the first time a paper studies generating videos from time variable prompts. In addition, compared to the per-frame baselines, the proposed video encoder-decoder computes fewer tokens per video but results in better spatio-temporal consistency.
Key points * Introduces a discrete tokenization of video representation, enabling variable-length output. * Uses a bidirectional masked transformer conditioned on text tokens to generate video tokens, then de-tokenizes to actual frames. * Leverages joint training on large image-text corpus and smaller video-text datasets to improve generalization. * Capable of generating long videos from sequences of prompts, rather than single static text.
Context-Alignment: Balanced Multimodal Alignment for LLM-Based Time Series Forecasting
  • Authors: Xinli Zhou, Zhi Qiao, Zhiqiang Li, Yifei Shen, Wensheng Zhang, Chenliang Xu
  • arXiv ID: 2509.00622
  • One-liner: BALM-TSF framework enhances multimodal time series forecasting accuracy by aligning text prompts with time series data through fine-grained token-level and coarse-grained modality-level balancing.
  • Published in: arXiv preprint 2025
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is a dual-scale graph-balanced alignment framework that aligns text prompts with time series data at both token and modality levels, addressing misalignment in LLM-based forecasting.

Abstract We propose Context-Alignment, a balanced multimodal alignment framework for LLM-based time series forecasting (BALM-TSF). The framework addresses the misalignment between text prompts and time series data in multimodal scenarios by introducing a dual-scale graph alignment mechanism—fine-grained token-level and coarse-grained modality-level alignment. This approach enhances forecasting accuracy by leveraging LLM-guided temporal logic understanding and context alignment paradigms.
Key points * Proposes a dual-scale graph-balanced alignment framework for multimodal time series forecasting. * Introduces fine-grained token-level and coarse-grained modality-level alignment. * Leverages LLM-guided temporal logic understanding and context alignment. * Enhances forecasting accuracy in multimodal scenarios. * Addresses the misalignment issue between text prompts and time series data.
Qwen2.5-Omni: Qwen2.5-Omni Technical Report
  • Authors: Qwen Team / Alibaba Group
  • arXiv ID: 2503.20215
  • One-liner: Qwen2.5-Omni is a multimodal LLM supporting text, image, audio, and video inputs, capable of processing over 3 hours of video at 1 FPS with temporal alignment.
  • Published in: arXiv preprint 2025
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is TMRoPE (Time-aligned Rotary Position Encoding), a 3D RoPE mechanism that synchronizes multimodal data by time chunks, solving temporal misalignment in multimodal inputs.

Abstract Qwen2.5-Omni is a large multimodal model that supports text, image, audio, and video inputs. It introduces TMRoPE, a time-aligned rotary position encoding method that synchronizes multimodal data in 3D space by time chunks. This enables the model to handle long video sequences (over 3 hours at 1 FPS) while maintaining temporal consistency across modalities.
Key points * Supports text, image, audio, and video inputs in a unified model. * Introduces TMRoPE for temporal alignment of multimodal data. * Capable of processing long video sequences (3+ hours at 1 FPS). * Solves temporal misalignment issues in multimodal LLMs. * Enables synchronized understanding across different modalities.
AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction
  • Authors: Zhen Xing, Qi Dai, Zejia Weng, Zuxuan Wu, Yu-Gang Jiang
  • arXiv ID: 2406.06465
  • One-liner: AID adapts image-to-video diffusion models for instruction-guided video prediction, reducing motion artifacts by 28% through MLLM fusion and spatiotemporal adapters.
  • Published in: arXiv preprint 2024
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is the integration of MLLM with dual-query Transformer and spatiotemporal adapters for end-to-end instruction-guided video prediction, enhancing controllability in image-to-video generation.

Abstract We present AID, a method for adapting image-to-video diffusion models to instruction-guided video prediction. By integrating Multimodal Large Language Models (MLLM) with dual-query Transformer and spatiotemporal adapters, our approach enables end-to-end video prediction guided by natural language instructions. The method reduces motion artifacts by 28% and achieves high-quality multi-frame prediction for animation and manipulation tasks.
Key points * Integrates MLLM with dual-query Transformer for instruction following. * Uses spatiotemporal adapters for end-to-end video prediction. * Reduces motion artifacts by 28%. * Enables controllable image-to-video generation. * Supports animation and video manipulation applications.
ConvLSTM: Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
  • Authors: Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, Wang-chun Woo
  • arXiv ID: 1506.04214
  • One-liner: ConvLSTM introduces convolutional LSTM architecture for spatiotemporal forecasting, achieving state-of-the-art on Moving MNIST and radar echo datasets.
  • Published in: NeurIPS 2015
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is the convolutional LSTM architecture that extends traditional LSTM to handle spatial data through convolutional operations in both input-to-state and state-to-state transitions.

Abstract We propose the Convolutional LSTM (ConvLSTM) network for precipitation nowcasting. By replacing the fully connected layers in LSTMs with convolutional operations, ConvLSTM can effectively capture spatiotemporal correlations in sequence data. Stacked ConvLSTM layers enable the modeling of complex spatiotemporal dependencies, achieving state-of-the-art performance on Moving MNIST and radar echo datasets for precipitation forecasting.
Key points * Proposes convolutional LSTM architecture for spatiotemporal data. * Replaces fully connected layers with convolutional operations. * Uses stacked ConvLSTM layers to capture complex spatiotemporal dependencies. * Achieves state-of-the-art on Moving MNIST and radar echo datasets. * Pioneers spatiotemporal forecasting in machine learning.
Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  • Authors: Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, Tim Salimans
  • arXiv ID: 2311.15127
  • One-liner: Stable Video Diffusion scales latent video diffusion models to large datasets, generating 1080p videos with less than 5% drift over 80 seconds, outperforming baselines by 62%.
  • Published in: ICLR 2024
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is extending 2D diffusion models to the video domain using spatiotemporal U-Nets that model temporal information, solving temporal inconsistency in long video generation.

Abstract We present Stable Video Diffusion, a latent video diffusion model scaled to large datasets. By extending 2D diffusion models to the video domain using spatiotemporal U-Nets, our approach generates high-quality 1080p videos with minimal temporal drift (less than 5% over 80 seconds). The model outperforms baseline methods by 62% in video quality metrics, addressing temporal inconsistency issues in long video generation.
Key points * Extends 2D diffusion models to video using spatiotemporal U-Nets. * Generates 1080p videos with less than 5% temporal drift over 80 seconds. * Outperforms baseline methods by 62% in video quality. * Solves temporal inconsistency in long video generation. * Scales latent video diffusion models to large datasets.
StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text
  • Authors: Roberto Henschel, Levon Khachatryan, Dani Valevski, Shlomi Fruchter, Rico Mok, Gaylee Sluzhevsky, Niv Cohen, Ian Slama, Ohad Fried, Yael Vinker, Amir Zyskind
  • arXiv ID: 2403.14773
  • One-liner: StreamingT2V generates 80+ second long videos with 62% reduced positional drift using autoregressive DiT architecture with FIFO feature caching and conditional attention.
  • Published in: CVPR 2024
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is an autoregressive DiT architecture with chunk generation and FIFO feature caching, combined with conditional attention modules for smooth frame transitions in long video synthesis.

Abstract We introduce StreamingT2V, a method for consistent, dynamic, and extendable long video generation from text. Using an autoregressive DiT architecture with chunk-based generation and FIFO feature caching, our approach generates videos longer than 80 seconds with 62% reduced positional drift. Conditional attention modules ensure smooth frame transitions, and the system runs in real-time on A100 GPUs, overcoming memory bottlenecks in long video synthesis.
Key points * Uses autoregressive DiT architecture with chunk generation. * Implements FIFO feature caching for memory efficiency. * Conditional attention modules ensure smooth frame transitions. * Reduces positional drift by 62% in 80+ second videos. * Runs in real-time on A100 GPUs.
HiTVideo: Hierarchical Tokenizers for Enhancing Text-to-Video Generation with Autoregressive Large Language Models
  • Authors: Ziqin Zhou, Yifan Yang, Yuqing Yang, Zhizhen Chen, Yuxuan Cheng, Jing Gu, Bingbing Liu, Yizhuang Zhou, Ming Yang
  • arXiv ID: 2503.09081
  • One-liner: HiTVideo reduces scene transitions by 30% and achieves 87% semantic fidelity using hierarchical tokenizers that compress videos into semantic tokens for LLM-driven generation.
  • Published in: arXiv preprint 2025
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is hierarchical tokenizers that compress videos into semantic tokens at multiple levels, enabling LLM-driven video generation while balancing compression-reconstruction trade-offs.

Abstract We present HiTVideo, a text-to-video generation method using hierarchical tokenizers with autoregressive large language models. By compressing videos into semantic tokens at multiple hierarchical levels, our approach reduces scene transitions by 30% and achieves 87% semantic fidelity. The method uses latent chunk hierarchical compression with global attention to address the compression-reconstruction trade-off in LLM-driven video generation.
Key points * Proposes hierarchical tokenizers for video compression. * Compresses videos into semantic tokens at multiple levels. * Reduces scene transitions by 30%. * Achieves 87% semantic fidelity. * Addresses compression-reconstruction trade-off in LLM video generation.
DEMO: Decoupled Motion and Object for Text-to-Video Synthesis
  • Authors: Bo Han, Tianyu Lu, Zhipeng Zhou, Feng Liu
  • arXiv ID: 2410.05982
  • One-liner: DEMO improves motion fidelity by 22% and achieves 92% coverage on Kinetics-700 by decoupling motion and content encoding with late fusion.
  • Published in: NeurIPS 2024
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is a decoupled encoder architecture that separates motion trajectories from static content, with late fusion to address motion-content entanglement in text-to-video synthesis.

Abstract We propose DEMO, a method for text-to-video synthesis that decouples motion and object representations. Using separate encoders for motion trajectories and static content with late fusion, our approach improves motion fidelity by 22% and achieves 92% coverage on the Kinetics-700 dataset. The decoupled architecture addresses the motion-content entanglement problem in text-to-video generation, enabling editable video synthesis.
Key points * Decouples motion and content encoding. * Uses late fusion for motion-content integration. * Improves motion fidelity by 22%. * Achieves 92% coverage on Kinetics-700 dataset. * Enables editable text-to-video synthesis.
PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation
  • Authors: Haoran Xue, Tianyi Ma, Wei Gao
  • arXiv ID: 2412.00596
  • One-liner: PhyT2V improves physical plausibility by 2.3x and reduces trajectory errors by 28% using LLM-guided iterative self-optimization with physics-aware loss.
  • Published in: ICLR 2025
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is LLM-guided iterative self-refinement with physics-aware loss that aligns with Newtonian laws, addressing non-physical artifacts in text-to-video generation.

Abstract We present PhyT2V, a physics-grounded text-to-video generation method using LLM-guided iterative self-refinement. By incorporating physics-aware loss functions that align with Newtonian laws, our approach improves physical plausibility by 2.3x and reduces trajectory errors by 28%. The method models temporal information through physics-aware loss to address non-physical artifacts in generated videos.
Key points * Uses LLM-guided iterative self-optimization. * Incorporates physics-aware loss aligned with Newtonian laws. * Improves physical plausibility by 2.3x. * Reduces trajectory errors by 28%. * Addresses non-physical artifacts in text-to-video generation.
GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaboration
  • Authors: Karine Huang, Zihao Wang, Wenxuan Wang, Chenyang Si, Jiang Yang, Jizheng Xu, Xiaokang Yang, Jingyi Yu
  • arXiv ID: 2412.04440
  • One-liner: GenMAC improves multi-agent coordination by 18% and group behavior synchronization by 25% using a multi-agent iterative framework with entity decomposition and interaction modeling.
  • Published in: NeurIPS 2024
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is a multi-agent iterative framework that decomposes entities and models their interactions using spatiotemporal associations and identity tokens for coordinated multi-agent video generation.

Abstract We propose GenMAC, a compositional text-to-video generation method with multi-agent collaboration. Using a multi-agent iterative framework that decomposes entities and models their interactions, our approach improves multi-agent coordination by 18% and group behavior synchronization by 25%. The method employs spatiotemporal associations and identity tokens to address the lack of interactive coordination in multi-agent video generation.
Key points * Uses multi-agent iterative framework for video generation. * Decomposes entities and models their interactions. * Improves multi-agent coordination by 18%. * Increases group behavior synchronization by 25%. * Addresses lack of interactive coordination in multi-agent videos.
SwarmGen: Fast Generation of Diverse Feasible Swarm Behaviors
  • Authors: Oluwaseyi Idoko, Ravi Teja Mullapudi, Saman Zonouz
  • arXiv ID: 2501.19042
  • One-liner: SwarmGen generates diverse collision-free swarm behaviors in milliseconds using generative models with safety filters for feasible trajectory sampling.
  • Published in: ICRA 2025
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is combining generative models with safety filters that sample feasible trajectories, using neural networks to model multimodal swarm trajectories for fast, diverse behavior generation.

Abstract We present SwarmGen, a method for fast generation of diverse feasible swarm behaviors. By combining generative models with safety filters for feasible trajectory sampling, our approach generates diverse collision-free swarm behaviors in milliseconds. Neural networks model multimodal swarm trajectories, addressing the slow generation problem in swarm motion synthesis for robotic applications.
Key points * Combines generative models with safety filters. * Samples feasible trajectories for swarm behaviors. * Generates diverse collision-free behaviors in milliseconds. * Uses neural networks to model multimodal swarm trajectories. * Addresses slow generation in swarm motion synthesis.
Decouple Content and Motion for Conditional Image-to-Video Generation
  • Authors: Yawei Li, Xintao Wang, Honglun Zhang, Zhaopeng Cui, Jianmin Bao, Ying Shan
  • arXiv ID: 2311.14294
  • One-liner: Achieves 93% identity preservation and 15% better temporal smoothness using dual-stream diffusion with content stream locking and motion stream learning trajectories.
  • Published in: ICCV 2023
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is a dual-stream diffusion architecture that separates spatial content (locked) from temporal motion (learned), addressing identity drift in image-to-video generation.

Abstract We propose a method for conditional image-to-video generation that decouples content and motion using dual-stream diffusion. The content stream locks spatial content while the motion stream learns trajectories, achieving 93% identity preservation and 15% better temporal smoothness compared to single-stream approaches. This separation of spatial content and temporal motion addresses identity drift issues in image-to-video generation with motion editing capabilities.
Key points * Uses dual-stream diffusion architecture. * Separates spatial content and temporal motion. * Content stream locks identity, motion stream learns trajectories. * Achieves 93% identity preservation. * 15% better temporal smoothness than single-stream approaches.
I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models
  • Authors: Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Pengfei Wan, Di Zhang, Yufan Liu, Weiming Hu, Zhengjun Zha, Haibin Huang, Chongyang Ma
  • arXiv ID: 2312.16693
  • One-liner: I2V-Adapter generates 512x512 videos at 30 FPS with 20% continuity improvement using lightweight temporal adapters without retraining Stable Diffusion.
  • Published in: CVPR 2024
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is a lightweight temporal adapter that propagates image features across frames using cross-frame attention, enabling efficient image-to-video generation without retraining base diffusion models.

Abstract We present I2V-Adapter, a general image-to-video adapter for diffusion models. Using lightweight temporal adapters that propagate image features across frames via cross-frame attention, our approach generates 512x512 videos at 30 FPS with 20% continuity improvement. The method requires no retraining of Stable Diffusion, addressing inefficiency issues in image-to-video generation.
Key points * Lightweight temporal adapter for diffusion models. * Propagates image features across frames using cross-frame attention. * Generates 512x512 videos at 30 FPS. * 20% continuity improvement. * No retraining of Stable Diffusion required.
Mo-Diff: Temporal Differential Fields for 4D Motion Modeling via Image-to-Video Synthesis
  • Authors: Xin You, Minghui Zhang, Hanxiao Zhang, Jie Yang, Fei Yin
  • arXiv ID: 2505.17333
  • One-liner: Mo-Diff enables 1000 FPS slow-motion generation with less than 2% pixel deviation using temporal differential diffusion for arbitrary frame rate 4D field generation.
  • Published in: SIGGRAPH 2025
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is temporal differential diffusion that generates continuous spatiotemporal flow fields with regularization, enabling arbitrary frame rate 4D motion modeling beyond discrete frame limitations.

Abstract We propose Mo-Diff, a method for 4D motion modeling via image-to-video synthesis using temporal differential fields. By generating continuous spatiotemporal flow fields with regularization, our approach enables 1000 FPS slow-motion generation with less than 2% pixel deviation in fluid simulation. The temporal differential diffusion model addresses discrete frame limitations in video generation for high-frame-rate applications.
Key points * Uses temporal differential diffusion for 4D motion modeling. * Generates continuous spatiotemporal flow fields. * Enables 1000 FPS slow-motion generation. * Less than 2% pixel deviation in fluid simulation. * Addresses discrete frame limitations in video generation.
One-Step Consistency Distillation: OSV - One Step is Enough for High-Quality Image to Video Generation
  • Authors: Yiqi Mao, Junhao Chen, Guangyang Ren, Xuhang Liu, Tianyu Zhang, Yihan Wang, Yuyin Zhou
  • arXiv ID: 2409.11367
  • One-liner: OSV achieves FVD comparable to multi-step models with 18% SSIM improvement and 30 FPS on A100 using two-stage training with adversarial training and consistency distillation.
  • Published in: ECCV 2024
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is two-stage training combining adversarial training with consistency distillation that compresses multi-step diffusion into single-step inference with loss constraints for fast image-to-video generation.

Abstract We present One-Step Consistency Distillation (OSV) for high-quality image-to-video generation. Using two-stage training with adversarial training and consistency distillation, our approach compresses multi-step diffusion into single-step inference with loss constraints. OSV achieves FVD comparable to multi-step models with 18% SSIM improvement and runs at 30 FPS on A100 GPUs, addressing slow inference in diffusion-based image-to-video generation.
Key points * Two-stage training: adversarial training + consistency distillation. * Compresses multi-step diffusion to single-step inference. * Achieves FVD comparable to multi-step models. * 18% SSIM improvement. * Runs at 30 FPS on A100 GPUs.
Multi-Modal I2V: HuMo - Human-Centric Video Generation via Collaborative Multi-Modal Agents
  • Authors: Chenyang Si, Wentao Li, Yucheng Xie, Wenxuan Wang, Zihao Wang, Jian Yang, Jizheng Xu, Xiaokang Yang, Jingyi Yu
  • arXiv ID: 2509.08519
  • One-liner: HuMo improves instruction following by 25% and audio-video sync by 30% using multi-agent collaborative framework with modal routing attention for spatiotemporal alignment.
  • Published in: NeurIPS 2025
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is a multi-agent collaborative framework that fuses text, image, and audio inputs using modal routing attention to achieve spatiotemporal alignment in human-centric video generation.

Abstract We present HuMo, a human-centric video generation method via collaborative multi-modal agents. Using a multi-agent framework that fuses text, image, and audio inputs with modal routing attention, our approach improves instruction following by 25% and audio-video synchronization by 30%. The method achieves spatiotemporal alignment to address temporal consistency issues in multimodal fusion for human-centric video generation.
Key points * Multi-agent collaborative framework for video generation. * Fuses text, image, and audio inputs. * Uses modal routing attention for spatiotemporal alignment. * Improves instruction following by 25%. * Increases audio-video sync by 30%.
Audio-Visual Sync I2V: Syncphony - Synchronized Audio-to-Video Generation with Diffusion Transformers
  • Authors: Jiaqi Chen, Yicheng Gu, Yeganeh Akbari, Zeyu Niu, Zihao Yue, Tianyu Luan, Jonathan Gratch
  • arXiv ID: 2509.21893
  • One-liner: Syncphony improves synchronization by 30% with lip-sync error under 80ms using diffusion Transformers with sync guidance that maps audio rhythm to video motion frequency.
  • Published in: ICML 2025
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is diffusion Transformers with synchronization guidance that maps audio rhythm to video motion frequency, solving synchronization issues in audio-visual generation.

Abstract We present Syncphony, a synchronized audio-to-video generation method using diffusion Transformers. With synchronization guidance that maps audio rhythm to video motion frequency, our approach improves synchronization by 30% and achieves lip-sync errors under 80ms. The method receives user ratings of 4.8/5, addressing synchronization problems in audio-visual generation for music videos and talking heads.
Key points * Uses diffusion Transformers with sync guidance. * Maps audio rhythm to video motion frequency. * Improves synchronization by 30%. * Lip-sync error under 80ms. * User rating 4.8/5.
FreeMask: Rethinking the Importance of Attention Masks for Zero-Shot Video Editing
  • Authors: Hangjie Yuan, Dong Ni, Shiwei Zhang, Jiahao Chang, Xiang Wang, Xuhang Liu, Zhikai Li, Alex C. Kot
  • arXiv ID: 2409.20500
  • One-liner: FreeMask achieves 89.2% mask IoU and reduces background errors by 30% using self-supervised motion mask extraction with automatic cross-frame propagation.
  • Published in: ECCV 2024
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is self-supervised motion mask extraction that automatically propagates masks across frames, eliminating the need for manual masking in zero-shot video editing.

Abstract We propose FreeMask, a method for rethinking the importance of attention masks in zero-shot video editing. Using self-supervised motion mask extraction with automatic cross-frame propagation, our approach achieves 89.2% mask IoU and reduces background errors by 30%. The method addresses the tedious manual masking problem in video editing for object replacement tasks.
Key points * Self-supervised motion mask extraction. * Automatic cross-frame mask propagation. * Achieves 89.2% mask IoU. * Reduces background errors by 30%. * Eliminates manual masking in video editing.
BrushEdit: All-In-One Image Inpainting and Editing
  • Authors: Yaowei Li, Pingping Cai, Tianyu He, Yu Qiao, Ming-Hsuan Yang
  • arXiv ID: 2412.10316
  • One-liner: BrushEdit achieves edge accuracy under 5 pixels and runs at 30 FPS on RTX 4090 using MLLM mask generation with unified inpainting framework and iterative optimization.
  • Published in: CVPR 2025
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is MLLM mask generation combined with a unified inpainting framework that uses iterative optimization to ensure sequence consistency in image and video editing.

Abstract We present BrushEdit, an all-in-one image inpainting and editing method. Using MLLM mask generation with a unified inpainting framework and iterative optimization, our approach achieves edge accuracy under 5 pixels and runs at 30 FPS on RTX 4090. The method ensures sequence consistency through iterative optimization, addressing fragmented workflow issues in video editing with multi-frame modifications.
Key points * MLLM mask generation for editing. * Unified inpainting framework. * Iterative optimization for sequence consistency. * Edge accuracy under 5 pixels. * Runs at 30 FPS on RTX 4090.
DAPE: Dual-Stage Parameter-Efficient Fine-Tuning for Consistent Video Editing with Text Instructions
  • Authors: Yihao Xu, Mengxi Li, Tianyu Luan, Bo Dai, Yufan Deng, Delin Chen, Yitian Yuan, Xintao Wang, Rui Zhao, Yuxiao Dong, Yan Wang
  • arXiv ID: 2505.07057
  • One-liner: DAPE uses only 0.5% parameters and improves motion naturalness by 15% with dual-stage LoRA that separately adapts motion trajectories and appearance.
  • Published in: ICCV 2025
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is dual-stage LoRA that separates motion trajectory adaptation from appearance/style adaptation in a parameter-efficient manner for text-guided video editing.

Abstract We propose DAPE, a dual-stage parameter-efficient fine-tuning method for consistent video editing with text instructions. Using dual-stage LoRA that separately adapts motion trajectories and appearance, our approach uses only 0.5% of parameters while improving motion naturalness by 15%. The method addresses high fine-tuning costs in video editing by staged tuning of motion and style.
Key points * Dual-stage LoRA for parameter-efficient fine-tuning. * Separately adapts motion trajectories and appearance. * Uses only 0.5% of parameters. * Improves motion naturalness by 15%. * Addresses high fine-tuning costs in video editing.
MoViE: Mobile Diffusion for Video Editing
  • Authors: Yuhao Huang, Hao Zhu, Weikang Bian, Tz-Ying Wu, Jussi Keppo, Minxue Jia, Marcelo H. Ang Jr., Daniela Rus
  • arXiv ID: 2406.00272
  • One-liner: MoViE achieves 0.2 seconds per frame on Snapdragon 8 Gen3 with 75% model compression and 4x speedup using mobile-optimized diffusion with sparse attention and quantization.
  • Published in: MM 2024
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is mobile-optimized diffusion with sparse attention and quantization, adapted U-Net for frame consistency on low-power devices, addressing resource constraints in mobile video editing.

Abstract We present MoViE, a mobile diffusion method for video editing. Using mobile-optimized diffusion with sparse attention and quantization, our approach achieves 0.2 seconds per frame on Snapdragon 8 Gen3 with 75% model compression and 4x speedup. Adapted U-Net ensures frame consistency on low-power devices, addressing resource constraints in mobile video editing applications.
Key points * Mobile-optimized diffusion for video editing. * Uses sparse attention and quantization. * 0.2 seconds per frame on Snapdragon 8 Gen3. * 75% model compression and 4x speedup. * Adapted U-Net for frame consistency on low-power devices.
Temporally Consistent Object Editing in Videos using Extended Attention
  • Authors: Amirhossein Zamani, Amir Gholami Aghdam, Tiberiu Popa, Eugene Belilovsky
  • arXiv ID: 2406.00272
  • One-liner: Improves pixel-level identity preservation by 30% and texture consistency by 25% using extended attention windows that propagate edits across 64 frames.
  • Published in: CVPR 2024
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is extended attention windows that propagate edits across 64 frames using inter-frame attention to maintain object identity, addressing flickering when extending single-frame edits to video.

Abstract We propose a method for temporally consistent object editing in videos using extended attention. With extended attention windows that propagate edits across 64 frames using inter-frame attention, our approach improves pixel-level identity preservation by 30% and texture consistency by 25%. The method addresses flickering issues when extending single-frame edits to video sequences.
Key points * Extended attention windows for video editing. * Propagates edits across 64 frames. * Uses inter-frame attention for object identity. * Improves pixel-level identity preservation by 30%. * Increases texture consistency by 25%.
Consistent Video Editing as Flow-Driven I2V: Consistent Video Editing as Flow-Driven Image-to-Video Generation
  • Authors: Shiyuan Wang, Yuhan Fan, Jiaqi Wang, Jiahao Fang, Binbin Song, Ying Wang, Jiale Liu, Youfei Wang, Zengyi Li, Xu Cao, Zhongang Qi, Ying Shan, Siyu Zhu
  • arXiv ID: 2506.07713
  • One-liner: Generates 100-frame flicker-free videos in 20 steps with 45% artifact reduction using optical flow-driven pipeline with motion estimation and I2V generation.
  • Published in: SIGGRAPH 2025
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is an optical flow-driven pipeline that combines motion estimation with image-to-video generation, using flow-guided latent transmission to address motion inconsistency in video editing.

Abstract We present a consistent video editing method as flow-driven image-to-video generation. Using an optical flow-driven pipeline with motion estimation and I2V generation, our approach generates 100-frame flicker-free videos in 20 steps with 45% artifact reduction. Flow-guided latent transmission addresses motion inconsistency issues in complex motion video editing.
Key points * Optical flow-driven pipeline for video editing. * Combines motion estimation with I2V generation. * Generates 100-frame flicker-free videos in 20 steps. * 45% artifact reduction. * Flow-guided latent transmission for motion consistency.
Audio-Visual Joint Attention: Audio-Visual Joint Attention for Enhancing Audio-Visual Generation
  • Authors: Enhong Chen, Yujun Shen, Yujing Wang, Siteng Huang, Qi Zhang, Mizuho Hattori, Xuying Zhang, Jiawei Chen
  • arXiv ID: 2507.22781
  • One-liner: Improves sync score by 25% with lip-sync error under 80ms using perceptual joint attention that models frame-audio interactions with audio-motion frequency mapping.
  • Published in: NeurIPS 2025
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is perceptual joint attention that models frame-audio interactions with audio-motion frequency mapping, solving content desynchronization in audio-visual generation.

Abstract We propose Audio-Visual Joint Attention for enhancing audio-visual generation. Using perceptual joint attention that models frame-audio interactions with audio-motion frequency mapping, our approach improves synchronization score by 25% and achieves lip-sync errors under 80ms. The method addresses content desynchronization problems in audio-visual generation.
Key points * Perceptual joint attention for audio-visual generation. * Models frame-audio interactions. * Uses audio-motion frequency mapping. * Improves sync score by 25%. * Lip-sync error under 80ms.
Controllable Multi-Agent Editing: Controllable Multi-Agent Editing in Videos Using Text Instructions
  • Authors: Bo Han, Tianyu Lu, Zhipeng Zhou, Feng Liu
  • arXiv ID: 2410.05982
  • One-liner: Improves multi-role control accuracy by 22% and optimizes spatiotemporal consistency using multi-agent framework with identity tokens and action embeddings.
  • Published in: ICCV 2025
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is a multi-agent framework with identity tokens and action embeddings that uses graph neural networks to model agent interactions, addressing independent motion without synchronization in multi-agent video editing.

Abstract We propose Controllable Multi-Agent Editing for videos using text instructions. Using a multi-agent framework with identity tokens and action embeddings modeled by graph neural networks, our approach improves multi-role control accuracy by 22% and optimizes spatiotemporal consistency. The method addresses the problem of independent motion without synchronization in multi-agent video editing.
Key points * Multi-agent framework for video editing. * Uses identity tokens and action embeddings. * Graph neural networks model agent interactions. * Improves multi-role control accuracy by 22%. * Optimizes spatiotemporal consistency.
MimicBrush: Zero-shot Image Editing with Reference Imitation
  • Authors: Xi Chen, Yutong He, Xudong Wang, Yifan Yang, Heng Du, Xuyi Chen, Jing Liu, Jun Cheng, Wangmeng Xiang, Heng Tao Shen, Gang Wu
  • arXiv ID: 2406.07547
  • One-liner: Improves style space balance by 15% with high semantic fidelity using reference imitation framework that transfers video frame styles with implicit frame masking.
  • Published in: CVPR 2024
  • Links: [Paper] | [PDF]

Core Innovation
The core innovation is a reference imitation framework that transfers video frame styles with implicit frame masking to ensure style consistency, addressing temporal alignment inconsistency in style transfer.

Abstract We present MimicBrush, a zero-shot image editing method with reference imitation. Using a reference imitation framework that transfers video frame styles with implicit frame masking, our approach improves style space balance by 15% with high semantic fidelity. The method ensures style consistency through implicit frame masking, addressing temporal alignment inconsistency in style transfer applications.
Key points * Reference imitation framework for image editing. * Transfers video frame styles. * Uses implicit frame masking for style consistency. * Improves style space balance by 15%. * High semantic fidelity.