Image-to-Video in Stable Video Diffusion

Definition

The paper presents Stable Video Diffusion, a latent video diffusion model for high-resolution text-to-video and image-to-video generation. The paper describes image-to-video as a task in which the model predicts a sequence of frames from a single conditioning image. The paper shows that its base model provides a powerful motion representation for downstream tasks such as image-to-video generation, and adaptability to camera motion-specific LoRA modules.

The paper identifies and evaluates the stages for successful training of video LDMs: text-to-image pretraining, video pretraining, and high-quality video finetuning. The paper demonstrates the necessity of a well-curated pretraining dataset for generating high-quality videos and presents a systematic curation process, including captioning and filtering strategies, to train a strong base model.

At a glance

The paper grounds its initial model on a pretrained image diffusion model, namely Stable Diffusion 2.1, to equip it with a strong visual representation. Following the architecture proposed in Blattmann et al., the paper inserts temporal convolution and attention layers after every spatial convolution and attention layer.

Stage I in the paper is image pretraining, i.e. a 2D text-to-image diffusion model.
Stage II in the paper is video pretraining, which trains on large amounts of videos.
Stage III in the paper is video finetuning, which refines the model on a small subset of high-quality videos at higher resolution.
The paper can finetune its base text-to-video model for the image-to-video task.
The paper does not use text-conditioning, but replaces text embeddings fed into the base model with the CLIP image embedding of the conditioning frame.
The paper concatenates a noise-augmented version of the conditioning frame channel-wise to the input of the UNet.
The paper trains two versions: one generating 14 frames and one generating 25 frames.

The source does not state the order of these points, so they are listed without arrows.

How it works

The paper finetunes its base model for image-to-video generation, where the video model receives a still input image as a conditioning. The paper does not use masking techniques and simply copies the frame across the time axis. The paper occasionally found that standard vanilla classifier-free guidance can lead to artifacts: too little guidance may result in inconsistency with the conditioning frame, while too much guidance can result in oversaturation.

Instead of using a constant guidance scale, the paper linearly increases the guidance scale across the frame axis, from small to high. The paper finetunes two image-to-video models, one predicting 14 frames and another predicting 25 frames. The paper states that the final checkpoints are chosen using a combination of visual inspection and human evaluation.

During training, the paper's base model and its high-resolution Text/Image-to-Video models are conditioned on the input video's frame rate and motion score. The paper says this conditioning allows it to vary the amount of motion in a generated video at inference time. The paper's background states that Video-LDMs train the main generative model in a latent space of reduced computational complexity, while the paper finetunes the full model, in contrast to works that only train temporal layers or are completely training-free.

The paper's limitations section says videos generated with its approach sometimes suffer from too little generated motion, and that video diffusion models are typically slow to sample and have high VRAM requirements, with its model being no exception.

Example

To facilitate controlled camera motion in image-to-video generation, the paper trains a variety of camera motion LoRAs within the temporal attention blocks of its model. The paper uses subsets of the data in which camera motion is categorized as horizontally moving, zooming, and static. The paper trains these additional parameters on a small dataset with rich camera-motion metadata.

For the camera motion LoRAs, the paper trains low-rank matrices of rank 16 for 5k iterations. The paper also finetunes its image-to-video model on multi-view datasets to obtain multiple novel views of an object simultaneously. The paper's related-work appendix states that, when combined with off-the-shelf text-to-image models, image-to-video models can be used to obtain a full text-(to-image)-to-video pipeline.

Related tools listed on this site.