SadTalker: Stylized Audio-Driven Talking-Head Video Generation System

Definition

The paper states that generating talking head videos through a face image and a piece of speech audio still contains many challenges. The paper proposes SadTalker, a stylized audio-driven talking-head video generation system through implicit 3D coefficient modulation. The paper says SadTalker generates 3D motion coefficients (head pose, expression) of the 3DMM from audio and implicitly modulates a novel 3D-aware face render for talking head generation.

The paper considers the motion coefficients of the 3DMM as the intermediate representation and divides its task into two major components. The paper observes that the 3D facial model contains a highly decoupled representation and can be used to learn each type of motion individually.

At a glance

The paper states that its motion coefficients (70) only contain the expression (64) and head pose (6).

The paper first extracts the coefficients from the original image.
The paper presents ExpNet to learn the accurate facial expression from audio by distilling both coefficients and 3D-rendered faces.
The paper designs PoseVAE via a conditional VAE to synthesize head motion in different styles.
The generated 3D motion coefficients are mapped to the unsupervised 3D keypoints space of the proposed face render, and the final video is synthesized.
The paper trains each sub-network of expression generation, head poses generation and face renderer individually, and its system can be inferred in an end-to-end style.
The paper says the proposed SadTalker produces diverse, realistic, synchronized talking videos from an input audio and a single reference image.

The source does not state the order of these points, so they are listed without arrows.

How it works

The paper also says that explicitly using 3D information suffers problems of stiff expression and incoherent video. The paper's mappingNet is built via several 1D convolutional layers. The paper states that both the encoder and decoder of PoseVAE contain several linear layers.

For training, the paper first trains face-vid2vid in a self-supervised fashion as in the original paper. Then, the paper trains the mapping net on the 3DMM coefficients of the ground truth video in a reconstruction style. ExpNet uses continuous 5 frames to learn.

PoseVAE is learned via continuous 32 frames. The paper notes that previous works mainly focus on generating lip motion since it has a strong connection with speech. The paper says PC-AVS disentangles the head pose and expression using implicit latent code, while Audio2Head can only generate the front talking face.

Example

The paper uses the VoxCeleb dataset for training, which contains over 100k videos of 1251 subjects. The paper says it generates overall 20 videos as its testing in the user studies. The paper says it conducted extensive experiments to demonstrate the superiority of its method in terms of motion and video quality.

The paper states that experiments show its method achieves state-of-the-art performance in terms of motion synchronization and video quality. The paper notes that since 3DMMs do not model the variation of eyes and teeth, the mappingNet in its Face Render will also struggle to synthesize the realistic teeth in some cases. The paper names another limitation: it only concerns the lip motion and eye blinking other than the other facial expressions, e.g., emotion and gaze direction, so the generated video has a fixed emotion, which also reduces the realism of generated content.

The paper states that both visible and invisible video watermarks will be inserted into the produced video for generated content identification.

Related tools listed on this site.