AniPortrait

What is AniPortrait?

AniPortrait is a novel framework for audio-driven synthesis of photorealistic portrait animations. Developed by researchers at Tencent Games Zhiji and Tencent, it generates high-quality animations from a single reference portrait image and an audio input. The system also supports video-driven face reenactment, allowing substantial pose differences between the reference image and source video.

Key features include audio-to-video generation, where the model synchronizes lip movements and facial expressions with the provided audio, and video-to-video retargeting with a new pose retarget strategy for flexible pose control. The framework incorporates a frame interpolation module to accelerate inference, enabling faster video generation. It also offers an audio2pose model and pre-trained weights for audio-to-video tasks, providing head pose control through generated pose_temp.npy files.

Benefits include photorealistic output, real-time or near-real-time performance with acceleration options, and ease of use via a Gradio demo on Hugging Face Spaces. Use cases span virtual avatars, digital humans, video dubbing, and interactive applications requiring realistic facial animation from audio.

Technically, AniPortrait builds on Stable Diffusion V1.5 and includes components such as denoising_unet, reference_unet, pose_guider, motion_module, audio2mesh, audio2pose, and film_net for frame interpolation. It requires Python >=3.10 and CUDA 11.7. Weights are organized under ./pretrained_weights and can be downloaded from provided sources. The framework is open-source on GitHub, with ongoing updates and community support.

Use Cases

Key Features

audio driven animation photorealistic portrait synthesis single reference image video driven face reenactment lip sync generation pose retarget strategy frame interpolation head pose control Gradio demo pre trained weights

Who is it for?

AI researchers, game developers, VFX artists, virtual avatar creators, digital human developers, video dubbing professionals, interactive application developers

Similar Tools