XAGen: 3D Expressive Human Avatars Generation


TL;DR: XAGen is a 3D-aware generative model that enables human synthesis with high-fidelity appearance and geometry, together with disentangled controllability for body, face, and hand.

XAGen can synthesize realistic 3D avatars with detailed geometry, while providing disentangled control over expressive attributes, i.e., facial expressions, jaw poses, body poses, and hand poses.


Links
PDF Cover
github icon
Paper Code
Abstract
Recent advances in 3D-aware GAN models have enabled the generation of realistic and controllable human body images. However, existing methods focus on the control of major body joints, neglecting the manipulation of expressive attributes, such as facial expressions, jaw poses, hand poses, and so on. In this work, we present XAGen, the first 3D generative model for human avatars capable of expressive control over body, face, and hands. To enhance the fidelity of small-scale regions like face and hands, we devise a multi-scale and multi-part 3D representation that models fine details. Based on this representation, we propose a multi-part rendering technique that disentangles the synthesis of body, face, and hands to ease model training and enhance geometric quality. Furthermore, we design multi-part discriminators that evaluate the quality of the generated avatars with respect to their appearance and fine-grained control capabilities. Experiments show that XAGen surpasses state-of-the-art methods in terms of realism, diversity, and expressive control abilities. Code and data will be made available on this Project Page.
Appearance / Geometry Visualization
Drag the blue separator to see the pixel-aligned RGB and geometry. Refresh the page if RGB and geometry are out of sync.
Novel Pose Generation
Drag the blue separator to see the pixel-aligned RGB and geometry. Refresh the page if RGB and geometry are out of sync.

Given different SMPL-X sequences with various hand, jaw, and body poses, XAGen can animate the generated 3D avatars accordingly.

Text-guided Synthesis
brown hair, red T-shirt, blue jeans blonde hair, pink T-shirt, black trousers
Audio-driven Animation

We use the audio stream and corresponding SMPL-X sequence provided by an open-source audio-to-motion method TalkSHOW to animate our synthesized avatars (video with sound). Please refer to Section 4.3 for more details.




We show more audio-driven animation results for the avatars synthesized by XAGen with multi-view rendering. These videos contain audio.

Method Overview
Bibtex 
@inproceedings{XAGen2023,
    title={XAGen: 3D Expressive Human Avatars Generation},
    author={Xu, Zhongcong and Zhang, Jianfeng and Liew, Junhao and Feng, Jiashi and Shou, Mike Zheng},
    booktitle={NeurIPS},
    year={2023}
}