GenLCA: 3D Diffusion for Full-Body Avatars from In-the-Wild Videos

We present GenLCA, a diffusion-based generative modelfor generating and editing photorealistic full-body avatars from text andimage inputs. The generated avatars are faithful to the inputs, whilesupporting high-fidelity facial and full-body animations. The core ideais a novel paradigm that enables training a full-body 3D diffusion modelfrom partially observable 2D data, allowing the training dataset to scaleto millions of real-world videos. This scalability contributes to the supe-rior photorealism and generalizability of GenLCA. Specifically, we scaleup the dataset by repurposing a pretrained feed-forward avatar recon-struction model as an animatable 3D tokenizer, which encodes unstruc-tured video frames into structured 3D tokens. However, most real-worldvideos only provide partial observations of body parts, resulting in ex-cessive blurring or transparency artifacts in the 3D tokens. To addressthis, we propose a novel visibility-aware diffusion training strategy thatreplaces invalid regions with learnable tokens and computes losses onlyover valid regions. We then train a flow-based diffusion model on the to-ken dataset, inherently maintaining the photorealism and animatabilityprovided by the pretrained avatar reconstruction model. Our approacheffectively enables the use of large-scale real-world video data to traina diffusion model natively in 3D. We demonstrate the efficacy of ourmethod through diverse and high-fidelity generation and editing results,outperforming existing solutions by a large margin.