CtrlCoMo: Controllable Co-Speech Motion Generation with Gesture–Action Disentanglement

Generating human motion synchronized with speech is es-sential for creating realistic virtual avatars. While recent work has madestrides in generating gestures from speech, these approaches often fallshort in enabling avatars to perform meaningful, context-specific actions.This paper presents CtrlCoMo, a motion generation framework that usestextual input alongside speech audio to provide explicit control over ac-tions in co-speech scenarios. To address the interference between ges-tures and actions in co-speech motions, we introduce Pyramid-VQ, anautoencoder that separates gestures from actions through hierarchicalquantization, with shallow layers capturing global semantics and deeplayers encoding localized gestures, guided by layer-wise CLIP-based se-mantic regularization. Additionally, the model provides explicit controlover gesture intensity via an AdaLN mechanism. For evaluation, we intro-duce CoHuMo, a large-scale co-speech motion dataset of 370 hours withrich human actions. This dataset encompasses a wide range of scene typesand languages, providing a robust benchmark for future research. Exten-sive evaluations are conducted on CoHuMo which validate our model’seffectiveness.