VoCa: Unified Autoregressive Modeling for Talking Audio-Video Generation

Talking-head generation from text typically relies on cas-caded pipelines that synthesize audio before video, leading to error ac-cumulation, latency, and audiovisual misalignment. Existing end-to-endmodels address some of these issues but often lack autoregressive capabil-ities. In this paper, we present VoCa, a unified autoregressive frameworkthat jointly generates speech and talking-head video from text transcriptsand a reference image. VoCa employs a large language model (LLM) tomap text into shared representations, which are concurrently processedby dedicated speech and video decoders to ensure strict synchronization.To bridge the semantic gap between audio-optimized features and hu-man motion, we introduce a speech refiner with a tailored multi-stagetraining strategy. Furthermore, to enable step-by-step autoregressive co-generation, we propose a unified window partitioning mechanism andan associated improved rolling-window denoising strategy. This adaptsa bidirectional diffusion transformer into a causal architecture synchro-nized with the LLM’s autoregressive generation windows. Extensive ex-periments show that VoCa matches or surpasses state-of-the-art baselinesin visual quality, improves audio–lip synchronization, and identity preser-vation. Code and models will be released upon acceptance.