JAM-Flow: Joint Audio-Motion Synthesis with Flow Matching

The intrinsic link between facial motion and speech is oftenoverlooked in generative modeling, where talking head synthesis andtext-to-speech (TTS) are typically addressed as separate tasks. Thispaper introduces JAM-Flow, a unified framework to simultaneously syn-thesize and condition on both facial motion and speech. Our approachleverages flow matching and a novel Multi-Modal Diffusion Transformerarchitecture, integrating specialized Motion-DiT and Audio-DiT modules.These are coupled via selective joint attention layers and incorporate keyarchitectural choices, such as temporally aligned positional embeddingsand localized joint attention masking, to enable effective cross-modalinteraction while preserving modality-specific strengths. By analyzing andleveraging pretrained representation embeddings, JAM-Flow is designedfor efficient, near real-time sampling. Trained with an inpainting-styleobjective, JAM-Flow supports a wide array of conditioning inputs (in-cluding text, reference audio, and reference motion) facilitating taskssuch as synchronized talking head generation from text, audio-drivenanimation, and much more, within a single, coherent model. JAM-Flowsignificantly advances multi-modal generative modeling by providing apractical solution for holistic audio-visual synthesis. Project website