AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation

Audio-video generation has recently gained unprecedentedresearch attention, aiming to synthesize high-quality sounding video con-tent with fine-grained synchronization and semantic alignment betweenthe auditory and visual components. The preceding methods predomi-nantly adopt a dual-branch design with separate tokenization and gen-eration modules per modality, neglecting the representation gap whilenecessitating intensive computational resources for proper training. In-spired by recent advancements in one-dimensional visual tokenization,we present AVTok, a novel unified tokenizer designated for holisticaudio-video generation. AVTok features a dual-stream transformer-basedarchitecture with shared encoder-decoder and modal-specific learnablequeries to efficiently and effectively encode an audio-video pair into acompact one-dimensional latent representation with a unified codebook.To cope with the heterogeneous information imbalance that hinders AV-Tok from exploiting aligned audio-visual information, we devise a hier-archical training strategy to progressively realize reconstruction capabil-ities for each modality. Extensive experiments demonstrate that AVTokexcels both in audio-video reconstruction and when integrated into down-stream pipelines for audio-to-video, video-to-audio, and class-conditionaljoint audio-video generation. AVTok paves the way for the challenge ofjoint audio-video tokenization and provides a potential direction to buildunified large multimodal models for audio-video generation.