SSBP: Stage-Specialized Block Pruning for Video Diffusion Models

Recent advances in video diffusion models have delivered re-markable generation quality, yet their generation cost remains high. Thisinefficiency is largely driven by the iterative denoising process, with com-putationally expensive per-step architectures. Step distillation enablesfew-step generation, but per-step latency remains high because a large-scale backbone is invoked at each denoising step. Several works seek toreduce per-step latency via block pruning using importance scores aver-aged across denoising steps. However, such averaged scores can obscurestage-critical blocks and bias the pruning process, removing blocks essen-tial for certain stages while leaving redundancy in others. In this paper,we propose a novel pruning paradigm, Stage-Specialized Block Pruning,that partitions the denoising process into multiple stages and assigns atailored pruned variant to each stage. Each variant consists of sharedblocks across multiple stages and stage-specific blocks for its assignedstage. With parameter sharing, the overall parameter budget is smallerthan the original model, while the per-step computational footprint is re-duced for lower latency. To derive stage-specialized variants, we introducea hierarchical selection mechanism consisting of an Inter-Stage VarianceSelector (IVS) and an Intra-Stage Quality Selector (IQS). IVS comparesthe variance of stage-wise block-importance distributions and prioritizesthe stage with higher variance for pruning, indicating greater removableredundancy. IQS mitigates noisy importance estimation by generatingmultiple low-importance pruning candidates within the selected stageand choosing the one with the highest video generation quality. IVS andIQS are applied iteratively until reaching a target latency budget. Fur-thermore, we introduce a two-phase training pipeline to recover the per-formance of these variants and facilitate sequential composition for few-step generation. Extensive experiments on the representative video diffu-sion model Wan2.1-1.3B demonstrate that our method reduces per-stepinference cost while maintaining competitive performance on VBench.