DIGS: Differentiable, Incremental, Global, Scalable Pruning for Language Models

The rapid scaling of large language models (LLMs) and vision-language models (VLMs) has sharply increased compute and storagecosts, motivating extensive research on model pruning. Yet most prun-ing pipelines underuse data scaling during pruning, i.e., increasing thepruning-phase token budget. We revisit L0-regularized structured chan-nel pruning, valued for differentiability, incremental scheduling, and globaldecision-making. We uncover a long-horizon instability: the Lagrangemultiplier grows during training, amplifies pruning gradients, and causeslate-stage oscillations in both gradients and loss. We derive sufficient L2-norm bounds on pruning-specific gradients and reformulate the loss toenforce these bounds per update, replacing dual ascent with a coefficientdetermined by the gradient constraint. Building on this foundation, weintroduce DIGS (Differentiable, Incremental, Global, Scalable pruning),which enables stable and efficient structured channel pruning withoutauxiliary stabilizers while supporting much larger pruning token bud-gets. Applied to the Qwen2.5VL LLM backbone, DIGS scales pruningbeyond 100B tokens and achieves competitive results on public multi-modal benchmarks. Within this setting, larger pruning token budgetsyield monotonic performance gains at fixed sparsity.