Beyond Filter Pruning: Top-K Spatial Selection for Efficient Neural Networks

Deep neural networks have shown remarkable performance across diverse domains, but their substantial model size and computational requirements hinder deployment in memoryand computationconstrained environments. Despite progress in pruning, existing methods primarily target inter-filter redundancy and leave significant spatial redundancy within convolutional kernels unaddressed. The resulting models retain many correlated spatial patterns, limiting achievable efficiency gains. We propose Top-K Pruning (TKP), a pruning framework that extends sparsity to the spatial dimension by retaining only the most informative positions within each convolutional kernel. TKP introduces a two-stage procedure: an auxiliary regularization phase that concentrates information into dominant spatial locations, followed by deterministic Top-K selection that yields semi-structured sparsity. This transforms dense convolutions into efficient selective-sampling operations with strictly bounded computational cost. Across diverse architectures, including CNNs, vision transformers, vision-language models, and a large language model, TKP consistently achieves strong accuracy–efficiency trade-offs. On CIFAR and ImageNet models, TKP matches or exceeds the accuracy of prior structured pruning methods while delivering up to 8.9× theoretical FLOP reduction. TKP remains robust under quantization-aware training, achieving 16–17× compression with minimal accuracy loss on ResNet–18 and VGG–19. Moreover, TKP generalizes to BLIP-Base and LLaMA-2-7B, outperforming state-of-the-art pruning baselines. These results highlight TKP as a simple and effective approach for removing spatial redundancy in modern vision and largescale models.