LANCE: Low Rank Activation Compression for Efficient On-Device Continual Learning

On-device learning is essential for personalization, privacy,and long-term adaptation in resource-constrained environments. Achiev-ing this requires efficient learning, both fine-tuning existing models andcontinually acquiring new tasks without catastrophic forgetting. Yet bothsettings are constrained by high memory cost of storing activations dur-ing backpropagation. Existing activation compression methods reducethis cost but rely on repeated low-rank decompositions, introducing com-putational overhead. Also, such methods have not been explored forcontinual learning. We propose LANCE (Low-rank Activation Compres-sion), a framework that performs one-shot higher-order Singular ValueDecomposition (SVD) to obtain a reusable low-rank subspace for ac-tivation projection. This eliminates repeated decompositions, reducingboth memory and computation. Moreover, fixed low-rank subspaces fur-ther enable on-device continual learning by allocating tasks to orthogo-nal subspaces without storing large task-specific matrices. Experimentsshow that LANCE reduces activation storage by 25× to 380× depend-ing on the architecture, while maintaining accuracy comparable to fullbackpropagation on CIFAR-10/100, Oxford-IIIT Pets, Flowers102, andCUB-200 datasets. On continual learning benchmarks (Split CIFAR-100,Split MiniImageNet, 5-Datasets), it performs competitively with orthog-onal gradient projection methods at a fraction of the memory cost. Theseresults position LANCE as a practical and scalable solution for efficientfine-tuning and continual learning on edge devices.3