Remembering Across Blocks: Topology-Conditioned Block-Progressive Memory for Skeleton-Based Action Recognition

Skeleton-based action recognition classifies human actionsfrom sequences of joint coordinates and is widely studied using graph con-volutional networks (GCNs). GCNs typically rely on repeated graph nodefeature aggregation via one-hop message passing to capture long-rangedependencies across entire joint sequences, while progressively refiningrepresentations from multiple perspectives through block-wise learnedtopology. However, this repeated aggregation can often degrade repre-sentations, making the node features in the final embedding increasinglyhomogenized and causing the final embedding to fail to capture local orfine-grained representations formed in early blocks. To address these is-sues, we propose a novel topology-conditioned block-progressive memoryfor skeleton GCNs. Along the GCN block axis, memory stores comple-mentary block-wise representations and fuses the accumulated memorystate into the final embedding, thereby mitigating representation degra-dation caused by repeated aggregation. At each block, a sample-specificmemory state is updated by minimizing a topology-conditioned recon-struction loss, where the reconstruction error quantifies how well theprevious memory explains the current block representation while captur-ing newly emerging, discriminative cues. We support two inference-timemodes: memory-free, which uses memory only during training, and anoptional memory-enabled mode, which also applies the same memoryupdate rule at test time for modest gains. Both variants achieve state-of-the-art results on large-scale benchmarks, including NTU RGB+D,NTU RGB+D 120, and Kinetics-Skeleton.