Selective Synergistic Learning for Video Object-Centric Learning

Typical video object-centric learning (VOCL) approachesemploy slot-based frameworks that rely on reconstruction-driven en-coder–decoder architectures, where learning is mediated by two spatialmaps: attention maps from the encoder and object maps from the decoder.As these two distinct maps exhibit different properties, a recent densealignment strategy attempted to reconcile this discrepancy by enforcingagreement across all spatio-temporal patches via contrastive learning.However, this indiscriminate alignment inadvertently propagates the in-herent weaknesses of each module, such as noisy encoder predictionsand blurred decoder boundaries. Moreover, computing dense similaritiesacross all pairs incurs a computational cost quadratic in the total numberof spatio-temporal patches, severely limiting scalability. Motivated bythis, we propose Selective Synergistic Learning (SSync). Instead of ex-haustive patch-to-patch alignment, SSync prevents error propagation byselectively distilling only the most reliable cues: leveraging the encoderstrictly for boundary refinement and the decoder for interior denoising.This is realized via a pseudo-labeling with linear complexity, eliminatingthe need for quadratic spatial comparisons. Also, to prevent the rein-forcement of architectural biases like slot redundancy, we introduce atransitive pseudo-label merging that consolidates overlapping slots basedon spatio-temporal activation consistency. Extensive studies demonstratethat SSync improves decomposition quality and serves as a versatile,plug-and-play module while also exhibiting exceptional robustness to slotconfigurations. Code is available at github.com/wjun0830/SSync.