SENTRY: SAM2-Enhanced Neighbor-Aware and Temporally Reasoned Memory for Visual Tracking

We revisit the memory update mechanism in SAM2-basedvisual object tracking and identify confidence-only mask selection asthe dominant cause of drift under occlusion, rapid motion, and dis-tractors. We introduce SENTRY, a training-free, plug-and-play, refine-before-write module that validates each memory update for short-horizontemporal consistency before committing it. SENTRY aggregates diversesegmentation hypotheses per frame, backtracks them into short tracklets,and uses neighbor-aware cycle-consistent matching against recent trajec-tories to favor temporally and geometrically consistent masks. It leavesthe base architecture untouched, replacing confidence-driven writes withconsistency-validated ones. For fair evaluation, we re-evaluate major open-source SAM2-based trackers across all available scales and datasets, fill-ing gaps in prior reports. Integrated into five strong baselines, SENTRYdelivers consistent gains across nine benchmarks, achieving new zero-shotSOTA on LaSOT, LaSOText , GOT-10k, VOT20, VOT22, and DiDi. De-spite these checks, the SAM2-L version runs at 32.8 FPS on an A100, andacross compatible hosts adds only about 0.4–0.6 GB VRAM. Our resultsprovide the first unified all-scale evaluation of SAM2-based trackers andshow that enforcing temporal validity at write time stabilizes memory-augmented tracking without retraining.