SAM-MT: Real-Time Interactive Multi-Target Video Segmentation

Modern Video Object Segmentation (VOS) involves track-ing and segmenting user-specified targets. While recent approaches haveachieved remarkable performance in single-target scenarios, extendingthem to multi-target settings typically involves replicating the single-target processing for each individual object, resulting in reduced framerates (FPS) with unbounded latency as target density scales. Built uponSegment Anything 2 (SAM2), we propose SAM-MT, which addressesthis by transforming the model into an interactive framework for real-time Multi-Target video segmentation. SAM-MT uses explicit queriespropagated and updated across frames to represent different individualtargets, in parallel with a shared representation for global context. It em-ploys decoupled masked attention to keep individual identities distinctfrom cross-target interference, and sparse memory for stable temporalevolution, along with specialized strategies for occlusion handling andoverlap prevention. SAM-MT successfully decouples latency from thenumber of targets, achieving real-time speed on par with single-targetbaselines (>36 FPS for 10 targets) while maintaining SAM2’s robustvideo segmentation performance.