Divide and Align: Disentangled Vision-Language Learning for Text-Based Person Anomaly Search

Text-based person anomaly search (TPAS) refers to the taskof retrieving people exhibiting normal or anomalous behaviors from nat-ural language descriptions. Existing TPAS models often learn a singlejoint embedding where appearance, action, and background informationare entangled, causing over-reliance on identity cues, poor alignment foraction-centric queries, and limited semantic connection between actionsand places where they occur. To address these issues, we propose Se-mantic Decoupled Alignment (SeDA), a disentangled vision–languageretrieval framework that explicitly factorizes both visual and textualrepresentations into appearance, action, and background components.SeDA introduces Semantic Token Projection, which derives three se-mantic queries from the global [CLS] token, softly aggregates modal-ity tokens relevant to each factor, and recomposes the resulting factortokens into a compact retrieval embedding. To enforce factor-specific se-mantics, we decompose each caption into appearance/action/backgroundsub-captions and supervise the corresponding tokens with a Feature De-coupling Loss, combined with contrastive and image–text matching ob-jectives. On the Person Anomaly Benchmark (1M pairs), SeDA achieves86.45% R@1 (+1.52 over SOTA), improves average multi-weather R@1by +2.43, and gains +3.74 R@1 under out-of-distribution evaluation.https://github.com/ErgastiAlex/SeDA