CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognition

Personalization in emotion recognition (ER) is essential foran accurate interpretation of subtle and subject-specific expressive pat-terns. Recent advances in vision–language models (VLMs) such as CLIPdemonstrate strong potential for leveraging joint image–text represen-tations in ER. However, CLIP-based methods either depend on CLIP’scontrastive pretraining or on LLMs to generate descriptive text prompts,which are noisy, computationally expensive, and fail to capture fine-grained expressions, leading to degraded performance. In this work, weleverage Action Units (AUs) as structured textual prompts within CLIPto model fine-grained facial expressions. AUs encode the subtle muscleactivations underlying expressions, providing localized and interpretablesemantic cues for more robust ER. We introduce CLIP-AU, a lightweightAU–guided temporal learning method that integrates interpretable AUsemantics into CLIP. It learns generic, subject-agnostic representationsby aligning AU prompts with facial dynamics, enabling fine-grained ERwithout CLIP fine-tuning or LLM-generated text supervision. AlthoughCLIP-AU models fine-grained AU semantics, it does not adapt to subject-specific variability in subtle expressions. To address this limitation, wepropose CLIP-AUTT, a video-based test-time personalization method thatdynamically adapts AU prompts to videos from unseen subjects. By com-bining entropy-guided temporal window selection with prompt tuning,CLIP-AUTT enables subject-specific adaptation while preserving tempo-ral consistency. Our experiments on three challenging video-based subtleER datasets — BioVid, StressID, and BAH — indicate that CLIP-AUand CLIP-AUTT outperform state-of-the-art CLIP-based FER and TTAmethods. Code: https://github.com/osamazeeshan/CLIP-AUTT.