What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features

Contrastive vision-language models such as CLIP map se-mantically opposite phrases (e.g., “a dog” vs. “not a dog”) to nearly iden-tical embeddings, rendering them insensitive to negation. We attributethis failure to a phenomenon we call Representational Collapse: by track-ing compositional divergence and visual alignment across the CLIP textencoder, we show that middle layers build compositional syntax, but thefinal layers collapse this structure as visual alignment rises, producinga syntax-blind final representation. To recover the lost negation signalwithout altering pretrained weights, we propose PeakPatch, a lightweightpost-hoc correction system that intercepts the encoder at its composi-tional peak while keeping CLIP fully frozen. An Embedding CorrectionNetwork (ECN) uses cross-attention to extract a negation-specific sig-nal from the peak layer, anchored to a stable baseline, and predicts adeviation vector that re-injects the lost syntax into the final-layer em-bedding space. A complementary Score Correction Network (SCN) pre-dicts bounded scalar score offsets for discriminative tasks. Both modulesare trained jointly end-to-end while all CLIP parameters remain frozen,adding only 5.2M parameters (3.5% of the backbone) and preserving thestandard cosine similarity interface. On NegBench, PeakPatch achieves74.3% on COCO MCQ (+35.1 over CLIP, +17.8 over the best encoderfine-tuning method) and 65.5% on VOC MCQ, while outperforming allfine-tuning baselines on fully out-of-distribution negation retrieval de-spite training only 3.5% of the parameters. The corrected embeddingsalso transfer to text-to-image generation (+18.4 negation score) and gen-eralize across ViT-B/32, ViT-L/14, and SigLIP backbones.Project page: https://stevencylu.github.io/PeakPatch/