Invisible Shortcuts: Why Vision Encoders Know Your Camera

Deep vision models exploit shortcuts, relying on cues thatcorrelate with supervision signals. Prior work has focused on visiblebiases, such as object-background or texture correlations. We identifya different source of shortcut learning: invisible metadata traces em-bedded at the pixel level, for metadata such as image processing andphoto acquisition. We hypothesize that large-scale semantic supervi-sion, whether through categorical labels (ImageNet) or billion-scale cap-tions (LAION), naturally induces metadata-semantics correlations dur-ing pretraining, leading models to convert low-level signals into predic-tive features. By introducing controlled metadata-semantics correlations,we show that stronger ones produce systematically higher sensitivity tometadata traces and larger performance degradation under metadatadistribution shifts. We further explore mitigation strategies applied dur-ing and after pretraining that reduce sensitivity not only to targetedmetadata but also to unseen ones, without sacrificing performance ondownstream tasks. Metadata sensitivity also has a positive side: it partlyexplains the strong generated-image detection ability of some encoders,while its mitigation can improve out-of-distribution generalization. Code: