Learn Once, Edit Anywhere: Visual Direction Transfer for Diffusion Models

The rapid advancement of diffusion models has enabled thegeneration of high-fidelity images from textual prompts, yet achievingprecise, disentangled control over specific attributes remains a significantchallenge. A fundamental limitation arises because visual differences be-tween images are often far more descriptive and nuanced than what canbe captured through human-crafted text descriptions, which frequentlyfail to convey fine-grained semantic details. To address this, we intro-duce ViDiT (Visual Direction Transfer for Diffusion), a frameworkthat expands the editing vocabulary by capturing latent semantics di-rectly from image-edit pairs. ViDiT learns the underlying transformationby optimizing a single, global, and continuous editing direction from asmall set of “before-and-after” examples. This optimization process trans-fers visual changes into the diffusion model’s conditioning space, allowingfor detailed edits that text alone cannot easily describe. ViDiT operateson a “Learn Once” principle, which completely eliminates the need formodel fine-tuning or expensive per-image optimization during inference.Once learned, these continuous directions enable “Edit Anywhere” capa-bilities, allowing users to apply highly disentangled manipulations, suchas changes in facial features, animal attributes, or artistic styles, to anyimage in a zero-shot manner with granular control over the edit inten-sity. Quantitative and qualitative evaluations demonstrate that ViDiToutperforms existing text-based editing methods in maintaining inputfaithfulness while achieving precise, scalable attribute control.