AV2T-Gen: Aerial Visible to Thermal Generation with Environment and Vehicle State Guidance

Accurate visible-to-thermal image generation holds signifi-cant application value in fields such as vehicle detection and autonomousdriving. However, existing methods often suffer from the problem thatthe generated results are inconsistent with environmental conditions andvehicle states (e.g., moving/static). Moreover, the scarcity of synchro-nized and calibrated visible-thermal image pairs has further hinderedthe development of these fields. We propose AV2T-Gen, an innovativeenvironment and vehicle guided visible-to-thermal generation frameworkbuilt upon the Instruct-Pix2Pix Diffusion Models for aerial imagery,which distinguishes itself from existing methods by explicitly incorpo-rating environment parameters and vehicle state information for thefirst time. The framework first maps images to the latent space usinga VAE to process image features efficiently, and then generates direc-tional features for vehicles in different environmental conditions usingmultimodal feature information. Meanwhile, we also proposed a visible-thermal paired dataset with environmental conditions and vehicle states.Experimental results have demonstrated that the generated thermal im-ages outperform existing approaches in terms of physical realism andvehicle detail discrimination significantly, with more accurate reflectionof environmental and vehicle characteristics. The code is available athttps://github.com/NPU-CVPG/AV2T.