Don’t Mask Out the Background! Natural-Light Photometric Stereo via Illumination Reconstruction

This paper introduces an inverse-rendering framework fornatural-light, uncalibrated photometric stereo (PS) that leverages back-ground cues captured alongside the target object. Natural-light PS (NaPS)acquires shading variations by moving or rotating the camera and objectunder fixed, uncontrolled illumination, such as indoor lighting, whilemaintaining their relative geometry. However, uncalibrated NaPS, inwhich the lighting conditions are unknown, remains inherently ill-posed.To tackle this challenge, we propose explicitly reconstructing the lightingenvironment from the image background, which is typically masked outin prior work, thereby converting uncalibrated NaPS into a tractableinverse-rendering problem. Specifically, we move and rotate the camera-object pair while keeping their relative pose fixed, and reconstruct thesurrounding illumination directly from the observed background usinga 3D Gaussian Splatting (3DGS) representation. We then optimize thetarget object’s shape and reflectance via inverse rendering under thereconstructed illumination. Experimental results demonstrate that ourinverse-rendering-based approach yields more accurate estimates of bothgeometry and reflectance than existing learning-based PS methods, espe-cially under realistic near-field indoor conditions.