NeLU3D: Neural Inverse Structured Light without Modeling the Projector

Structured Light (SL) is a popular technique that delivers precise 3D shape acquisition across diverse materials and object sizes. SL requires, however, careful modeling of the set-up, including geometric and radiometric calibration of the camera-projector pair; without it, reconstruction quality degrades. We propose NeLU3D, a neural inverse SL method without explicitly modeling the projector. We calibrate the camera–projector system using a neural network that maps 3D positions to a set of projected patterns. Then, with as few as four monochromatic images (or two RGB images), our approach uses differentiable volume rendering to fit a surface to match SL captures. We scan over twenty-five objects of different shapes and reflectances to demonstrate the feasibility and quality of our method in a handful of projector-camera set-ups, including an extremely low-cost projector and an analog projector with fixed RGB pattern. We also showcase sub-millimeter accuracy with suboptimal patterns, where previous methods recover noisy 3D surfaces. We release an open-source implementation at https://github.com/ geometryprocessing/neural-lookup.