Reconstruction by Generation: 3D Multi-Object Scene Reconstruction from Sparse Observations

Accurately reconstructing complex full multi-object scenesfrom sparse observations remains a core challenge in computer vision anda key step toward scalable and reliable simulation for robotics. In thiswork, we introduce RecGen, a generative framework for probabilisticjoint estimation of object and part shapes, as well as their pose underocclusions and partial visibility from one or multiple RGB-D images. Byleveraging compositional synthetic scene generation and strong 3D shapepriors, RecGen generalizes across diverse object types and real-worldenvironments. RecGen achieves state-of-the-art performance on complex,heavily occluded datasets, robustly handling severe occlusions, symmetricobjects, objects parts, and intricate geometry and texture. Despite usingnearly 80% fewer training meshes than the previous state of the artSAM3D, RecGen outperforms it by 30.1% in geometric shape quality,9.1% in texture reconstruction, and 33.9% in pose estimation. We willrelease our code, training data and evaluation benchmark at our website.