Sparse auto-regressive modeling for scene generation from multi-view images

Generating complete 3D scenes from sparse, unconstrainedviews is a fundamental challenge in 3D vision which requires reason-ing beyond observed content while remaining computationally tractable.Existing feed-forward reconstruction methods are inherently limited tocontent visible in the input images, while 3D generative modeling is hin-dered by the high computational cost of dense volumetric representationsand the scarcity of large-scale 3D supervision. We introduce SPAR3S, asparse voxel-aligned 3D latent generative model for conditional scenecompletion without requiring ground-truth 3D data for supervision. Ourkey insight is to formulate 3D scene generation in a structured, compact,voxel-aligned 3D latent space where only occupied voxels are represented.We learn this sparse latent space directly from multi-view images usingphotometric supervision via differentiable 3D Gaussian Splatting. Givena partial set of observed voxels encoded from sparse input views, scenecompletion reduces to predicting the missing latent tokens and theirspatial support within the voxel grid. To this end, we train a maskedautoregressive transformer that jointly models voxel occupancy and la-tent token values, enabling efficient and spatially consistent generation ofunseen regions. We demonstrate the effectiveness of our method on syn-thetic indoor scenes, achieving higher novel-view quality than prior work.We further validate its generalization on RealEstate10k, highlighting itsapplicability to real-world data.