Extending a Large View Synthesis Model for Multi-view Panoptic Segmentation

Large view synthesis models synthesize novel views throughcross-view attention without explicit 3D representations, and recent stud-ies have shown that they learn accurate spatial correspondence fromRGB supervision alone. We observe that this correspondence generalizesbeyond appearance. When non-photorealistic signals such as binary en-coded panoptic labels are passed through the model, they are propagatedto novel views with consistent spatial structure. These results indicatethat the correspondence learned for RGB view synthesis can also propa-gate view-independent per-pixel labels. From this observation, we studyhow large view synthesis models can be extended beyond appearancerendering to 3D scene understanding. We propose a panoptic segmen-tation pipeline that reuses a frozen view synthesis model to propagatepanoptic labels from input views to novel views, without 3D reconstruc-tion or any segmentation-specific training of the view synthesis model.Given panoptic labels on the input views, we encode them into binarychannel representations and pass them through the same model to rendertarget-view segmentation. On ScanNet, our method achieves segmenta-tion quality on par with Gaussian based approaches requiring explicit3D reconstruction, while outperforming them in novel view synthesis bymore than 7 dB. The label propagation also transfers across datasets,surpassing these approaches on Replica without any fine-tuning.