Easing 3D Pattern Reasoning with Side-view Features for Semantic Scene Completion

This paper proposes a side-view context inpainting strategy (SidePaint) to ease the reasoning of unknown 3D patterns for semantic scene completion. Based on the observation that the learning burden on pattern completion increases with spatial complexity and feature sparsity, the SidePaint strategy is designed to decompose the complex 3D pattern learning into easier 2D context inpainting with dense feature volumes. Specifically, our approach densely lifts image features into 3D volume space with distance-aware projection, and reasons missing patterns in 2D side-view feature maps sliced from feature volumes. With the learning burden relieved by decreasing pattern complexity in 2D space, our SidePaint strategy enables more effective semantic completion than directly learning 3D patterns. Extensive experiments demonstrate the effectiveness of our SidePaint strategy on several challenging semantic scene completion benchmarks.`