R3DP: Real-Time 3D-Aware Policy for Embodied Manipulation

Embodied manipulation requires accurate 3D understanding of objects and their spatial relations to plan and execute contactrich actions. While large-scale 3D foundation models provide strong priors, their computational cost incurs prohibitive latency for real-time control. We propose Real-time 3D-aware Policy (R3DP), which integrates powerful 3D priors into manipulation policies without sacrificing real-time performance. A core innovation of R3DP is the asynchronous fast–slow collaboration module, which sophisticatedly integrates largescale 3D model’s priors into the policy without compromising real-time performance. The system queries the pre-trained slow system (VGGT) only on sparse key frames, while simultaneously employing a lightweight Temporal Feature Prediction Network (TFPNet) to predict features for all intermediate frames. By leveraging historical data to exploit temporal correlations, TFPNet explicitly improves task success rates through consistent feature estimation. Additionally, we introduce a Multi-View Feature Fuser (MVFF) that aggregates features across views by explicitly incorporating camera intrinsics and extrinsics. R3DP offers a plug-and-play solution for integrating 3D foundation models into realtime inference systems. We evaluate R3DP against multiple baselines across different visual configurations. R3DP effectively harnesses largescale 3D priors to achieve superior results, outperforming single-view and multi-view DP by 32.9% and 51.4% in average success rates. Furthermore, by decoupling heavy 3D understanding from policy execution, R3DP achieves a 44.8% reduction in inference time compared to a naive DP+VGGT integration. Project page and code are available at: https://dazazh.github.io/r3dp-project-page/