RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representation

Pre-trained Vision Foundation Models (VFMs) have becomecentral to modern computer vision due to their powerful semantic rep-resentations and strong generalization ability. However, their patchifiedor pooled outputs are inherently low-resolution, limiting their effective-ness in tasks requiring fine-grained, pixel-level reasoning. Existing fea-ture upsampling approaches either degrade semantic fidelity or rely onVFM-specific retraining and heavy architectures, hindering efficiency andscalability. To address these challenges, we propose RaysUp, an ultra-lightweight, task-agnostic, and VFM-agnostic feature upsampling frame-work that reconstructs high-resolution feature maps at arbitrary resolu-tions. Unlike conventional 2D interpolation or attention-based schemes,RaysUp lifts feature reconstruction into a geometry-aware ray domain.Specifically, we introduce a Spatially Decoupled Guidance Encoder fordirection-aware guidance encoding, an Any-Resolution Cross-Attentionmechanism for resolution-flexible reconstruction, and a novel Ray Posi-tional Encoding (RayPE) that injects implicit 3D geometric priors via6D Plücker ray coordinates. Finally, A Geometry-Aware NeighborhoodAttention module further ensures content-adaptive bilateral aggregationwhile preserving geometric consistency. Extensive experiments across di-verse dense prediction tasks demonstrate that RaysUp achieves state-of-the-art performance while using only 16% of the parameters of AnyUpand delivering approximately 7× faster inference. These results highlighta substantially improved accuracy–efficiency trade-off and establish Ray-sUp as a practical and scalable solution for universal feature upsampling.Code is available at https://github.com/MAP-RaysUp/RaysUp.