Aligning Anything: Hierarchical Motion Estimation for Video Frame Interpolation

Existing advanced video frame interpolation (VFI) methods strug-gle to achieve accurate per-pixel motion or object-level motion estimation. Thereasons lie in that pixel-level motion estimation allows for infinite possibilities,making it challenging to guarantee global motion consistency. Conversely, lo-cal pixel-level motion tends to be discontinuous whereas object-level motionestimation is typically optimized for global motion. Therefore, a hierarchicalmotion learning scheme is imperative to enhance the consistency and continu-ity of motion prediction in VFI. To this end, we marry the object-level motionto the pixel-level motion to construct a hierarchical motion estimation frame-work. This approach elaborately incorporates object priors derived from open-world knowledge models, such as Segment Anything Model (SAM), to facili-tate latent object-level motion learning. In particular, a hybrid contextual fea-ture extraction module (HCE) is employed to aggregate both pixel-wise and se-mantic representations. This is further reinforced by hierarchical motion loss(HML) and hierarchical feature loss (HFL), which respectively simulate the con-sistent and continuous motion patterns for individual objects. Our frameworksupports two configurations: a full variant with HCE, HML, and HFL for higheraccuracy, and an efficient variant with only HML and HFL, which introducesno additional inference overhead. Extensive experiments demonstrate consistentimprovements over state-of-the-art VFI methods across multiple benchmarks.https://github.com/hhhhhumengshun/SAM-VFI