Skip to yearly menu bar Skip to main content


Poster

TC4D: Trajectory-Conditioned Text-to-4D Generation

Sherwin Bahmani · Xian Liu · Wang Yifan · Ivan Skorokhodov · Victor Rong · Ziwei Liu · Xihui Liu · Jeong Joon Park · Sergey Tulyakov · Gordon Wetzstein · Andrea Tagliasacchi · David B Lindell

[ ]
Thu 3 Oct 7:30 a.m. PDT — 9:30 a.m. PDT

Abstract:

Recent techniques for text-to-4D generation synthesize dynamic 3D scenes using supervision from pre-trained text-to-video models. However, existing representations for motion, such as deformation models or time-dependent neural representations, are limited in the amount of motion they can generate—they cannot synthesize motion extending far beyond the bounding box used for volume rendering. The lack of a more flexible motion model contributes to the gap in realism between 4D generation methods and recent, near-photorealistic video generation models. Here, we propose TC4D: trajectory-conditioned text-to-4D generation, which factors motion into global and local components. We represent the global motion of a scene bounding box using rigid transformation along a spline, and we learn local deformations that conform to the global trajectory using supervision from a text-to-video model. Our approach enables the synthesis of scenes animated along arbitrary trajectories, compositional scene generation, and significant improvements to the realism and amount of generated motion, which we evaluate qualitatively and through a user study.

Live content is unavailable. Log in and register to view live content