E.T. the Exceptional Trajectory: Text-to-camera-trajectory generation with character awareness

Stories and emotions in movies emerge through the effect of well-thought-out directing decisions, and in particular camera placement and movements over time. Crafting compelling camera motion trajectories remains a complex iterative process even for skilful artists. While recent work has been focusing on speeding up the process, current solutions remain limited in describing and generating complex and content-aware camera trajectories, especially for movies, where the story evolves around moving characters and hence the camera follows them. To alleviate this, in this paper, we propose a diffusion-based approach, named DIRECTOR, which generates complex compositions of camera trajectories from high-level textual inputs that describe the relation and synchronisation between the camera and characters. Furthermore, we propose a dataset called Exceptional Trajectories (E.T.) with camera motion trajectories along with textual captions with descriptive information of camera and character motion. To our knowledge, this is the first movie dataset of its kind. For proper evaluation, we also provide a robust and accurate language trajectory feature representation. Extensive experiments and analysis show that DIRECTOR successfully leverages both the caption and camera trajectories and sets the new state of the art on this task. Our work represents a significant advancement in democratizing the art of cinematography for amateur and experienced users.