Fine-Grained Text-to-Video Retrieval for Camera-Trap Data

Automatically retrieving videos from large camera-trap data-sets remains challenging. Text-to-Video retrieval (TVR) methods basedon large video-language models (VLMs) have potential to retrieve eventsof interest by describing them with simple text queries. However, cur-rent methods often lack spatiotemporal understanding and do not gen-eralize well to ecological data. In this work, we introduce Prompting-MammAlps, the first camera-trap TVR benchmark, and propose a fine-grained and interpretable TVR method. Specifically, we trained a vi-sion transformer to perform spatiotemporal action localization, and con-vert its output to structured text, describing each video. Independently,ethology-inspired queries are processed by a Large-Language Model (LLM)based coding agent to parse the structured text per video and retrievevideos accordingly. We harnessed the LLM to use functions from a cus-tom parsing library to minimize the risk of LLM hallucinations and toimprove method interpretability. This retrieval approach applied on thePrompting-MammAlps benchmark achieved a set-based F1-score of 34%on a test set of 135 ecologically-relevant queries and 775 candidate videos.In comparison the best zero-shot VLM achieved a F1-score of 18%, whilealso lacking interpretability.Project page: eceo-epfl.github.io/prompting-mammalps/