HitMem: Hierarchical Temporal 3D Memory with Multi-Modal Context-Aware Retrieval for Dynamic Environments

Executing long-term tasks in dynamic environments requiresembodied agents to maintain robust and adaptive 3D scene represen-tations. However, most existing 3D memory frameworks rely on staticworld assumptions. When objects are displaced by human activities orunobserved events, agents encounter memory-observation conflicts andoften require costly geometric recomputations or inefficient global re-exploration. To address this, we propose HitMem, a hierarchical tempo-ral 3D memory framework with a multi-modal context-aware retrievalmechanism. Through continuous perception, HitMem unifies semanticand spatial information into a lightweight topological graph that cap-tures support relationships, while a temporal decay mechanism dynami-cally regulates memory activeness to mitigate the impact of stale repre-sentations. In addition, the multi-modal context-aware retrieval mecha-nism defaults to filtering candidates using integrated semantic, spatial,and temporal memory features, and activates a specialized two-stage re-trieval process when object displacement is detected. This process com-bines spatial constraints inferred from external agent trajectories withsemantic common sense grounded in class affinities, efficiently identi-fying high-probability candidate regions. Extensive evaluations on ourconstructed Dyna-THOR benchmark demonstrate that HitMem signif-icantly improves object relocation accuracy, reduces exploration costs,and enhances task execution performance in dynamic environments.