Object detection is a pivotal task in computer vision, focused upon localising and categorising objects within the distribution for which they are trained. Nonetheless, the capability of an object detector to localise objects out of the training distribution remains largely unexplored. Whilst recent approaches in object-level out-of-distribution (OoD) detection heavily rely on class-wise labels, such approaches contradict truly open-world scenarios where the number of classes is often unknown. In this context, anomaly detection focuses on detecting unseen instances rather than identifying an object as OoD. This work aims to bridge this gap by leveraging an open-world object detector in conjunction with a self-supervised OoD detector via virtual outlier synthesis. This is achieved by using the detector backbone features to first learn object pseudo-classes in an unsupervised manner. Subsequently, these pseudo-classes serve as the basis for the class-conditional virtual outlier sampling of anomalous features that are classified by an OoD head. Our approach empowers our overall object detector architecture to learn anomaly-aware feature representations without relying on class labels, hence enabling truly open-world object anomaly detection. Empirical validation of our approach demonstrates its effectiveness across diverse datasets encompassing various imaging modalities (visible, infrared, and X-ray). Moreover, our method establishes state-of-the-art performance on object-level anomaly detection, achieving an average recall score improvement of over 5.4% for natural images and 23.5% for a security X-ray dataset compared to the current approaches. In addition, our method can detect anomalies in datasets where current approaches fail. Code is available at .