Beyond Imitation: Learning Safe End-to-End Autonomous Driving from Hard Negatives

Existing imitation learning methods for end-to-end autonomousdriving predominantly learn from successful demonstrations by minimiz-ing geometric deviations from expert trajectories. This paradigm implic-itly assumes that spatial proximity implies behavioral safety, leading toa critical objective mismatch: trajectories with nearly identical imita-tion losses may exhibit drastically different safety outcomes, where oneremains recoverable while the other results in collision. To address thislimitation, we propose BeyondDrive, a failure-aware imitation learn-ing framework that jointly learns from successful and failed driving be-haviors. First, we introduce a flow matching-based negative trajectory† Corresponding author. Code link: https://github.com/wjl2244/BeyondDrive.This work was supported by Beijing Natural Science Foundation-Xiaomi InnovationJoint Fund L253007, by Beijing Nova Program (202604841268) and Beijing NaturalScience Foundation under Grant 4242052.generator that synthesizes safety-critical yet expert-proximate trajecto-ries, enabling explicit modeling of safety asymmetry. Second, we developa diversity-aware sampling strategy that mitigates mode collapse andimproves coverage of diverse failure modes during negative trajectorygeneration. Third, we propose a Repulsive Distance Loss that simultane-ously attracts predictions toward expert demonstrations while repellingthem from hard negative trajectories, thereby establishing discrimina-tive safety boundaries in trajectory space. Applied to the uni-modalbaseline Latent TransFuser, BeyondDrive achieves 89.7 PDMS on theNAVSIMv1 closed-loop benchmark, outperforming prior state-of-the-artmethods. Moreover, BeyondDrive generalizes effectively across differentautonomous driving architectures, including multi-modal planners, andfurther demonstrates strong zero-shot transferability on the HUGSIMbenchmark.