iMED: A Multi-Endoscope Dataset for Surgical 3D Perception

We introduce iMED, the first synchronized multi-endoscopedataset for robot-assisted minimally invasive surgery (RAMIS), designedto address a fundamental limitation in surgical vision benchmarking: theabsence of independent held-out viewpoints. Existing surgical datasetscapture single-trajectory sequences from a single endoscope, making itimpossible to distinguish geometric generalization from photometric in-terpolation along a narrow forward-facing path. iMED provides 340 se-quences (≈170K synchronized timepoints with 4 views per timepoint)recorded simultaneously from two independent stereo endoscopes acrossex vivo, postmortem, and live surgical settings, spanning 14 specimenswith diverse anatomical regions and motion regimes. The dataset in-cludes calibrated camera intrinsics, ArUco-based frame-wise pose esti-mates with uncertainty quantification, instrument segmentation masks,and rich clinical metadata. Using a train-on-one-endoscope, test-on-anotherevaluation protocol, we benchmark 23 state-of-the-art methods acrossrigid and deformable novel view synthesis, pose estimation, feature match-ing, and monocular depth estimation. Our experiments show that meth-ods relying primarily on photometric supervision degrade substantiallyunder this held-out-endoscope setting, while methods with explicit ge-ometric regularization are more robust across views in our benchmark.Notably, large-scale foundation models transfer surprisingly well to sur-gical imagery, often outperforming domain-specific models, suggestingpitfalls in current fine-tuning protocols. iMED establishes a new evalua-tion protocol for geometric generalization in surgical vision. Dataloaderand dataset links can be found at github.com/surgical-vision/imed.