Unified Video Dense Prediction from Disjoint Data

Scene understanding requires simultaneous prediction aboutgeometry, appearance, and semantics. However, existing task-specific an-notations are fragmented across incompatible, domain-specific datasets.Current unified systems circumvent this by restricting training to fullyco-annotated data, or by incurring the large computational cost of pseudo-labeling. To mitigate this, we introduce UniD, a unified video model thatjointly predicts eight dense scene properties—depth, surface normals, se-mantic segmentation, boundaries, human parts, albedo, shading, and ma-terials—all learned from disjoint, domain-specific datasets. We propose asimple yet effective distillation step in which per-task experts supervisea unified backbone through lightweight task projectors, eliminating theneed for annotation overlap or pseudo-labeling. Our key insight is thatthe strong visual priors of a pretrained diffusion model are sufficientto bridge the domain gaps introduced by disjoint training sources, en-abling robust generalization to scene-task combinations never seen dur-ing training. UniD achieves competitive performance against per-taskspecialists and multi-task baselines, with strong generalization to out-of-distribution scenarios and enhanced temporal and cross-task consistency.Code and video results are available at https://unid-video.github.io.