SWSL: Semantic-aware Weakly Supervised Learning for 3D Motion Generation using 2D Motion Data

Data scarcity is a major constraint for data-driven Text-to-3D Motion generation. Prior attempts exploit abundant 2D motion datato address it, but typically treat 3D generation as a post-hoc stage of 2Dprediction, resulting in camera-pose-dependent multi-stage pipelines andunderutilization of rich 2D motion semantics (e.g., temporal dynamics).To respond, we propose SWSL, a Semantic-aware Weakly SupervisedLearning framework that leverages 2D-motion semantics to directly opti-mize end-to-end Text-to-3D Motion models. SWSL comprises three mod-ules: Semantic Space Construction (SSC), which first constructs a sharedsemantic space aligning text, 2D-motion, and 3D-motion; Semantic-awarePseudo-label Enhancement (SPE), which then uses weak supervisionfrom 2D-motion embeddings to refine 3D pseudo-labels; and Semantic-level Feedback Optimization (SFO), which finally supplies embedding-level feedback for optimization. SWSL offers: (i) reliance reduction ontext-3D motion data by utilizing text-2D motion pairs; (ii) 2D data con-straint looseness by extracting semantics from single-view, easy-accessible2D motions; and (iii) scalability along mainstream Text-to-3D Motionadvancement. On HumanML3D, SWSL yields consistent FID reductionsunder both a domain-specific protocol and a more general TMR-basedprotocol: -0.012 and -0.017 on simple baselines, along with -0.010 on acomplicated one in a domain-specific setting, and -0.048, -0.038, and -0.056 under TMR setting - demonstrating its efficacy, robustness, andgeneralization beyond the domain-specific bias.