SignBind-LLM: Multi-Stage Modality Fusion for Sign Language Translation

Current sign language translation (SLT) systems attempt tolearn all aspects of signing—manual gestures, high-speed fingerspelling,and asynchronous non-manual facial cues—within a single end-to-endnetwork. Learning multiple tasks without detailed supervision leads topoor recognition of fingerspelled proper nouns and technical terms, andleaves rich disambiguating information from lip movements largely unex-ploited. We introduce SignBind-LLM, a modular framework that ad-dresses these limitations through three dedicated expert streams: one forcontinuous signing, one for fingerspelling, and one for lipreading. Eachexpert is pre-trained independently using CTC on approximately twomillion automatically generated pseudo-gloss sequences, removing theneed for manual gloss annotation. A lightweight transformer with learnedtemporal alignment fuses the expert outputs, and a pre-trained languagemodel translates the resulting pseudo-gloss sequences into fluent spokenEnglish. At matched decoder scale (250M parameters), our architecturealready surpasses all prior methods, confirming that the gains are archi-tectural rather than a consequence of scaling the language model. Scal-ing to a larger decoder sets a new state-of-the-art across How2Sign: 23.1,BOBSL: 7.0, and ChicagoFSWild+: 73.6%, while requiring significantlylower training cost than prior approaches.