Dual Masked Generative Adversarial Transformer for Unsupervised Domain Adaptation

Transformer has recently garnered significant interest in un-supervised domain adaptation tasks due to its superior generalizationability. State-of-the-art methods leverage masked image modeling withconsistency regularization to improve target domain performance. How-ever, such strategy becomes less effective when the domain gap becomeslarge, as the domain gap between masked domain and original domainis not minimized. How to address the adaptation problem when domaingap becomes large is an important research problem in domain adap-tation. In this paper, we propose Dual Masked Generative AdversarialTransformer (Dual-MGAT), which simultaneously aligns the maskedtarget and masked source domain towards source and target domainto address the large domain gap problem. Specifically, we formulate ageneral framework for masked domain learning. To reduce the domaingap between the masked domain (e.g. masked target) and the originaldomain (e.g. source), we note that there exists two types of discrepan-cies, namely the information gap due to masked image modeling and thedomain distribution discrepancy. To this end, we propose the maskedgenerative adversarial adaptation technique, which introduces a trans-former decoder to bridge the masked feature space towards the originalfeature space via [CLS] token feature generation and utilizes a class con-ditional domain discriminator for adversarial distribution alignment toreduce the domain distribution discrepancy. We further investigate thedual branch of masked source to target domain adaptation, which boostsperformance. We provide a theoretical analysis of our framework from amasked domain adaptation perspective. Extensive experimental studiesdemonstrate the superiority of our framework.