This paper aims to adapt the model to the target environment by leveraging large unlabeled target data and small user feedback readily available in real-world applications. We find that existing semi-supervised domain adaptation (SemiSDA) methods often suffer from poorly improved adaptation performance when directly utilizing such data. We analyze this phenomenon via a novel concept called Negatively Biased Feedback (NBF), which stems from the observation that user feedback is more likely for data points where the model produces incorrect predictions. To leverage such feedback without the problem, we propose a scalable adapting approach, Class-space Defending, which can seamlessly combine with existing SemiSDA methods. This approach helps the SemiSDA method to adapt the model with a balanced supervised signal by utilizing our defending samples throughout the adaptation process. We demonstrate the problem caused by NBF and the efficacy of our approach across various benchmarks, including image classification, semantic segmentation, and a real-world medical imaging application. Our extensive experiments show that significant performance improvements can be achieved by integrating our approach with multiple state-of-the-art SemiSDA methods.