Cross-domain few-shot classification (CDFSC) attempts to recognize novel classes with few labeled samples, which is mostly challenged by the low-data problem as well as extreme domain shift between base and novel target classes. Current CDFSC methods always employ a lightweight backbone and continue to use a linear-probe-like traditional fine-tuning (Trad-FT) paradigm. While for recently emerging large-scale pre-trained model (LPM), which has more parameters with considerable prior knowledge, employing Trad-FT will face significant risks of overfitting and prior knowledge damage. In this paper, we propose semantic-guided robustness-invariance tuning (SRIT), a novel fine-tuning paradigm including modulus-matching-based image-text mixup (MMIT-Mixup) and robustness-invariance fine-tuning (RI-FT), to address the CDFSC challenge of LPM. Concretely, SRIT focuses on achieving robust class-specific representation. It first considers textual labels as a robust and domain-invariant conductor, and MMIT-Mixup injects the domain-invariant and class-specific knowledge to obtain domain-invariant prototypes. Then, RI-FT optimizes the distance between features and prototypes to enhance the robustness of visual-encoder. We consider several types of LPMs and conduct extensive experiments on two benchmarks, which reveals that SRIT is a general solution for LPM's CDFSC challenge and outperforms the existing methods with a large margin.