Class-incremental learning is a challenging problem, where the goal is to train a model that can classify data from an increasing number of classes over time. With the advancement of vision-language pre-trained models such as CLIP, they demonstrate good generalization ability that allows them to excel in class-incremental learning with completely frozen parameters. However, further adaptation to downstream tasks by simply fine-tuning the model leads to severe forgetting. Most existing works with pre-trained models assume that the forgetting of old classes is uniform when the model acquires new knowledge. In this paper, we propose a method that leverages the textual features of class names to measure the degree of influence on old classes by new classes and adjusts their representations accordingly to reduce forgetting. In addition, we also propose a decomposed parameter fusion method for the adapter module. It can greatly reduce the forgetting caused by fine-tuning the adapter modules with new data. Experiments on several conventional benchmarks show that our method achieves state-of-the-art results.