Low illumination significantly impacts the performance of learning-based models trained in well-lit conditions. Although current methods alleviate this issue through either image-level enhancement or feature-level adaptation, they often focus solely on the image itself, ignoring how the task-relevant target varies along with different illumination. In this paper, we propose a target-aware representation learning framework designed to improve high-level task performance in low-illumination environments. We first achieve a bi-directional domain alignment from both image appearance and semantic features to bridge data under varying illumination conditions. To better focus on the target itself, we design a target highlighting strategy, incorporated with the saliency mechanism and Temporal Gaussian Mixture Model to emphasize the location and movement of task-relevant targets. We also design a mask token-based representation learning scheme to learn a more robust target-aware feature. Our framework ensures more robust and effective feature representation for high-level vision tasks in low illumination. Extensive experiments conducted on CODaN, ExDark, and ARID datasets validate the effectiveness of our method for both image and video-based tasks, such as classification, detection, and action recognition. The code will be released upon acceptance.