Mental fatigue, a detrimental psychophysiological state induced by high-intensity cognitive tasks, impairs athletes’ attention, reaction, and decision-making, increasing the risk of errors and injuries. Traditional questionnaire-based assessments of mental fatigue suffer from subjectivity and response bias, whereas objective examining and analyzing methods such as electroencephalography (EEG) are often costly and time-consuming, highlighting the need for efficient and convenient objective approaches. This study proposes a hybrid convolutional neural network (CNN)–Transformer model that combines CNN-based feature extraction with Transformer-based global dependency modeling for accurate and efficient mental fatigue recognition. The model was evaluated on the Shanghai Jiao Tong University Emotion EEG Dataset (SEED) and the Sustained-Attention Driving Task (SADT) dataset. The model proposed in this study achieved accuracies of 78.07% and 85.42%, respectively, outperforming conventional methods and demonstrating good cross-subject generalization. Furthermore, channel analysis highlighted the occipital regions’ signal as key contributors to fatigue detection, providing theoretical basis for the development of portable and lightweight device for mental fatigue monitoring. Overall, this work provides a feasible solution for efficient and objective mental fatigue detection, and has potential applications in athletic training monitoring and performance optimization.