Mental fatigue state recognition method based on convolution neural network and long short-term memory_Journal of Biomedical Engineering

Authors：

WANG Hui ¹ , ZHANG Pin ¹ , JIN Fenghu ² , ZHAO Baoyong ¹ , ZENG Qinbo ² ,  XIAO Wendong ^1,3,4

1. School of Automation, University of Science And Technology Beijing, Beijing 100083, P. R. China;
2. China Ordnance Equipment Group Automation Research Institute Co., Mianyang, Sichuan 621000, P. R. China;
3. Shunde Innovation School, University of Science, and Technology Beijing, Shunde, Guangdong 528399, P. R. China;
4. Beijing Engineering Research Center of Industrial Spectrum Imaging, Beijing 100083, P. R. China;

Corresponding?author：

XIAO Wendong, Email: wdxiao@ustb.edu.cn

Keywords：

Psychological fatigue; Electrocardiogram signals; Convolution neural network; Long short-term memory

DOI：

10.7507/1001-5515.202306016

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

The pace of modern life is accelerating, the pressure of life is gradually increasing, and the long-term accumulation of mental fatigue poses a threat to health. By analyzing physiological signals and parameters, this paper proposes a method that can identify the state of mental fatigue, which helps to maintain a healthy life. The method proposed in this paper is a new recognition method of psychological fatigue state of electrocardiogram signals based on convolutional neural network and long short-term memory. Firstly, the convolution layer of one-dimensional convolutional neural network model is used to extract local features, the key information is extracted through pooling layer, and some redundant data is removed. Then, the extracted features are used as input to the long short-term memory model to further fuse the ECG features. Finally, by integrating the key information through the full connection layer, the accurate recognition of mental fatigue state is successfully realized. The results show that compared with traditional machine learning algorithms, the proposed method significantly improves the accuracy of mental fatigue recognition to 96.3%, which provides a reliable basis for the early warning and evaluation of mental fatigue.

Citation： WANG Hui, ZHANG Pin, JIN Fenghu, ZHAO Baoyong, ZENG Qinbo, XIAO Wendong. Mental fatigue state recognition method based on convolution neural network and long short-term memory. Journal of Biomedical Engineering, 2024, 41(1): 34-40. doi: 10.7507/1001-5515.202306016 Copy

1.	Lipovac D, ?itnik J, Burnard M D. A pilot study examining the suitability of the mental arithmetic task and single-item measures of affective states to assess affective, physiological, and attention restoration at a wooden desk. Journal of Wood Science, 2022, 68: 35.
2.	Lee K F A, Chan E, Car J, et al. Lowering the sampling rate: Heart rate response during cognitive fatigue. Biosensors, 2022, 12(5): 315.
3.	Matsuura Y, Ochi G. The potential of heart rate variability monitoring for mental health assessment in top wheel gymnastics athletes: a single case design. Applied Psychophysiology and Biofeedback, 2023, 48(3): 335-343.
4.	Wang H, Han M, Avouka T, et al. Research on fatigue identification methods based on low-load wearable ECG monitoring devices. Review of Scientific Instruments, 2023, 94(4): 045103.
5.	Lu K, Sj?rs Dahlman A, Karlsson J, et al. Detecting driver fatigue using heart rate variability: a systematic review. Accident Analysis & Prevention, 2022, 178: 106830.
6.	Matuz A, van der Linden D, Darnai G, et al. Generalisable machine learning models trained on heart rate variability data to predict mental fatigue. Scientific Reports, 2022, 12(1): 20023.
7.	Mehmood I, Li H, Qarout Y, et al. Deep learning-based construction equipment operators’ mental fatigue classification using wearable EEG sensor data. Advanced Engineering Informatics, 2023, 56: 101978.
8.	Zorzos I, Kakkos I, Miloulis S T, et al. Applying neural networks with time-frequency features for the detection of mental fatigue. Applied Sciences, 2023, 13(3): 1512.
9.	Wang F, Wan Y, Li M, et al. Recent advances in fatigue detection algorithm based on EEG. Intelligent Automation & Soft Computing, 2023, 35(3): 3573-3586.
10.	Ettahiri H, Vicente J M F, Fechtali T. EEG signals in mental fatigue detection: a comparing study of machine learning technics vs deep learning//International Work-Conference on the Interplay Between Natural and Artificial Computation. Cham: Springer International Publishing, 2022: 625-633.
11.	Ye C, Yin Z, Zhao M, et al. Identification of mental fatigue levels in a language understanding task based on multi-domain EEG features and an ensemble convolutional neural network. Biomedical Signal Processing and Control, 2022, 72: 103360.
12.	Zhang Y, Chen Y, Pan Z. A deep temporal model for mental fatigue detection//2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2018: 1879-1884.
13.	Schmidt P, Reiss A, Duerichen R, et al. Introducing wesad, a multimodal dataset for wearable stress and affect detection//Proceedings of the 20th ACM International Conference on Multimodal Interaction. Boulder: ICMI, 2018: 400-408.
14.	毛玲, 孫即祥, 張國敏, 等. 基于形態濾波的心電信號基線矯正算法. 信號處理, 2008, 24(4): 582-585.
15.	王曉娜, 宋世德. 基于 NI ELVISⅡ 平臺設計 Bainter 陷波器抑制工頻干擾實驗. 實驗室科學, 2020, 23(4): 35-38.
16.	梁瑩, 馬小龍, 朝樂蒙, 等. 基于經驗小波變換的心電信號基線漂移噪聲去除預處理方法研究. 醫療衛生裝備, 2022, 43(6): 7-13.
17.	Wang Y, Deepu C J, Lian Y. A computationally efficient QRS detection algorithm for wearable ECG sensors//2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 2011: 5641-5644.
18.	邵寧寧. 基于卷積神經網絡的心律失常分類研究. 秦皇島: 燕山大學, 2021.
19.	武東輝, 許靜, 陳繼斌等. 基于融合注意力機制與CNN-LSTM的人體行為識別算法. 科學技術與工程, 2023, 23(2): 681-689.
20.	鄭濤, 劉輝, 陳薇, 等. 基于CNN-LSTM的水泥熟料f-CaO預測模型. 控制工程, 2022. DOI: 10.14107/j.cnki.kzgc.20220261.
21.	王愈軒, 劉爾佳, 黃永章. 基于CNN-LSTM-lightGBM組合的超短期風電功率預測方法. 科學技術與工程, 2022, 22(36): 16067-16074.

1. Lipovac D, ?itnik J, Burnard M D. A pilot study examining the suitability of the mental arithmetic task and single-item measures of affective states to assess affective, physiological, and attention restoration at a wooden desk. Journal of Wood Science, 2022, 68: 35.
2. Lee K F A, Chan E, Car J, et al. Lowering the sampling rate: Heart rate response during cognitive fatigue. Biosensors, 2022, 12(5): 315.
3. Matsuura Y, Ochi G. The potential of heart rate variability monitoring for mental health assessment in top wheel gymnastics athletes: a single case design. Applied Psychophysiology and Biofeedback, 2023, 48(3): 335-343.
4. Wang H, Han M, Avouka T, et al. Research on fatigue identification methods based on low-load wearable ECG monitoring devices. Review of Scientific Instruments, 2023, 94(4): 045103.
5. Lu K, Sj?rs Dahlman A, Karlsson J, et al. Detecting driver fatigue using heart rate variability: a systematic review. Accident Analysis & Prevention, 2022, 178: 106830.
6. Matuz A, van der Linden D, Darnai G, et al. Generalisable machine learning models trained on heart rate variability data to predict mental fatigue. Scientific Reports, 2022, 12(1): 20023.
7. Mehmood I, Li H, Qarout Y, et al. Deep learning-based construction equipment operators’ mental fatigue classification using wearable EEG sensor data. Advanced Engineering Informatics, 2023, 56: 101978.
8. Zorzos I, Kakkos I, Miloulis S T, et al. Applying neural networks with time-frequency features for the detection of mental fatigue. Applied Sciences, 2023, 13(3): 1512.
9. Wang F, Wan Y, Li M, et al. Recent advances in fatigue detection algorithm based on EEG. Intelligent Automation & Soft Computing, 2023, 35(3): 3573-3586.
10. Ettahiri H, Vicente J M F, Fechtali T. EEG signals in mental fatigue detection: a comparing study of machine learning technics vs deep learning//International Work-Conference on the Interplay Between Natural and Artificial Computation. Cham: Springer International Publishing, 2022: 625-633.
11. Ye C, Yin Z, Zhao M, et al. Identification of mental fatigue levels in a language understanding task based on multi-domain EEG features and an ensemble convolutional neural network. Biomedical Signal Processing and Control, 2022, 72: 103360.
12. Zhang Y, Chen Y, Pan Z. A deep temporal model for mental fatigue detection//2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2018: 1879-1884.
13. Schmidt P, Reiss A, Duerichen R, et al. Introducing wesad, a multimodal dataset for wearable stress and affect detection//Proceedings of the 20th ACM International Conference on Multimodal Interaction. Boulder: ICMI, 2018: 400-408.
14. 毛玲, 孫即祥, 張國敏, 等. 基于形態濾波的心電信號基線矯正算法. 信號處理, 2008, 24(4): 582-585.
15. 王曉娜, 宋世德. 基于 NI ELVISⅡ 平臺設計 Bainter 陷波器抑制工頻干擾實驗. 實驗室科學, 2020, 23(4): 35-38.
16. 梁瑩, 馬小龍, 朝樂蒙, 等. 基于經驗小波變換的心電信號基線漂移噪聲去除預處理方法研究. 醫療衛生裝備, 2022, 43(6): 7-13.
17. Wang Y, Deepu C J, Lian Y. A computationally efficient QRS detection algorithm for wearable ECG sensors//2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE, 2011: 5641-5644.
18. 邵寧寧. 基于卷積神經網絡的心律失常分類研究. 秦皇島: 燕山大學, 2021.
19. 武東輝, 許靜, 陳繼斌等. 基于融合注意力機制與CNN-LSTM的人體行為識別算法. 科學技術與工程, 2023, 23(2): 681-689.
20. 鄭濤, 劉輝, 陳薇, 等. 基于CNN-LSTM的水泥熟料f-CaO預測模型. 控制工程, 2022. DOI: 10.14107/j.cnki.kzgc.20220261.
21. 王愈軒, 劉爾佳, 黃永章. 基于CNN-LSTM-lightGBM組合的超短期風電功率預測方法. 科學技術與工程, 2022, 22(36): 16067-16074.

Journal of Biomedical Engineering

Mental fatigue state recognition method based on convolution neural network and long short-term memory

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content