Follow control of upper limb rehabilitation training based on Kinect and NAO robot_Journal of Biomedical Engineering

Authors：

LIU Xiaoguang ^1,2 , LI Simin ^1,2 , LIANG Tie ^1,2 , LI Jun ^1,2 , LOU Cunguang ^1,2 , WANG Hongrui ^1,2 ,  LIU Xiuling ^1,2

1. College of Electronic and Information Engineering, Hebei University, Baoding, Hebei 071002, P. R. China;
2. Key Laboratory of Digital Medical Engineering of Hebei Province, Baoding, Hebei 071002, P. R. China;

Corresponding?author：

LIU Xiuling, Email: liuxiuling121@hotmail.com

Keywords：

Kinect; Follow control; NAO; Inverse kinematics; Median filter

DOI：

10.7507/1001-5515.202111009

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Gesture imitation is a common rehabilitation strategy in limb rehabilitation training. In traditional rehabilitation training, patients need to complete training actions under the guidance of rehabilitation physicians. However, due to the limited resources of the hospital, it cannot meet the training and guidance needs of all patients. In this paper, we proposed a following control method based on Kinect and NAO robot for the gesture imitation task in rehabilitation training. The method realized the joint angles mapping from Kinect coordination to NAO robot coordination through inverse kinematics algorithm. Aiming at the deflection angle estimation problem of the elbow joint, a virtual space plane was constructed and realized the accurate estimation of deflection angle. Finally, a comparative experiment for deflection angle of the elbow joint angle was conducted. The experimental results showed that the root mean square error of the angle estimation value of this method in right elbow transverse deflection and vertical deflection directions was 2.734° and 2.159°, respectively. It demonstrates that the method can follow the human movement in real time and stably using the NAO robot to show the rehabilitation training program for patients.

Citation： LIU Xiaoguang, LI Simin, LIANG Tie, LI Jun, LOU Cunguang, WANG Hongrui, LIU Xiuling. Follow control of upper limb rehabilitation training based on Kinect and NAO robot. Journal of Biomedical Engineering, 2022, 39(6): 1189-1198. doi: 10.7507/1001-5515.202111009 Copy

1.	王嘉津, 左國坤, 張佳楫, 等. 腕功能康復機器人按需輔助控制策略研究. 生物醫學工程學雜志, 2020, 37(1): 129-135.
2.	程洪, 黃瑞, 邱靜, 等. 康復機器人及其臨床應用綜述. 機器人, 2021, 43(5): 606-619.
3.	Afsar S I, Mirzayev I, Yemisci O U, et al. Virtual reality in upper extremity rehabilitation of stroke patients: a randomized controlled trial. J Stroke Cerebrovasc Dis, 2018, 27(12): 3473-3478.
4.	Schaham N G, Zeilig G, Weingarden H, et al. Game analysis and clinical use of the Xbox-Kinect for stroke rehabilitation. Int J Rehabil Res, 2018, 41(4): 323-330.
5.	Sanchez L, Courtine A, Gerard-Flavian G, et al. The use of a social assistive robot: NAO for post strokes rehabilitation therapy: a preliminary study. Comput Methods Biomech Biomed Eng, 2019, 22(Sup1): S478-S480.
6.	Bakhti K K A, Laffont I, Muthalib M, et al. Kinect-based assessment of proximal arm non-use after a stroke. J Neuroeng Rehabil, 2018, 15(1): 1-12.
7.	Semblantes P A, Andaluz V H, Lagla J, et al. Visual feedback framework for rehabilitation of stroke patients. Inform Med Unlocked, 2018, 13: 41-50.
8.	Dubey V N, Manna S K. Design of a game-based rehabilitation system using kinect sensor// 2019 Design of Medical Devices Conference. Minneapolis: ASME, 2019: V001T03A005.
9.	Cai L, Ma Y, Xiong S, et al. Validity and reliability of upper limb functional assessment using the microsoft Kinect V2 sensor. Appl Bionics Biomech, 2019, 2019: 7175240.
10.	侯增廣, 趙新剛, 程龍, 等. 康復機器人與智能輔助系統的研究進展. 自動化學報, 2016, 42(12): 1765-1779.
11.	吳宏健, 李莉娜, 李龍, 等. 腦卒中后手功能康復機器人綜合干預研究進展. 生物醫學工程學雜志, 2019, 36(1): 151-156.
12.	Yav?an E, U?ar A. Gesture imitation and recognition using Kinect sensor and extreme learning machines. Measurement, 2016, 94: 852-861.
13.	Elbasiony R, Gomaa W. Humanoids skill learning based on real-time human motion imitation using Kinect. Intel Serv Robot, 2018, 11(2): 149-169.
14.	Hu N, Zheng L. Human action imitation system based on NAO robot// 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC). Chengdu: IEEE, 2019: 2261-2264.
15.	Wei Y, Zhao J. Designing human-like behaviors for anthropomorphic arm in humanoid robot NAO. Robotica, 2020, 38(7): 1205-1226.
16.	黃玲濤, 王彬, 倪濤, 等. 基于Kinect的機器人抓取系統研究. 農業機械學報, 2019, 50(1): 390-399.
17.	于建均, 門玉森, 阮曉鋼, 等. 基于Kinect的NAO機器人動作模仿系統的研究與實現. 智能系統學報, 2016, 11(2): 180-187.
18.	滑少揚. 基于Kinect的NAO機器人的模仿動作的研究. 秦皇島: 燕山大學, 2017.
19.	Zhang Z, Niu Y, Kong L, et al. A real-time upper-body robot imitation system. Int J Robot Control, 2019, 2(1): 49.
20.	Wang Z, Liang R, Chen Z, et al. Fast and intuitive kinematics mapping for human-robot motion imitating: A virtual-joint-based approach. IFAC-PapersOnLine, 2020, 53(2): 10011-10018.
21.	Assad-Uz-Zaman M, Islam M R, Rahman M H, et al. Kinect controlled NAO robot for telerehabilitation. J Intell Syst, 2020, 30(1): 224-239.
22.	Assad-Uz-Zaman M, Islam M R, Rahman M H, et al. Robot sensor system for supervised rehabilitation with real-time feedback. Multimed Tools Appl, 2020, 79(3): 26643-26660.
23.	冷舒, 吳克, 居鶴華. 機械臂運動學建模及解算方法綜述. 宇航學報, 2019, 40(11): 1262-1273.
24.	馬叢俊, 趙濤, 向國菲, 等. 基于逆運動學的柔性機械臂末端定位控制. 機械工程學報, 2021, 57(13): 163-171.
25.	Zhang T, Song Y, Wu H, et al. A novel method to identify DH parameters of the rigid serial-link robot based on a geometry model. Ind Robot, 2020, 48(1): 157-167.
26.	Liu X, Qiu C, Zeng Q, et al. Kinematics analysis and trajectory planning of collaborative welding robot with multiple manipulators. Procedia CIRP, 2019, 81: 1034-1039.
27.	Alam M M, Ibaraki S, Fukuda K. Kinematic modeling of six-axis industrial robot and its parameter identification: a tutorial. Int J Auto Tech-Kor, 2021, 15(5): 599-610.
28.	Xu N, Peng X, Peng L, et al. Modeling and kinematics analysis of a novel 5-DOF upper limb exoskeleton rehabilitation robot// 2020 39th Chinese Control Conference (CCC). Shenyang: IEEE, 2020: 1052-1057.
29.	Guo F, Cai H, Ceccarelli M, et al. Enhanced D-H: an improved convention for establishing a robot link coordinate system fixed on the joint. Ind Robot, 2019, 47(2): 197-205.
30.	Guida R, De Simone M C, Da?i? P, et al. Modeling techniques for kinematic analysis of a six-axis robotic arm// IOP Conference Series: Materials Science and Engineering. Wuhan: IOP, 2019, 568(1): 012115.
31.	胡奎, 張繼文, 董云飛, 等. 針對關節限位優化的7自由度機械臂逆運動學解法. 清華大學學報(自然科學版), 2020, 60(12): 1007-1015.
32.	周書華, 張文輝, 聞志, 等. 直角坐標機器人基于D-H參數的運動學建模與軌跡規劃. 電工技術, 2020(24): 78-80, 83.
33.	T?lgyessy M, Dekan M, Chovanec ?, et al. Evaluation of the azure Kinect and its comparison to Kinect V1 and Kinect V2. Sensors, 2021, 21(2): 413.
34.	Assad-Uz-Zaman M, Islam M R, Rahman M H. Upper-extremity rehabilitation with NAO robot// 5th International Conference of Control, Dynamic Systems, and Robotics. Niagara Falls: CDSR, 2018: 117.
35.	劉曉光, 李夢楠, 王立玲, 等. 基于表面肌電信號的康復過程中肌疲勞有效性分析. 生物醫學工程學雜志, 2019, 36(1): 80-84.

1. 王嘉津, 左國坤, 張佳楫, 等. 腕功能康復機器人按需輔助控制策略研究. 生物醫學工程學雜志, 2020, 37(1): 129-135.
2. 程洪, 黃瑞, 邱靜, 等. 康復機器人及其臨床應用綜述. 機器人, 2021, 43(5): 606-619.
3. Afsar S I, Mirzayev I, Yemisci O U, et al. Virtual reality in upper extremity rehabilitation of stroke patients: a randomized controlled trial. J Stroke Cerebrovasc Dis, 2018, 27(12): 3473-3478.
4. Schaham N G, Zeilig G, Weingarden H, et al. Game analysis and clinical use of the Xbox-Kinect for stroke rehabilitation. Int J Rehabil Res, 2018, 41(4): 323-330.
5. Sanchez L, Courtine A, Gerard-Flavian G, et al. The use of a social assistive robot: NAO for post strokes rehabilitation therapy: a preliminary study. Comput Methods Biomech Biomed Eng, 2019, 22(Sup1): S478-S480.
6. Bakhti K K A, Laffont I, Muthalib M, et al. Kinect-based assessment of proximal arm non-use after a stroke. J Neuroeng Rehabil, 2018, 15(1): 1-12.
7. Semblantes P A, Andaluz V H, Lagla J, et al. Visual feedback framework for rehabilitation of stroke patients. Inform Med Unlocked, 2018, 13: 41-50.
8. Dubey V N, Manna S K. Design of a game-based rehabilitation system using kinect sensor// 2019 Design of Medical Devices Conference. Minneapolis: ASME, 2019: V001T03A005.
9. Cai L, Ma Y, Xiong S, et al. Validity and reliability of upper limb functional assessment using the microsoft Kinect V2 sensor. Appl Bionics Biomech, 2019, 2019: 7175240.
10. 侯增廣, 趙新剛, 程龍, 等. 康復機器人與智能輔助系統的研究進展. 自動化學報, 2016, 42(12): 1765-1779.
11. 吳宏健, 李莉娜, 李龍, 等. 腦卒中后手功能康復機器人綜合干預研究進展. 生物醫學工程學雜志, 2019, 36(1): 151-156.
12. Yav?an E, U?ar A. Gesture imitation and recognition using Kinect sensor and extreme learning machines. Measurement, 2016, 94: 852-861.
13. Elbasiony R, Gomaa W. Humanoids skill learning based on real-time human motion imitation using Kinect. Intel Serv Robot, 2018, 11(2): 149-169.
14. Hu N, Zheng L. Human action imitation system based on NAO robot// 2019 IEEE 3rd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC). Chengdu: IEEE, 2019: 2261-2264.
15. Wei Y, Zhao J. Designing human-like behaviors for anthropomorphic arm in humanoid robot NAO. Robotica, 2020, 38(7): 1205-1226.
16. 黃玲濤, 王彬, 倪濤, 等. 基于Kinect的機器人抓取系統研究. 農業機械學報, 2019, 50(1): 390-399.
17. 于建均, 門玉森, 阮曉鋼, 等. 基于Kinect的NAO機器人動作模仿系統的研究與實現. 智能系統學報, 2016, 11(2): 180-187.
18. 滑少揚. 基于Kinect的NAO機器人的模仿動作的研究. 秦皇島: 燕山大學, 2017.
19. Zhang Z, Niu Y, Kong L, et al. A real-time upper-body robot imitation system. Int J Robot Control, 2019, 2(1): 49.
20. Wang Z, Liang R, Chen Z, et al. Fast and intuitive kinematics mapping for human-robot motion imitating: A virtual-joint-based approach. IFAC-PapersOnLine, 2020, 53(2): 10011-10018.
21. Assad-Uz-Zaman M, Islam M R, Rahman M H, et al. Kinect controlled NAO robot for telerehabilitation. J Intell Syst, 2020, 30(1): 224-239.
22. Assad-Uz-Zaman M, Islam M R, Rahman M H, et al. Robot sensor system for supervised rehabilitation with real-time feedback. Multimed Tools Appl, 2020, 79(3): 26643-26660.
23. 冷舒, 吳克, 居鶴華. 機械臂運動學建模及解算方法綜述. 宇航學報, 2019, 40(11): 1262-1273.
24. 馬叢俊, 趙濤, 向國菲, 等. 基于逆運動學的柔性機械臂末端定位控制. 機械工程學報, 2021, 57(13): 163-171.
25. Zhang T, Song Y, Wu H, et al. A novel method to identify DH parameters of the rigid serial-link robot based on a geometry model. Ind Robot, 2020, 48(1): 157-167.
26. Liu X, Qiu C, Zeng Q, et al. Kinematics analysis and trajectory planning of collaborative welding robot with multiple manipulators. Procedia CIRP, 2019, 81: 1034-1039.
27. Alam M M, Ibaraki S, Fukuda K. Kinematic modeling of six-axis industrial robot and its parameter identification: a tutorial. Int J Auto Tech-Kor, 2021, 15(5): 599-610.
28. Xu N, Peng X, Peng L, et al. Modeling and kinematics analysis of a novel 5-DOF upper limb exoskeleton rehabilitation robot// 2020 39th Chinese Control Conference (CCC). Shenyang: IEEE, 2020: 1052-1057.
29. Guo F, Cai H, Ceccarelli M, et al. Enhanced D-H: an improved convention for establishing a robot link coordinate system fixed on the joint. Ind Robot, 2019, 47(2): 197-205.
30. Guida R, De Simone M C, Da?i? P, et al. Modeling techniques for kinematic analysis of a six-axis robotic arm// IOP Conference Series: Materials Science and Engineering. Wuhan: IOP, 2019, 568(1): 012115.
31. 胡奎, 張繼文, 董云飛, 等. 針對關節限位優化的7自由度機械臂逆運動學解法. 清華大學學報(自然科學版), 2020, 60(12): 1007-1015.
32. 周書華, 張文輝, 聞志, 等. 直角坐標機器人基于D-H參數的運動學建模與軌跡規劃. 電工技術, 2020(24): 78-80, 83.
33. T?lgyessy M, Dekan M, Chovanec ?, et al. Evaluation of the azure Kinect and its comparison to Kinect V1 and Kinect V2. Sensors, 2021, 21(2): 413.
34. Assad-Uz-Zaman M, Islam M R, Rahman M H. Upper-extremity rehabilitation with NAO robot// 5th International Conference of Control, Dynamic Systems, and Robotics. Niagara Falls: CDSR, 2018: 117.
35. 劉曉光, 李夢楠, 王立玲, 等. 基于表面肌電信號的康復過程中肌疲勞有效性分析. 生物醫學工程學雜志, 2019, 36(1): 80-84.

Journal of Biomedical Engineering

Follow control of upper limb rehabilitation training based on Kinect and NAO robot

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content