Interpretation and application of the quality assessment tool (QATSM-RWS) for real-world studies in systematic reviews and meta-analysis_Chinese Journal of Evidence-Based Medicine

Authors：

ZHANG Qiang ¹ ,  LU Cuncun ² ,  ZHAO Wenxia ¹ , LIU Minghao ¹ , YAN Le ¹ , ZHANG Lihui ¹ , ZHAO Qing ¹ , LI Ling ¹ , WANG Siying ¹ , JIA Pan ¹ , SHANG Dongfang ¹ , FANG Jinhua ³

1. The First Affiliated Hospital of Henan University of Chinese Medicine, Zhengzhou 450000, P. R. China;
2. Institute of Basic Research in Clinical Medicine, China Academy of Chinese Medical Sciences, Beijing 100700, P. R. China;
3. Jianshi Hospital of Traditional Chinese Medicine, Enshi 445300, P. R. China;

Corresponding?author：

LU Cuncun, Email: cuncunlu2017@163.com; ZHAO Wenxia, Email: wenxia-zhao@163.com

Keywords：

Real-world study; Systematic review; Meta-analysis; Quality evaluation; Tool; MASLD

DOI：

10.7507/1672-2531.202406189

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Focusing on research quality is a crucial aspect of modern evidence-based medical practice, providing substantial evidence to underpin clinical decision-making. The increase in real-world studies in recent years has presented challenges, with varying quality stemming from issues such as data integrity and researchers’ expertise levels. Although systematic reviews and meta-analyses are essential references for clinical decisions, their reliability is contingent upon the quality of the primary studies. Making clinical decisions based on inadequate research poses inherent risks. With the lack of a specialized tool for evaluating the quality of real-world studies within systematic reviews and meta-analyses, the Gebrye team has introduced a new assessment tool - QATSM-RWS. Comprising 5 modules and 14 items, this tool aims to improve real-world research evaluation. This article aims to elaborate on the tool’s development process and content, using this tool to evaluate a published real-world study as an example and providing valuable guidance for domestic researchers utilizing this innovative tool.

Citation： ZHANG Qiang, LU Cuncun, ZHAO Wenxia, LIU Minghao, YAN Le, ZHANG Lihui, ZHAO Qing, LI Ling, WANG Siying, JIA Pan, SHANG Dongfang, FANG Jinhua. Interpretation and application of the quality assessment tool (QATSM-RWS) for real-world studies in systematic reviews and meta-analysis. Chinese Journal of Evidence-Based Medicine, 2024, 24(12): 1451-1457. doi: 10.7507/1672-2531.202406189 Copy

1.	Liao X, Yao C, Zhang J, et al. Recent advancement in integrating artificial intelligence and information technology with real-world data for clinical decision-making in China: a scoping review. J Evid Based Med, 2023, 16(4): 534-546.
2.	Chen Z, Liu X, Hogan W, et al. Applications of artificial intelligence in drug development using real-world data. Drug Discov Today, 2021, 26(5): 1256-1264.
3.	Xu J, Wu W, Zhang X, et al. The use of real-world evidence for regulatory decisions in China. Clin Pharmacol Ther, 2024, 116(1): 82-95.
4.	盧存存, 陳子佳, 王志飛, 等. 基于真實世界數據的觀察性因果推斷研究新框架(目標試驗模擬)及其在中醫藥領域中的應用展望. 協和醫學雜志, 2024, 15(2): 422-428.
5.	Brunstr?m M, Thomopoulos C, Carlberg B, et al. Methodological aspects of meta-analyses assessing the effect of blood pressure-lowering treatment on clinical outcomes. Hypertension, 2022, 79(3): 491-504.
6.	Büttner F, Winters M, Delahunt E, et al. Identifying the 'incredible'! Part 2: spot the difference - a rigorous risk of bias assessment can alter the main findings of a systematic review. Br J Sports Med, 2020, 54(13): 801-808.
7.	Sterne JAC, Savovi? J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ, 2019, 366: l4898.
8.	Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol, 2010, 25(9): 603-605.
9.	盧存存, 陳子佳, 張強, 等. 基于真實世界數據的目標試驗模擬研究: 現狀與展望. 中國循證醫學雜志, 2023, 23(4): 492-496.
10.	Gebrye T, Mbada C, Hakimi Z, et al. Development of quality assessment tool for systematic reviews and meta-analyses of real-world studies: a Delphi consensus survey. Rheumatol Int, 2024, 44(7): 1275-1281.
11.	Gebrye T, Fatoye F, Mbada C, et al. A scoping review on quality assessment tools used in systematic reviews and meta-analysis of real-world studies. Rheumatol Int, 2023, 43(9): 1573-1581.
12.	Ma YL, Ke JF, Wang JW, et al. Blood lactate levels are associated with an increased risk of metabolic dysfunction-associated fatty liver disease in type 2 diabetes: a real-world study. Front Endocrinol Lausanne, 2023, 14: 1133991.
13.	秦雪妮, 陳維生, 邵華, 等. 真實世界研究在醫藥領域的應用及研究方法. 藥學進展, 2021, 45(7): 512-523.
14.	王瑞平, 李斌. 臨床醫學研究選題的規范和要點. 上海醫藥, 2023, 44(17): 47-50.
15.	Busse C, August E. How to write and publish a research paper for a peer-reviewed journal. J Cancer Educ, 2021, 36(5): 909-913.
16.	Bao H, Teplitskiy M. A simulation-based analysis of the impact of rhetorical citations in science. Nat Commun, 2024, 15(1): 431.
17.	Bozkurt S, Cahan EM, Seneviratne MG, et al. Reporting of demographic data and representativeness in machine learning models using electronic health records. J Am Med Inform Assoc, 2020, 27(12): 1878-1884.
18.	Desai RJ, Wang SV, Sreedhara SK, et al. Process guide for inferential studies using healthcare data from routine clinical practice to evaluate causal effects of drugs (PRINCIPLED): considerations from the FDA Sentinel Innovation Center. BMJ, 2024, 384: e076460.
19.	Stürmer T, Wang T, Golightly YM, et al. Methodological considerations when analysing and interpreting real-world data. Rheumatology Oxford, 2020, 59(1): 14-25.
20.	國家藥監局藥審中心. 藥物真實世界研究設計與方案框架指導原則(試行). 2023.
21.	Fleurence RL, Kent S, Adamson B, et al. Assessing real-world data from electronic health records for health technology assessment: the SUITABILITY checklist: a good practices report of an ISPOR Task Force. Value Health, 2024, 27(6): 692-701.
22.	翟靜波, 鄭文科, 王輝, 等. 真實世界研究樣本量估計的統計學考慮. 世界中醫藥, 2019, 14(12): 3123-3126.
23.	趙瑞霞, 邵明義, 符宇, 等. 基于真實世界中醫電子病歷數據的原發性肝癌療效評價難點及解決策略. 中醫雜志, 2019, 60(23): 2009-2012.
24.	賈玉龍, 姚明宏, 徐嘉悅, 等. 真實世界研究中 Zelen 設計應用探討. 中國循證醫學雜志, 2022, 22(5): 615-620.
25.	張強, 蒙萍, 單愛蓮. 關于藥物臨床試驗方案中納入、排除標準的若干思考. 中國臨床藥理學雜志, 2017, 33(2): 99-101.
26.	彭曉霞, 舒嘯塵, 譚婧, 等. 基于真實世界數據評價治療結局的觀察性研究設計技術規范. 中國循證醫學雜志, 2019, 19(7): 779-786.
27.	史夢龍, 張晨瑤, 吳曉蕾, 等. “以患者為中心藥物研發”中患者體驗數據的收集技術要點. 中國循證醫學雜志, 2024, 24(4): 478-483.
28.	Kottner J, Beaton D, Clarke M, et al. Core outcome set developers should consider and specify the level of granularity of outcome domains. J Clin Epidemiol, 2024, 169: 111307.
29.	王琳琳, 趙瑞霞, 符宇, 等. 真實世界研究中隨訪面臨的問題及解決策略. 空軍軍醫大學學報, 2022, 43(9): 1051-1054.
30.	高培, 王楊, 羅劍鋒, 等. 基于真實世界數據評價治療結局研究的統計分析技術規范. 中國循證醫學雜志, 2019, 19(7): 787-793.
31.	晉菲斐, 閻小妍, 董沖亞, 等. 真實世界研究方案設計要點及統計分析解析. 中國卒中雜志, 2022, 17(12): 1294-1298.
32.	張穎, 王麗瓊, 費宇彤, 等. 統計分析計劃(SAP)報告指南的解讀及其對中醫藥臨床研究的啟示. 中醫雜志, 2019, 60(5): 385-390.
33.	Castelo-Branco L, Pellat A, Martins-Branco D, et al. ESMO guidance for reporting oncology real-world evidence (GROW). Ann Oncol, 2023, 34(12): 1097-1112.
34.	金鑫瑤, 鄭文科, 張俊華, 等. 推進真實世界研究的透明化. 世界中醫藥, 2019, 14(12): 3106-3110.
35.	李雨芯, 熊俊, 張政, 等. 中藥系統評價/Meta分析報告規范(PRISMA-CHM)解讀. 中國循證醫學雜志, 2023, 23(11): 1351-1359.
36.	Akl EA, Hakoum M, Khamis A, et al. A framework is proposed for defining, categorizing, and assessing conflicts of interest in health research. J Clin Epidemiol, 2022, 149: 236-243.
37.	Siena LM, Papamanolis L, Siebert MJ, et al. Industry involvement and transparency in the most cited clinical trials, 2019-2022. JAMA Netw Open, 2023, 6(11): e2343425.
38.	DeVos E, Simon EL, Aluisio A. Funding sources for research: a research primer for low- and middle-income countries. Afr J Emerg Med, 2020, 10(Suppl 2): S130-S134.
39.	Jansen MS, Dekkers OM, le Cessie S, et al. Real-world evidence to inform regulatory decision making: a scoping review. Clin Pharmacol Ther, 2024, 115(6): 1269-1276.
40.	Lu C, Ke L, Zhang Q, et al. Quality of systematic reviews with meta-analyses of resveratrol: a methodological systematic review. Phytother Res, 2024, 38(1): 11-21.
41.	Fatoye F, Smith P, Gebrye T, et al. Real-world persistence and adherence with oral bisphosphonates for osteoporosis: a systematic review. BMJ Open, 2019, 9(4): e027049.
42.	Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ, 2016, 355: i4919.

1. Liao X, Yao C, Zhang J, et al. Recent advancement in integrating artificial intelligence and information technology with real-world data for clinical decision-making in China: a scoping review. J Evid Based Med, 2023, 16(4): 534-546.
2. Chen Z, Liu X, Hogan W, et al. Applications of artificial intelligence in drug development using real-world data. Drug Discov Today, 2021, 26(5): 1256-1264.
3. Xu J, Wu W, Zhang X, et al. The use of real-world evidence for regulatory decisions in China. Clin Pharmacol Ther, 2024, 116(1): 82-95.
4. 盧存存, 陳子佳, 王志飛, 等. 基于真實世界數據的觀察性因果推斷研究新框架(目標試驗模擬)及其在中醫藥領域中的應用展望. 協和醫學雜志, 2024, 15(2): 422-428.
5. Brunstr?m M, Thomopoulos C, Carlberg B, et al. Methodological aspects of meta-analyses assessing the effect of blood pressure-lowering treatment on clinical outcomes. Hypertension, 2022, 79(3): 491-504.
6. Büttner F, Winters M, Delahunt E, et al. Identifying the 'incredible'! Part 2: spot the difference - a rigorous risk of bias assessment can alter the main findings of a systematic review. Br J Sports Med, 2020, 54(13): 801-808.
7. Sterne JAC, Savovi? J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ, 2019, 366: l4898.
8. Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol, 2010, 25(9): 603-605.
9. 盧存存, 陳子佳, 張強, 等. 基于真實世界數據的目標試驗模擬研究: 現狀與展望. 中國循證醫學雜志, 2023, 23(4): 492-496.
10. Gebrye T, Mbada C, Hakimi Z, et al. Development of quality assessment tool for systematic reviews and meta-analyses of real-world studies: a Delphi consensus survey. Rheumatol Int, 2024, 44(7): 1275-1281.
11. Gebrye T, Fatoye F, Mbada C, et al. A scoping review on quality assessment tools used in systematic reviews and meta-analysis of real-world studies. Rheumatol Int, 2023, 43(9): 1573-1581.
12. Ma YL, Ke JF, Wang JW, et al. Blood lactate levels are associated with an increased risk of metabolic dysfunction-associated fatty liver disease in type 2 diabetes: a real-world study. Front Endocrinol Lausanne, 2023, 14: 1133991.
13. 秦雪妮, 陳維生, 邵華, 等. 真實世界研究在醫藥領域的應用及研究方法. 藥學進展, 2021, 45(7): 512-523.
14. 王瑞平, 李斌. 臨床醫學研究選題的規范和要點. 上海醫藥, 2023, 44(17): 47-50.
15. Busse C, August E. How to write and publish a research paper for a peer-reviewed journal. J Cancer Educ, 2021, 36(5): 909-913.
16. Bao H, Teplitskiy M. A simulation-based analysis of the impact of rhetorical citations in science. Nat Commun, 2024, 15(1): 431.
17. Bozkurt S, Cahan EM, Seneviratne MG, et al. Reporting of demographic data and representativeness in machine learning models using electronic health records. J Am Med Inform Assoc, 2020, 27(12): 1878-1884.
18. Desai RJ, Wang SV, Sreedhara SK, et al. Process guide for inferential studies using healthcare data from routine clinical practice to evaluate causal effects of drugs (PRINCIPLED): considerations from the FDA Sentinel Innovation Center. BMJ, 2024, 384: e076460.
19. Stürmer T, Wang T, Golightly YM, et al. Methodological considerations when analysing and interpreting real-world data. Rheumatology Oxford, 2020, 59(1): 14-25.
20. 國家藥監局藥審中心. 藥物真實世界研究設計與方案框架指導原則(試行). 2023.
21. Fleurence RL, Kent S, Adamson B, et al. Assessing real-world data from electronic health records for health technology assessment: the SUITABILITY checklist: a good practices report of an ISPOR Task Force. Value Health, 2024, 27(6): 692-701.
22. 翟靜波, 鄭文科, 王輝, 等. 真實世界研究樣本量估計的統計學考慮. 世界中醫藥, 2019, 14(12): 3123-3126.
23. 趙瑞霞, 邵明義, 符宇, 等. 基于真實世界中醫電子病歷數據的原發性肝癌療效評價難點及解決策略. 中醫雜志, 2019, 60(23): 2009-2012.
24. 賈玉龍, 姚明宏, 徐嘉悅, 等. 真實世界研究中 Zelen 設計應用探討. 中國循證醫學雜志, 2022, 22(5): 615-620.
25. 張強, 蒙萍, 單愛蓮. 關于藥物臨床試驗方案中納入、排除標準的若干思考. 中國臨床藥理學雜志, 2017, 33(2): 99-101.
26. 彭曉霞, 舒嘯塵, 譚婧, 等. 基于真實世界數據評價治療結局的觀察性研究設計技術規范. 中國循證醫學雜志, 2019, 19(7): 779-786.
27. 史夢龍, 張晨瑤, 吳曉蕾, 等. “以患者為中心藥物研發”中患者體驗數據的收集技術要點. 中國循證醫學雜志, 2024, 24(4): 478-483.
28. Kottner J, Beaton D, Clarke M, et al. Core outcome set developers should consider and specify the level of granularity of outcome domains. J Clin Epidemiol, 2024, 169: 111307.
29. 王琳琳, 趙瑞霞, 符宇, 等. 真實世界研究中隨訪面臨的問題及解決策略. 空軍軍醫大學學報, 2022, 43(9): 1051-1054.
30. 高培, 王楊, 羅劍鋒, 等. 基于真實世界數據評價治療結局研究的統計分析技術規范. 中國循證醫學雜志, 2019, 19(7): 787-793.
31. 晉菲斐, 閻小妍, 董沖亞, 等. 真實世界研究方案設計要點及統計分析解析. 中國卒中雜志, 2022, 17(12): 1294-1298.
32. 張穎, 王麗瓊, 費宇彤, 等. 統計分析計劃(SAP)報告指南的解讀及其對中醫藥臨床研究的啟示. 中醫雜志, 2019, 60(5): 385-390.
33. Castelo-Branco L, Pellat A, Martins-Branco D, et al. ESMO guidance for reporting oncology real-world evidence (GROW). Ann Oncol, 2023, 34(12): 1097-1112.
34. 金鑫瑤, 鄭文科, 張俊華, 等. 推進真實世界研究的透明化. 世界中醫藥, 2019, 14(12): 3106-3110.
35. 李雨芯, 熊俊, 張政, 等. 中藥系統評價/Meta分析報告規范(PRISMA-CHM)解讀. 中國循證醫學雜志, 2023, 23(11): 1351-1359.
36. Akl EA, Hakoum M, Khamis A, et al. A framework is proposed for defining, categorizing, and assessing conflicts of interest in health research. J Clin Epidemiol, 2022, 149: 236-243.
37. Siena LM, Papamanolis L, Siebert MJ, et al. Industry involvement and transparency in the most cited clinical trials, 2019-2022. JAMA Netw Open, 2023, 6(11): e2343425.
38. DeVos E, Simon EL, Aluisio A. Funding sources for research: a research primer for low- and middle-income countries. Afr J Emerg Med, 2020, 10(Suppl 2): S130-S134.
39. Jansen MS, Dekkers OM, le Cessie S, et al. Real-world evidence to inform regulatory decision making: a scoping review. Clin Pharmacol Ther, 2024, 115(6): 1269-1276.
40. Lu C, Ke L, Zhang Q, et al. Quality of systematic reviews with meta-analyses of resveratrol: a methodological systematic review. Phytother Res, 2024, 38(1): 11-21.
41. Fatoye F, Smith P, Gebrye T, et al. Real-world persistence and adherence with oral bisphosphonates for osteoporosis: a systematic review. BMJ Open, 2019, 9(4): e027049.
42. Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ, 2016, 355: i4919.

Chinese Journal of Evidence-Based Medicine

Interpretation and application of the quality assessment tool (QATSM-RWS) for real-world studies in systematic reviews and meta-analysis

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content