Research progress of multimodal magnetic resonance imaging brain tumor segmentation based on fused neural network model_Journal of Biomedical Engineering

Authors：

 YANG Wenyang ¹ , DU Chao ¹ , ZHANG Ruijie ¹ , Keung Steven ²

1. School of Computer Science, Xi’an Shiyou University, Xi’an 710065, P. R. China;
2. Department of Medical Imaging and Radiation Sciences, Monash University, Clayton, VIC 3800, Australian;

Corresponding?author：

YANG Wenyang, Email: ywy80910@163.com

Keywords：

Brain tumor segmentation; Multimodal; Feature fusion; Neural network

DOI：

10.7507/1001-5515.202411015

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

In clinical diagnosis of brain tumors, accurate segmentation based on multimodal magnetic resonance imaging (MRI) is essential for determining tumor type, extent, and spatial boundaries. However, differences in imaging mechanisms, information emphasis, and feature distributions among multimodal MRI data have posed significant challenges for precise tumor modeling and fusion-based segmentation. In recent years, fusion neural networks have provided effective strategies for integrating multimodal information and have become a major research focus in multimodal brain tumor segmentation. This review systematically summarized relevant studies on fusion neural networks for multimodal brain tumor segmentation published since 2019. First, the fundamental concepts of multimodal data fusion and model fusion were introduced. Then, existing methods were categorized into three types according to fusion levels: prediction fusion models, feature fusion models, and stage fusion models, and their structural characteristics and segmentation performance were comparatively analyzed. Finally, current limitations were discussed, and potential development trends of fusion neural networks for multimodal MRI brain tumor segmentation were summarized. This review aims to provide references for the design and optimization of future multimodal brain tumor segmentation models.

Citation： YANG Wenyang, DU Chao, ZHANG Ruijie, Keung Steven. Research progress of multimodal magnetic resonance imaging brain tumor segmentation based on fused neural network model. Journal of Biomedical Engineering, 2026, 43(1): 199-207. doi: 10.7507/1001-5515.202411015 Copy

1.	Tataei S N, Ranjbarzadeh R, Jafarzadeh G S, et al. Glioma brain tumor segmentation in four MRI modalities using a convolutional neural network and based on a transfer learning method// Iano Y, Saotome O, Kemper Vásquez G L, et al. Proceedings of the 7th Brazilian Technology Symposium (BTSym’21). BTSym 2021. Smart Innovation, Systems and Technologies. Cham: Springer, 2021, 207: 386-402.
2.	Mancuso V, Bruni F, Stramba-Badiale C, et al. How do emotions elicited in virtual reality affect our memory? A systematic review. Comput Hum Behav, 2023, 146: 107812.
3.	Balwant M K. A review on convolutional neural networks for brain tumor segmentation: methods, datasets, libraries, and future directions. IRBM, 2022, 43(6): 521-537.
4.	Aziz P A, Memon S F, Hussain M, et al. Supratotal resection: An emerging concept of glioblastoma multiforme surgery—systematic review and meta-analysis. World Neurosurg, 2023, 179: e46-e55.
5.	Jyothi P, Singh A R. Deep learning models and traditional automated techniques for brain tumor segmentation in MRI: a review. Artif Intell Rev, 2023, 56(4): 2923-2969.
6.	Padmapriya T, Sriramakrishnan P, Kalaiselvi T, et al. Advancements of MRI-based brain tumor segmentation from traditional to recent trends: a review. Curr Med Imaging Rev, 2022, 18(12): 1261-1275.
7.	Ranjbarzadeh R, Bagherian Kasgari A, Jafarzadeh Ghoushchi S, et al. Brain tumor segmentation based on deep learning and an attention mechanism using MRI multi-modalities brain images. Sci Rep, 2021, 11(1): 10930.
8.	Zhou R, Wang J, Xia G, et al. Cascade residual multiscale convolution and mamba-structured U-Net for advanced brain tumor image segmentation. Entropy, 2024, 26(5): 385.
9.	Wu X, Yang X, Li Z, et al. Multimodal brain tumor image segmentation based on DenseNet. Plos One, 2024, 19(1): e0286125.
10.	Zhu Z, Wang Z, Qi G, et al. Brain tumor segmentation in MRI with multi-modality spatial information enhancement and boundary shape correction. Pattern Recogn, 2024, 153: 110553.
11.	Kazerooni A F, Khalili N, Liu X, et al. The brain tumor segmentation (BraTS) challenge 2023: focus on pediatrics (CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDS). arXiv, 2024: 2305.17033.
12.	Alam M A, Sohel A, Hasan K M, et al. Advancing brain tumor detection using machine learning and artificial intelligence: a systematic literature review of predictive models and diagnostic accuracy. Strateg Data Manag Innov, 2024, 1(1): 37-55.
13.	Sailunaz K, Alhajj S, ?zyer T, et al. A survey on brain tumor image analysis. Med Biol Eng Comput, 2024, 62(1): 1-45.
14.	Zhao F, Zhang C, Geng B. Deep multimodal data fusion. Acm Comput Surv, 2024, 56(9): 1-36.
15.	Wu X, Hong D, Chanussot J. UIU-Net: U-Net in U-Net for infrared small object detection. IEEE Trans Image Process, 2022, 32: 364-376.
16.	Zhou Z, Siddiquee M M R, Tajbakhsh N, et al. U-Net++: redesigning skip connections to exploit multiscale features in image segmentation. IEEE Trans Med Imaging, 2019, 39(6): 1856-1867.
17.	Liu X, Pang Y, Jin R, et al. Dual-domain reconstruction network with V-Net and K-Net for fast MRI. Magn Reson Med, 2022, 88(6): 2694-2708.
18.	Vossough A, Khalili N, Familiar A M, et al. Training and comparison of nnU-Net and DeepMedic methods for autosegmentation of pediatric brain tumors. Am J Neuroradiol, 2024, 45(8): 1081-1089.
19.	Vahedifard F, Liu X, Kocak M, et al. Artificial intelligence in fetal resting-state functional MRI brain segmentation: a comparative analysis of 3D U-Net, V-Net, and HighRes-Net models. arXiv, 2023: 2311.10844.
20.	Isensee F, Jaeger P F, Kohl S A A, et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods, 2021, 18(2): 203-211.
21.	Ranjbarzadeh R, Caputo A, Tirkolaee E B, et al. Brain tumor segmentation of MRI images: a comprehensive review on the application of artificial intelligence tools. Comput Biol Med, 2023, 152: 106405.
22.	Ramya P, Thanabal M S, Dharmaraja C. Brain tumor segmentation using cluster ensemble and deep super learner for classification of MRI. J Amb Intel Hum Comp, 2021, 12(10): 9939-9952.
23.	Guan X, Zhao Y, Nyatega C O, et al. Brain tumor segmentation network with multi-view ensemble discrimination and kernel-sharing dilated convolution. Brain Sci, 2023, 13(4): 650.
24.	Kamnitsas K, Bai W, Ferrante E, et al. Ensembles of multiple models and architectures for robust brain tumour segmentation. arXiv, 2017: 1711.01468.
25.	Zeineldin R A, Karar M E, Burgert O, et al. Multimodal CNN networks for brain tumor segmentation in MRI: a BraTS 2022 challenge solution// Bakas S, Crimi A, Baid U, et al. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2022. Lecture Notes in Computer Science. Cham: Springer, 2023, 13769: 127-137.
26.	Zeineldin R A, Karar M E, Coburger J, et al. DeepSeg: deep neural network framework for automatic brain tumor segmentation using magnetic resonance FLAIR images. Int J Comput Ass Rad, 2020, 15(6): 909-920.
27.	Gong Q, Chen Y, He X, et al. DeepScan: exploiting deep learning for malicious account detection in location-based social networks. IEEE Commun Mag, 2018, 56(11): 21-27.
28.	Hu J, Gu X, Gu X. Mutual ensemble learning for brain tumor segmentation. Neurocomputing, 2022, 504: 68-81.
29.	Kamnitsas K, Ledig C, Newcombe V F J, et al. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal, 2017, 36: 61-78.
30.	張奕涵, 柏正堯, 尤逸琳, 等. 自適應模態融合雙編碼器MRI腦腫瘤分割網絡. 中國圖象圖形學報, 2024, 29(3): 768-781.
31.	蕭飛鵬, 宋亞男, 徐榮華, 等. 基于多編碼器的腦腫瘤分割算法研究. 無線互聯科技, 2024, 21(8): 71-76.
32.	方新林, 方艷紅, 王迪. 基于多模態特征融合的腦瘤圖像分割方法. 中國醫學物理學雜志, 2022, 39(6): 682-689.
33.	Aboussaleh I, Riffi J, Fazazy K E, et al. Efficient U-Net architecture with multiple encoders and attention mechanism decoders for brain tumor segmentation. Diagnostics, 2023, 13(5): 872.
34.	丁熠, 鄭偉, 耿技, 等. 基于多層級并行神經網絡的多模態腦腫瘤圖像分割框架. 中國圖象圖形學報, 2023, 28(7): 2182-2194.
35.	陳柏年, 韓雨童, 何濤, 等. 基于級聯動態注意力U-Net的腦腫瘤分割方法. 計算機科學, 2023, 50(S2): 1031-1037.
36.	Ullah F, Nadeem M, Abrar M, et al. Brain tumor segmentation from MRI images using handcrafted convolutional neural network. Diagnostics, 2023, 13(16): 2650.
37.	Saleh A H, Atila ü, Menemencio?lu O. Multimodal fusion for enhanced semantic segmentation in brain tumor imaging: integrating deep learning and guided filtering via advanced 3D semantic segmentation architectures. Int J Imag Sys Tech, 2024, 34(5): e23152.
38.	Bouchet P, Deloges J B, Canton-Bacara H, et al. An efficient cascade of U-Net-like convolutional neural networks devoted to brain tumor segmentation// Bakas S, Crimi A, Baid U, et al. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2022. Lecture Notes in Computer Science. Cham: Springer, 2023, 13769: 149-161.
39.	Jia H, Cai W, Huang H, et al. H²NF-Net for brain tumor segmentation using multimodal MR imaging: 2nd place solution to BraTS Challenge 2020 segmentation task// Crimi A, Bakas S. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2020. Lecture Notes in Computer Science. Cham: Springer, 2021, 12659: 58-68.
40.	Baid U, Shah N A, Talbar S. Brain tumor segmentation with cascaded deep convolutional neural network// Crimi A, Bakas S. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2019. Lecture Notes in Computer Science. Cham: Springer, 2020, 11993: 90-98.
41.	夏峰, 邵海見, 鄧星. 融合跨階段深度學習的腦腫瘤MRI圖像分割. 中國圖象圖形學報, 2022, 27(3): 873-884.
42.	Liang J, Yang C, Zhong J, et al. BTSwin-Unet: 3D U-shaped symmetrical Swin transformer-based network for brain tumor segmentation with self-supervised pre-training. Neural Process Lett, 2023, 55(4): 3695-3713.

1. Tataei S N, Ranjbarzadeh R, Jafarzadeh G S, et al. Glioma brain tumor segmentation in four MRI modalities using a convolutional neural network and based on a transfer learning method// Iano Y, Saotome O, Kemper Vásquez G L, et al. Proceedings of the 7th Brazilian Technology Symposium (BTSym’21). BTSym 2021. Smart Innovation, Systems and Technologies. Cham: Springer, 2021, 207: 386-402.
2. Mancuso V, Bruni F, Stramba-Badiale C, et al. How do emotions elicited in virtual reality affect our memory? A systematic review. Comput Hum Behav, 2023, 146: 107812.
3. Balwant M K. A review on convolutional neural networks for brain tumor segmentation: methods, datasets, libraries, and future directions. IRBM, 2022, 43(6): 521-537.
4. Aziz P A, Memon S F, Hussain M, et al. Supratotal resection: An emerging concept of glioblastoma multiforme surgery—systematic review and meta-analysis. World Neurosurg, 2023, 179: e46-e55.
5. Jyothi P, Singh A R. Deep learning models and traditional automated techniques for brain tumor segmentation in MRI: a review. Artif Intell Rev, 2023, 56(4): 2923-2969.
6. Padmapriya T, Sriramakrishnan P, Kalaiselvi T, et al. Advancements of MRI-based brain tumor segmentation from traditional to recent trends: a review. Curr Med Imaging Rev, 2022, 18(12): 1261-1275.
7. Ranjbarzadeh R, Bagherian Kasgari A, Jafarzadeh Ghoushchi S, et al. Brain tumor segmentation based on deep learning and an attention mechanism using MRI multi-modalities brain images. Sci Rep, 2021, 11(1): 10930.
8. Zhou R, Wang J, Xia G, et al. Cascade residual multiscale convolution and mamba-structured U-Net for advanced brain tumor image segmentation. Entropy, 2024, 26(5): 385.
9. Wu X, Yang X, Li Z, et al. Multimodal brain tumor image segmentation based on DenseNet. Plos One, 2024, 19(1): e0286125.
10. Zhu Z, Wang Z, Qi G, et al. Brain tumor segmentation in MRI with multi-modality spatial information enhancement and boundary shape correction. Pattern Recogn, 2024, 153: 110553.
11. Kazerooni A F, Khalili N, Liu X, et al. The brain tumor segmentation (BraTS) challenge 2023: focus on pediatrics (CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDS). arXiv, 2024: 2305.17033.
12. Alam M A, Sohel A, Hasan K M, et al. Advancing brain tumor detection using machine learning and artificial intelligence: a systematic literature review of predictive models and diagnostic accuracy. Strateg Data Manag Innov, 2024, 1(1): 37-55.
13. Sailunaz K, Alhajj S, ?zyer T, et al. A survey on brain tumor image analysis. Med Biol Eng Comput, 2024, 62(1): 1-45.
14. Zhao F, Zhang C, Geng B. Deep multimodal data fusion. Acm Comput Surv, 2024, 56(9): 1-36.
15. Wu X, Hong D, Chanussot J. UIU-Net: U-Net in U-Net for infrared small object detection. IEEE Trans Image Process, 2022, 32: 364-376.
16. Zhou Z, Siddiquee M M R, Tajbakhsh N, et al. U-Net++: redesigning skip connections to exploit multiscale features in image segmentation. IEEE Trans Med Imaging, 2019, 39(6): 1856-1867.
17. Liu X, Pang Y, Jin R, et al. Dual-domain reconstruction network with V-Net and K-Net for fast MRI. Magn Reson Med, 2022, 88(6): 2694-2708.
18. Vossough A, Khalili N, Familiar A M, et al. Training and comparison of nnU-Net and DeepMedic methods for autosegmentation of pediatric brain tumors. Am J Neuroradiol, 2024, 45(8): 1081-1089.
19. Vahedifard F, Liu X, Kocak M, et al. Artificial intelligence in fetal resting-state functional MRI brain segmentation: a comparative analysis of 3D U-Net, V-Net, and HighRes-Net models. arXiv, 2023: 2311.10844.
20. Isensee F, Jaeger P F, Kohl S A A, et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nat Methods, 2021, 18(2): 203-211.
21. Ranjbarzadeh R, Caputo A, Tirkolaee E B, et al. Brain tumor segmentation of MRI images: a comprehensive review on the application of artificial intelligence tools. Comput Biol Med, 2023, 152: 106405.
22. Ramya P, Thanabal M S, Dharmaraja C. Brain tumor segmentation using cluster ensemble and deep super learner for classification of MRI. J Amb Intel Hum Comp, 2021, 12(10): 9939-9952.
23. Guan X, Zhao Y, Nyatega C O, et al. Brain tumor segmentation network with multi-view ensemble discrimination and kernel-sharing dilated convolution. Brain Sci, 2023, 13(4): 650.
24. Kamnitsas K, Bai W, Ferrante E, et al. Ensembles of multiple models and architectures for robust brain tumour segmentation. arXiv, 2017: 1711.01468.
25. Zeineldin R A, Karar M E, Burgert O, et al. Multimodal CNN networks for brain tumor segmentation in MRI: a BraTS 2022 challenge solution// Bakas S, Crimi A, Baid U, et al. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2022. Lecture Notes in Computer Science. Cham: Springer, 2023, 13769: 127-137.
26. Zeineldin R A, Karar M E, Coburger J, et al. DeepSeg: deep neural network framework for automatic brain tumor segmentation using magnetic resonance FLAIR images. Int J Comput Ass Rad, 2020, 15(6): 909-920.
27. Gong Q, Chen Y, He X, et al. DeepScan: exploiting deep learning for malicious account detection in location-based social networks. IEEE Commun Mag, 2018, 56(11): 21-27.
28. Hu J, Gu X, Gu X. Mutual ensemble learning for brain tumor segmentation. Neurocomputing, 2022, 504: 68-81.
29. Kamnitsas K, Ledig C, Newcombe V F J, et al. Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Med Image Anal, 2017, 36: 61-78.
30. 張奕涵, 柏正堯, 尤逸琳, 等. 自適應模態融合雙編碼器MRI腦腫瘤分割網絡. 中國圖象圖形學報, 2024, 29(3): 768-781.
31. 蕭飛鵬, 宋亞男, 徐榮華, 等. 基于多編碼器的腦腫瘤分割算法研究. 無線互聯科技, 2024, 21(8): 71-76.
32. 方新林, 方艷紅, 王迪. 基于多模態特征融合的腦瘤圖像分割方法. 中國醫學物理學雜志, 2022, 39(6): 682-689.
33. Aboussaleh I, Riffi J, Fazazy K E, et al. Efficient U-Net architecture with multiple encoders and attention mechanism decoders for brain tumor segmentation. Diagnostics, 2023, 13(5): 872.
34. 丁熠, 鄭偉, 耿技, 等. 基于多層級并行神經網絡的多模態腦腫瘤圖像分割框架. 中國圖象圖形學報, 2023, 28(7): 2182-2194.
35. 陳柏年, 韓雨童, 何濤, 等. 基于級聯動態注意力U-Net的腦腫瘤分割方法. 計算機科學, 2023, 50(S2): 1031-1037.
36. Ullah F, Nadeem M, Abrar M, et al. Brain tumor segmentation from MRI images using handcrafted convolutional neural network. Diagnostics, 2023, 13(16): 2650.
37. Saleh A H, Atila ü, Menemencio?lu O. Multimodal fusion for enhanced semantic segmentation in brain tumor imaging: integrating deep learning and guided filtering via advanced 3D semantic segmentation architectures. Int J Imag Sys Tech, 2024, 34(5): e23152.
38. Bouchet P, Deloges J B, Canton-Bacara H, et al. An efficient cascade of U-Net-like convolutional neural networks devoted to brain tumor segmentation// Bakas S, Crimi A, Baid U, et al. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2022. Lecture Notes in Computer Science. Cham: Springer, 2023, 13769: 149-161.
39. Jia H, Cai W, Huang H, et al. H²NF-Net for brain tumor segmentation using multimodal MR imaging: 2nd place solution to BraTS Challenge 2020 segmentation task// Crimi A, Bakas S. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2020. Lecture Notes in Computer Science. Cham: Springer, 2021, 12659: 58-68.
40. Baid U, Shah N A, Talbar S. Brain tumor segmentation with cascaded deep convolutional neural network// Crimi A, Bakas S. Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2019. Lecture Notes in Computer Science. Cham: Springer, 2020, 11993: 90-98.
41. 夏峰, 邵海見, 鄧星. 融合跨階段深度學習的腦腫瘤MRI圖像分割. 中國圖象圖形學報, 2022, 27(3): 873-884.
42. Liang J, Yang C, Zhong J, et al. BTSwin-Unet: 3D U-shaped symmetrical Swin transformer-based network for brain tumor segmentation with self-supervised pre-training. Neural Process Lett, 2023, 55(4): 3695-3713.

Previous Article
Research progress and technical analysis of dining robots
Next Article
Progress and application of machine learning in sports injury research

Journal of Biomedical Engineering

Research progress of multimodal magnetic resonance imaging brain tumor segmentation based on fused neural network model

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content