Deep Learning Approach for Cardiac MRI Images
Subject Areas : Image Processing
Afshin
Sandooghdar
1
(ACECR)
Farzin
Yaghmaee
2
(ُSemnan University -Electrical & Computer Engineering Department)
Keywords: Deep Learning, Neural Networks, Magnetic Resonance Imaging (MRI), Disease Prediction.,
Abstract :
Deep Learning (DL) is the most widely used image-analysis process, especially in medical image processing. Though DL has entered image processing to solve Machine Learning (ML) problems, identifying the most suitable model based on evaluation of the epochs is still an open question for scholars in the field. There are so many types of function approximators like Decision Tree, Gaussian Processes and Deep Learning, used in multi-layered Neural Networks (NNs), which should be evaluated to determine their effectiveness. Therefore, this study aimed to assess an approach based on DL techniques for modern medical imaging methods according to Magnetic Resonance Imaging (MRI) segmentation. To do so, an experiment with a random sampling approach was conducted. One hundred patient cases were used in this study for training, validation, and testing. The method used in this study was based on full automatic processing of segmentation and disease classification based on MRI images. U-Net structure was used for the segmentation process, with the use of cardiac Right Ventricular Cavity (RVC), Left Ventricular Cavity (LVC), Left Ventricular Myocardium (LVM), and information extracted from the segmentation step. With train and using random forest classifier, and Multilayer Perceptron (MLP), the task of predicting the pathologic target class was conducted. Segmentation extracted information was in the form of comprehensive features handcrafted to reflect demonstrative clinical strategies. Our study suggests 92% test accuracy for cardiac MRI image segmentation and classification. As for the MLP ensemble, and for the random forest, test accuracy was equal to 91% and 90%, respectively. This study has implications for scholars in the field of medical image processing.
[1]“Automated Cardiac Diagnosis Challenge.” https://www.who.int/news-room/fact-sheets/detail/cardiovascular-diseases-(cvds).
[2]C. Chen et al., “Deep learning for cardiac image segmentation: A review,” arXiv, 2019, doi: 10.3389/fcvm.2020.00025.
[3]M. H. Hesamian, W. Jia, X. He, and P. Kennedy, “Deep Learning Techniques for Medical Image Segmentation: Achievements and Challenges,” J. Digit. Imaging, vol. 32, no. 4, pp. 582–596, 2019, doi: 10.1007/s10278-019-00227-x.
[4] M. Samieiyeganeh, R. W. B. O. K. Rahmat, F. B. Khalid, and K. A. Bin Kasmiran, “An overview of deep learning techniques in echocardiography image segmentation,” Journal of Theoretical and Applied Information Technology, vol. 98, no. 22. pp. 3561–3572, 2020.
[5] V. François-Lavet, P. Henderson, R. Islam, M. G. Bellemare, and J. Pineau, “An introduction to deep reinforcement learning,” Foundations and Trends in Machine Learning, vol. 11, no. 3–4. pp. 219–354, 2018, doi: 10.1561/2200000071.
[6] O. Bernard et al., “Deep Learning Techniques for Automatic MRI Cardiac Multi-Structures Segmentation and Diagnosis: Is the Problem Solved?,” IEEE Trans. Med. Imaging, vol. 37, no. 11, pp. 2514–2525, 2018, doi: 10.1109/TMI.2018.2837502.
[7] A. S. Lundervold and A. Lundervold, “An overview of deep learning in medical imaging focusing on MRI,” arXiv. 2018.
[8] C. A. Miller et al., “Quantification of left ventricular indices from SSFP cine imaging: Impact of real-world variability in analysis methodology and utility of geometric modeling,” J. Magn. Reson. Imaging, vol. 37, no. 5, pp. 1213–1222, 2013, doi: 10.1002/jmri.23892.
[9] M. Khened, V. A. Kollerathu, and G. Krishnamurthi, “Fully convolutional multi-scale residual DenseNets for cardiac segmentation and automated cardiac diagnosis using ensemble of classifiers,” Med. Image Anal., vol. 51, pp. 21–45, 2019, doi: 10.1016/j.media.2018.10.004.
[10] Q. Zheng, H. Delingette, N. Duchateau, and N. Ayache, “3-D Consistent and Robust Segmentation of Cardiac Images by Deep Learning With Spatial Propagation,” IEEE Trans. Med. Imaging, vol. 37, no. 9, pp. 2137–2148, 2018, doi: 10.1109/TMI.2018.2820742.
[11] A. Suinesiaputra et al., “from Cardiac MRI: A Collation Study.” pp. 88–97, 2012.
[12] C. Petitjean et al., “Right ventricle segmentation from cardiac MRI: A collation study,” Medical Image Analysis, vol. 19, no. 1. pp. 187–202, 2015, doi: 10.1016/j.media.2014.10.004.
[13] X. Zhuang et al., “Cardiac Segmentation on Late Gadolinium Enhancement MRI: A Benchmark Study from Multi-Sequence Cardiac MR Segmentation Challenge,” no. Mi, 2020, [Online]. Available: http://arxiv.org/abs/2006.12434.
[14] C. Petitjean and J. N. Dacher, “A review of segmentation methods in short axis cardiac MR images,” Medical Image Analysis, vol. 15, no. 2. pp. 169–184, 2011, doi: 10.1016/j.media. 2010.12.004.
[15] L. M. C. Leon, K. C. Ciesielski, and P. A. V. Miranda, “Efficient Hierarchical Multi-Object Segmentation in Layered Graphs,” Math. Morphol. - Theory Appl., vol. 5, no. 1, pp. 21–42, 2021, doi: 10.1515/mathm-2020-0108.
[16] H. Wei, W. Xue, and D. Ni, “Left ventricle segmentation and quantification with attention-enhanced segmentation and shape correction,” ACM Int. Conf. Proceeding Ser., pp. 226–230, 2019, doi: 10.1145/3364836.3364881.
[17] I. Symposium and B. Imaging, “SEGMENTING THE LEFT VENTRICLE IN CARDIAC IN CARDIAC MRI : FROM HANDCRAFTED TO DEEP REGION BASED DESCRIPTORS Daniela O . Medley , Carlos Santiago , and Jacinto C . Nascimento Institute for Systems and Robotics , Instituto Superior T ´ ecnico , Lisbon , Por,” no. Isbi, pp. 644–648, 2019.
[18] W. Wang, Y. Wang, Y. Wu, T. Lin, S. Li, and B. Chen, “Quantification of Full Left Ventricular Metrics via Deep Regression Learning with Contour-Guidance,” IEEE Access, vol. 7, pp. 47918–47928, 2019, doi: 10.1109/ACCESS.2019.2907564.
[19] W. Bai, W. Shi, C. Ledig, and D. Rueckert, “Multi-atlas segmentation with augmented features for cardiac MR images,” Medical Image Analysis, vol. 19, no. 1. pp. 98–109, 2015, doi: 10.1016/j.media.2014.09.005.
[20] S. Moradi et al., “MFP-Unet: A novel deep learning based approach for left ventricle segmentation in echocardiography,” Phys. Medica, vol. 67, no. 110, pp. 58–69, 2019, doi: 10.1016/j.ejmp.2019.10.001.
[21] E. Shelhamer, J. Long, and T. Darrell, “IEEE Transactions on Pattern Analysis and Machine Intelligence Fully Convolutional Networks for Semantic Segmentation.” [Online]. Available: http://www.ieee.org/publications_standards/publications/rights/index.html.
[22] R. P. K. Poudel, P. Lamata, and G. Montana, “Recurrent fully convolutional neural networks for multi-slice MRI cardiac segmentation,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10129 LNCS. pp. 83–94, 2017, doi: 10.1007/978-3-319-52280-7_8.
[23] Y. Jang, Y. Hong, S. Ha, S. Kim, and H. J. Chang, “Automatic segmentation of LV and RV in cardiac MRI,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 10663 LNCS, pp. 161–169, 2018, doi: 10.1007/978-3-319-75541-0_17.
[24] J. Duan et al., “Automatic 3D Bi-Ventricular Segmentation of Cardiac Images by a Shape-Refined Multi- Task Deep Learning Approach,” IEEE Trans. Med. Imaging, vol. 38, no. 9, pp. 2151–2164, 2019, doi: 10.1109/TMI.2019.2894322.
[25] A. S. Fahmy, H. El-Rewaidy, M. Nezafat, S. Nakamori, and R. Nezafat, “Automated Analysis of Myocardial Native T1 Mapping Images Using Fully Convolutional Neural Networks,” Journal of Cardiovascular Magnetic Resonance, vol. 21, no. 7. pp. 1–12, 2019.
[26] M. Kendirci, S. Nowfar, and W. J. G. Hellstrom, “The impact of vascular risk factors on erectile function,” Timely topics in medicine. Cardiovascular diseases, vol. 9. p. E11, 2005.
[27] E. Ermis et al., “The relationship between erectile dysfunction and the Atherogenic Index of Plasma,” Int. J. Impot. Res., vol. 32, no. 4, pp. 462–468, 2020, doi: 10.1038/s41443-019-0167-2.
[28] C. Zotti, Z. Luo, A. Lalande, and P. M. Jodoin, “Convolutional Neural Network with Shape Prior Applied to Cardiac MRI Segmentation,” IEEE J. Biomed. Heal. Informatics, vol. 23, no. 3, pp. 1119–1128, May 2019, doi: 10.1109/JBHI.2018.2865450.
[29] J. Patravali, S. Jain, and S. Chilamkurthy, “2D-3D fully convolutional neural networks for cardiac MR segmentation,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10663 LNCS. pp. 130–139, 2018, doi: 10.1007/978-3-319-75541-0_14.
[30] S. Jahangard and M. Shahedi, “U-Net Based Architecture for an Improved Multiresolution Segmentation in Medical Images,” pp. 1–22, 2020.
[31] F. B. M. Suah, “Preparation and characterization of a novel Co(II) optode based on polymer inclusion membrane,” Analytical Chemistry Research, vol. 12. pp. 40–46, 2017, doi: 10.1016/j.ancr.2017.02.001.
[32] P. F. Christ et al., “Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9901 LNCS. pp. 415–423, 2016, doi: 10.1007/978-3-319-46723-8_48.
http://jist.acecr.org ISSN 2322-1437 / EISSN:2345-2773 |
Journal of Information Systems and Telecommunication
|
Deep Learning Approach for Cardiac MRI Images |
Afshin Sandooghdar1, Farzin Yaghmaee 1*
|
1. Electrical and Computer Engineering Department, Semnan University, Semnan, Iran
|
Received: 22 May 2021 / Revised: 08 Aug 2021/ Accepted: 02 Sep 2021 |
Abstract
Deep Learning (DL) is the most widely used image-analysis process, especially in medical image processing. Though DL has entered image processing to solve Machine Learning (ML) problems, identifying the most suitable model based on evaluation of the epochs is still an open question for scholars in the field. There are so many types of function approximators like Decision Tree, Gaussian Processes and Deep Learning, used in multi-layered Neural Networks (NNs), which should be evaluated to determine their effectiveness. Therefore, this study aimed to assess an approach based on DL techniques for modern medical imaging methods according to Magnetic Resonance Imaging (MRI) segmentation. To do so, an experiment with a random sampling approach was conducted. One hundred patient cases were used in this study for training, validation, and testing. The method used in this study was based on full automatic processing of segmentation and disease classification based on MRI images. U-Net structure was used for the segmentation process, with the use of cardiac Right Ventricular Cavity (RVC), Left Ventricular Cavity (LVC), Left Ventricular Myocardium (LVM), and information extracted from the segmentation step. With train and using random forest classifier, and Multilayer Perceptron (MLP), the task of predicting the pathologic target class was conducted. Segmentation extracted information was in the form of comprehensive features handcrafted to reflect demonstrative clinical strategies. Our study suggests 92% test accuracy for cardiac MRI image segmentation and classification. As for the MLP ensemble, and for the random forest, test accuracy was equal to 91% and 90%, respectively. This study has implications for scholars in the field of medical image processing.
Keywords: Deep Learning; Neural Networks; Magnetic Resonance Imaging (MRI); Disease Prediction.
1- Introduction
According to a World Health Organization (WHO) report in 2016, approximately 17.9 people have deceased; due to cardiovascular disease. [1] To avoid the growing number of deaths caused by cardiovascular disease, diagnosis of the disease has attracted the attention of physicians. At present, physicians diagnose cardiovascular disease based on the patient's medical history and physical examinations. Their experience is also involved in interpreting and treating the disease.
The absence of computer-based processors results in biased results. In addition, there are higher chances of making a mistake in interpretation when the outcomes are non-producible. Needless to mention that such diagnosis is more expensive.
Thus, there is a need to perform an automated process on medical images that includes diagnosis and treatment with high accuracy. In some cases, there are no physicians available for doing the process of diagnosis and treatment.
Factors such as labor time, subjective inclination, and intended reproducibility have hindered comprehensive quantitative estimations. This situation perpetuates the significance of Machine Learning (ML) methods with a combination of Deep Learning as a practical solution to implement automating and segmentation on medical images.
Artificial Intelligence (Al) is the basis of Machine learning (ML), through which computers can learn a task using data rather than being specifically programmed.
One of the main categories in ML is deep Learning (DL). DL has neural networks designed to employ data to capture hierarchical levels of abstraction. They are made up of several stacked layers.
Convolutional Neural Networks (CNNs) have provided excellent results for segmenting medical images. [2] Convolutional Neural Network (CNN) is a subcategory of neural networks that includes a stack of layers like convolution layers, and pooling layers. Each layer does a specific action. [3] The input layer is the beginning layer that is connected to an input image directly. The number of pixels in the image is equal to the number of neurons in this layer. The next layer is the convolutional layer, which is helpful for sequential data and images. Parameters in the layers include a set of kernels (filters) that designers define, and they have arbitrary sizes. [4]
First layers in image classification, learn how to detect patterns, edges, and textures. Subsequent layers can detect the objects, either entirely or partially. [5] There are several types of deep neural networks for use in medical imaging. For example, the gold standard in cardiac function research is cardiac magnetic resonance imaging (CMR), which assesses the left and right ventricular ejection fractions (EF) and stroke volumes (SV), as well as the mass of the left ventricle and the thickness of the myocardium. [6]
Cardiac MRI is an imaging technique based on a non-invasive structure that can visualize the heart anatomy.it works based on radio-frequency waves and generates images of the heart. [7]
Appropriate results are not gained in clinical practices, which are performed semi-automatically with inaccurate segmentation. In addition, it will be very time-consuming. The accuracy and speed in the semi-automatic method are lower compared to the automatic method. [8][9]
MRI images have several problems that are fully expressed in [10]. Here, we briefly mention a few of them
1. Intensity and form variability of cardiac functions in patients with various pathologies.
2. Trabeculae and papillary muscles with intensities equivalent to the myocardium are present.
3. There is a lack of contrast between the myocardium and its surroundings.
4. Blood flow induces brightness heterogeneities in the left and right ventricular cavities.
In the years before 2012, various works have been done on segmenting MRI images that have used methods other than deep learning. In these articles, various datasets have been used, which only included the ground truth for the ED and ES ventricular volume. [11][12][13]
This paper has used a data set called automatic Cardiac Diagnosis Challenge (ACDC), which is a public data set. Compared to previous datasets, ACDC has a more extensive scope on cardiac. It has manual expert segmentation of the Left and Right Ventricle (LV/RV).
The main objective of this paper is to evaluate convolutional neural networks (CNNs) based on U-Net structure on the ACDC data set for segmentation and classification of cardiac images.
Using the method presented in this paper, Computer-Aided Diagnosis (CAD) is possible for patients with myocardial infarction.
In the next section, we evaluate MRI segmentation methods and mention related works based on deep learning techniques. In the following, we will explain our method. In the end, the results of research and implementation based on U-Net are presented. In this section, the idea that we are currently working on is also presented.
In the rest of this paper, we discuss related work in section 2, evaluation method in section 3, and results in section 4. Finally, we discuss conclusions and the future works with using deep learning methods in medical image processing.
2- Related Works
Certainly, before developing methods based on deep learning, the usual methods in image processing were used to segment images, in the following, some of which are mentioned.
In 2011, Petitjean et al. presented a comprehensive analysis of segmentation methods for defining the LV/RV in short-axis cardiac MR images. [14]
In this paper, the authors have published the results of about 70 articles in this field. The presented methods can be divided into two parts, which are weak and strong prior methods. Weak prior includes weak assumptions like anatomical information or spatial intensity.
The first category includes techniques based on image, like, threshold, and dynamic programming. [15], classification methods based on pixels, like clustering and Gaussian mixture model fitting. [16], active contour and level set [17] and finally approaches based on the graph. [18]
The second category includes methods based on strong prior, which includes atlas-based methods [19] and active shape and appearance methods. [20] All the mentioned cases need training data set with manual annotations. In the following, we review the cases that have used deep learning techniques.
Most of the works in this section are based on segmenting cardiac MRI (e.g., Left and Right Ventricle and Left Atrium). This is indeed a limitation caused by limited data sets.
The first one that used a Fully Convolutional Network (FCN) [21] for segmentation of LV and RV on MRI images was Tran [22]. Tran and his colleagues received very good results based on FCN and performed much better than the old methods in the case of speed and accuracy. In recent years, many researchers have worked on the FCN network. Most of their focus has been on improving the network to increase segmentation capacity and greater accuracy. As example, we can mention [23], [24] and [25]. Tran studied weighted cross-entropy, weighted Dice loss, deep supervision loss, and focal loss to enhance segmentation accuracy. As a result of the low resolution and motion of cardiac MRI scanners, these loss functions work based on 2D networks rather than 3D networks for segmentation. [26][27]
Employing 2D networks for cardiac segmentation has shortcomings. Not only do they operate slice by slice, but also, they ignore inter-slice dependencies. As a result, locating and segmenting the heart on difficult slices like apical and basal slices, where the ventricle contours aren't well defined, is challenging. Various works have attempted to resolve this problem by adding additional contextual details to direct 2D FCN. [28]
Other researchers work based on extract spatial information from adjacent slices for doing better segmentation using Recurrent Neural Networks (RNNs) [22], [29]. Based on our research, very few papers have been done on atrial segmentation. Table 1 shows the summary of the mentioned papers prepared by Chen and his colleagues. [2]
Table 1: Deep learning methods on cardiac MRI segmentation. S.A.: Short-Axis view prepared by Chen and colleagues
Selected works FCN- based | Description | Types of Images | Structure |
Tarn(2016) | 2D FCN | S.A. | Bi-Ventricle |
Jang et al. (2017) | 2D M-Net with weighted cross entropy | S.A. | Bi-Ventricle |
Khened (2019) | 2D U-net | S.A. | Bi-Ventricle |
Fahmy (2019) | 2D FCN | S.A. | LV |
Poudel (2016) | 2D FCN + RNN | S.A. | Bi-Ventricle |
Patravali (2017) | 2D Multi Channel FCN | S.A. | Bi-Ventricle |
3- Methodology
U-Net is a well-known structure in segmenting medical images. This network was provided by Ronneberger et al. [30] based on the deconvolution concept presented by [31]. FCN inspires the U-Net model. Furthermore, U-Net has a depth of 19 layers. The superior r design of skip connections between different stages of the network is one of the benefits of the U-Net. [32] It is vital to overcome the trade-off, as it is a negative concept in U-Net. Trade-off reduces localization caused by the lack of pooling layers. The pooling layers are required for large-sized patches.
In order to overcome the trade-off between localization and the use of context, U-Net is modified. [2] The main modification is the shortcut connections between the layers with equal resolution. The high-resolution of the deconvolution layers is the result of these connections.
U-Net is designed for semantic segmentation. The role of semantic segmentation is to group sections of an image that belong to the same object class together. U-Net has contracted and expansive paths. The main feature of the contracting path is the convolutional network. This network has repeated application of two 3x3 convolutions, which are followed by a Rectified Linear Unit (ReLU) and a 2x2 max pooling operation with stride 2 for down-sampling.
In order to decrease the storage and difficulties of transmitting images, the spatial resolution should be reduced. At the same time, the same 2D representation is kept. This process is known as down sampling. In addition, the number of feature channels should be doubled at each down sampling step.
The expansive path, on the other hand, entails an up-sampling of the feature map. It has a 2x2 convolution ("up-convolution") designed to cut the number of feature channels into halves.
It is important to up-sample the resolution of the feature map. The main approaches used for this purpose are down-sampling the resolution by summarizing a local area with a single value (average or max pooling), "un-pooling" operations, up-sample the resolution by distributing a single value into a higher resolution, and pooling operations.
In order to avoid the loss of border pixels, cropping should be implemented in convolutions. Thus, a 1x1 convolution is used to map each 64-component feature vector to the desired number of classes at the final convolution. The U-Net has a total of 23 convolutional layers which for a network with a decoder and encoder structure. This structure is depicted in Figure 1.
Fig. 1 The structure of the U-Net [29].
In this paper, we evaluated the U-Net structure with the ACDC Data set. Our goal is to measure the accuracy of segmentation and classification of MRI images with this network because one advantage of using this network is that it can provide acceptable results with a small amount of data. Our data set include MRI images for 150 patients.
The ACDC using two types of scanners with different magnetic strengths, one 1.5T and the second 3.0T. Each time series has between 28 and 40 3D volumes that cover the cardiac cycle. These volumes are segmented manually by clinical experts on end-diastolic (ED) and end-systolic (ES) phase instants done on LVC, LVM, and RVC. ACDC dataset includes five groups. Except for one of the healthy patients, four groups refer to pathological patients. These healthy patients are those who have had a myocardial infarction (MINF), dilated cardiomyopathy (DCM), hypertrophic cardiomyopathy (HCM), abnormal right ventricle (ARV), and healthy or normal subjects (NOR). The height and weight of patients give more information about this status. The dataset is part into 100 training and 50 test patients. Division and classification ground truth is given as it were for the 100 training cases.
In this paper, domain-specific features extracted in each time step of MRI based on multi-structure segmentation, which are stimulated by the workflow of a cardiologist, to then train an ensemble of classifiers for disease prediction. Figure 2.
Fig. 2 Outline of the using method. Model-based on 2D and 3D from segmentation process averaged and then after extracting dynamic volume and instant features, used for disease prediction based on the ensemble of a classifier.
The methods used in this article are segmenting the images and classification based on the properties extracted from segmentation. These processes are explained below. Figure 3.
Fig. 3 Base Flowchart
3-1- Segmentation
We used the U-net network to segment the images [29]. In this way, we have made changes in the network, and it can be used for 2D and 3D images.
The 3D segmentation demonstrates a context aggregation pathway through a localization pathway. These two ways at various scales are interconnected. This is because we can combine context features with the corresponding local information Figure 4.
Operations like upscaling and pooling are performed in the x-y plane because of the low resolution of input (z- resolution). The setting within the z-dimension is exclusively totaled through the 3D convolutions.
Fig. 4 Architecture of segmentation network based on 3 Dimensional, for the 2D size of patch is 352 × 352 and uses 48 initial features and 2Dimentional convolutions. Gray blocks are features that extracted and each of them includes ReLU, by batch normalization
It is mentioned that, Because of network layers that include 18 layers, there are no residual connections.
Upscaling (pooling) operations are responsible for halved 26 feature maps. In fact, there are four upscaling operations, and each of them doubled the initial number of twenty-six feature maps, so, resulting in a maximum of 416 feature maps at the bottom of the U-net structure.
Our training 3D U-Net is based on 300 epochs in 5-fold cross-validation that include a pixel-wise categorical cross-entropy loss. The 2D network is the same as the 3D network with 2D convolutions. Obviously, a 2D network requires less memory, so we use 48 initial feature maps.
For the training network, we use, batch size of 10 with input patches with a size of 352*352 and a multiclass dice loss, Eq. (1)
(1)
shows the SoftMax output of the network, encoding of Ground Truth (GT) segmentation maps denotes with v. Both u and v are of size i × k with i being the number of pixels in the training patch and k ∈ K being the classes.
3-2- Classification
After the segmentation process, we can obtain two sets of features. These features are used for the classification of disease. Table 2 shows features of these two sets.
Table 2: Both Ed and ES used for extracted instant features.
Instant Features | LVM | LVC | RVC |
Maximum Thickness | ● | - | - |
Minimum Thickness | ● | - | - |
Std. Thickness | ● | - | - |
Mean Thickness | ● | - | - |
Mass | ● | - | - |
Maximum Circumference | ● | - | ● |
Mean Circumference | ● | - | ● |
Mean Circularity | ● | - | ● |
Dynamic Volume Features
Dynamic Volume Features | LVM , LVC , RVC |
Maximum Volume | Covers All Features (●) |
Minimum Volume | |
Dynamic Ejection Fraction | |
Volume Median | |
Volume Standard Deviation | |
Volume Skewness | |
Volume Kurtosis |
Instant Features include information of local and global shape, which global information include circumference, circularity, the thickness of LVM, and so on, and local information including, size of RVC at the apex, LVM thickness between RVC and LVC, etc.
For Dynamic Volume Features, the segmentation process is done to specify anatomical structures in all time steps in the MRI.
For the classification step, an ensemble of 50 MLP trained based on features of Table 2.
Pathology classification with a random forest was also used.
The number of hidden layers in the Structure of Multilayer perceptron is 4. 32 units are also used for training in each layer and batch normalization, ReLU with a Gaussian noise layer (σ = 0.1) used in training. We divided our data set into two parts, 75 percent for training selected randomly and 25 percent of the dataset for epoch selection.
Multilayer perceptron trained based on 400 epochs. We trained a random forest is trained with Thousand trees. Throughout Multilayer Perceptron score, can be obtained During testing, with the SoftMax outputs of all MLPs were averaged, which was recombined in this way with the arbitrary random forest yield to get the final ensemble prediction.
3-3- Experimental Analysis
The impact of the proposed network architecture, loss function, data-augmentation scheme, and the influence of Region of Interest (ROI) cropping and post-processing was measured in this study through an experiment. Similar to many studies that use the ACDC training dataset, the unit of the evaluation was Dice score and Hausdor Distance (HD) in mm. Also, the TensorFlow software was employed to design the neural networks. A desktop computer with the following specifications was used to run the experiment.
GPU: NVIDIA-Titan-X GPU, CPU: Intel Core i7-4930K 12-core CPUs @ 3.40GHz, RAM: 64GB
3-4- ACDC Dataset
The dataset used in this study entailed 100 cases of patients (1:8k 2D images). This data set was divided into 70 cases for training, 15 cases for validation, and 15 cases for testing. A random sampling method, i.e., stratified sampling, was employed. Stratified sampling can be employed when the total number of cases is known and can ensure that each stratum entails an equal number of cases from different cardiac disease groups. Each case had approximately 20 2D images with ground-truth annotations for LV, RV, and MYO at the ED and ES phases.
3-5- Training
The researcher used a training patch with 16 ROI cropped 2D images. The images had 128x128 dimensions. This network was trained for 200 epochs.
In order to select a suitable model, the models were evaluated after each epoch. Focus was accorded to the highest Dice score for MYO class on the validation set. Figure 5 is an example of the intermediate feature maps of the trained network in the ACDC-2017 dataset.
Fig. 5 The figures compare illustrate the feature maps of the trained model. (A) Input image fed to the network, (B) posterior probability maps after soft-max output, (C) The final prediction of labels, (i) - (iii) visualization of the initial layers kernels- 3x3, 5x5 and 7x7, (iv) - (vi) Filter response to the input image (A).
4- Results
The results of the five-fold Cross Validation (CV) are shown in Table 3. The results include Hausdorff distances. Individual dice scores are also mentioned in Table 3.
Table 3: Segmentation process with Dice scores and Hausdorff distances for five-fold CV
| Case | Dice | Hausdorff (mm) | ||||
|
| RCV | LMV | LVC | RVC | LVM | LVC |
DCM | ED ES Result | 0.940 | 0.903 | 0.967 | 20.86 | 8.20 | 7.115 |
0.869 | 0.912 | 0.914 | 17.941 | 8.160 | 5.886 | ||
0.906 | 0.909 | 0.943 | 19.828 | 8.154 | 6.543 | ||
HCM | ED | 0.937 | 0.902 | 0.967 | 12.720 | 8.706 | 7.255 |
EC | 0.876 | 0.905 | 0.934 | 18.325 | 11.353 | 14.514 | |
Result | 0.909 | 0.904 | 0.951 | 15.322 | 10.104 | 11.023 |
MINF | ED | 0.936 | 0.894 | 0.963 | 13.383 | 9.631 | 6.880 |
EC | 0.888 | 0.904 | 0.905 | 18.636 | 11.75 | 9.598 | |
Result | 0.912 | 0.902 | 0.933 | 16.106 | 10.732 | 8.115 |
NOR | ED | 0.938 | 0.886 | 0.972 | 9.764 | 7.231 | 4.624 |
ES | 0.884 | 0.901 | 0.944 | 11.407 | 9.164 | 7.665 | |
Result | 0.911 | 0.898 | 0.955 | 10.612 | 8.398 | 6.332 |
RV | ED | 0.946 | 0.908 | 0.963 | 14.725 | 10.716 | 9.393 |
ES | 0.853 | 0.914 | 0.921 | 15.124 | 11.768 | 11.225 | |
Result | 0.901 | 0.9101 | 0.944 | 15.131 | 11.608 | 10.623 |
Compared to the 2D model, the 3D model provided better results. The results can be seen in Table 4.
Table 4: Results from CV on the training set are shown for the 2D and 3D model
| Dice | Hausdorff (mm) | |||||
RVC | LVM | LVC | RVC | LVM | LVC | ||
CV | 2D | 0.905 | 0.904 | 0.946 | 14.292 | 8.898 | 7.053 |
3D | 0.878 | 0.874 | 0.926 | 16.288 | 10.435 | 9.776 | |
Result | 0.909 | 0.904 | 0.946 | 15.292 | 9.666 | 8.414 | |
TEST | Result | 0.926 | 0.912 | 0.954 | 11.133 | 8.698 | 7.142 |
The dice score is a statistical procedure employed to measure the similarity of two samples. On the contrary, Hausdorff gauges the difference between two subsets of a metric space. For classification, we train classification ensemble with features extracted from the segmentation step, the test accuracy of classification was 92 percent. For the MLP ensemble, 91 percent was achieved and for the random forest was 90%. Table 5 indicated confusion matrices.
Table 5: Error matrices or confusion matrices of the ensemble predictions from CV showed training and test set. the predicted class is based on Rows and the target class shows in columns
NOR | 18 | 0 | 1 | 0 | 1 |
DCM | 0 | 19 | 0 | 1 | 0 |
HCM | 0 | 0 | 19 | 1 | 0 |
MNF | 0 | 1 | 0 | 19 | 0 |
RVA | 1 | 0 | 0 | 0 | 19 |
| NOR | DCM | HCM | MNF | RVA |
NOR | 10 | 0 | 0 | 0 | 0 |
DCM | 0 | 9 | 0 | 1 | 0 |
HCM | 1 | 0 | 9 | 0 | 0 |
MNF | 0 | 2 | 0 | 8 | 0 |
RVA | 0 | 0 | 0 | 0 | 10 |
| NOR | DCM | HCM | MNF | RVA |
5- Conclusion and Future Works:
In this paper, we proposed a fully automatic process for MRI image segmentation and classification .in the first part of our method, we developed segmentation architecture based on U-Net structure and trained the network on ED and ES. The first part can be done on 3D and 2D images.
Features extraction is the output of the segmentation step and is used in the second step (ensemble classifier) to predict the diagnosis. The test accuracy of classification was 92 percent. For the MLP ensemble, 91 percent was achieved and for the random forest was 90%. Table 5 indicated confusion matrices.
Also in this paper, We developed a precise multi-structural segmentation method that is taught only in the case of ED and ES phases. Our approach revolves around the use of both two-dimensional and three-dimensional models and uses the corresponding benefits through combination. The results are strong against incisional inconsistencies, different magnetic resonance imaging protocols of the heart as well as various injuries. We scored 0.954 (LVC), 0.926 (RVC), and 0.912 (LVM) on the ACDC test suite.
The findings of the current study can increase the outcome of computer image processors and enhance science in the arena of computer-aided diagnosis. Physicians need for semi-automated or automated processors can be fulfilled by increasing the production of digital information. The use of computer-based image processors results in quick production of robust and accurate data. This is among the main contributions of this study.
Future work
Machine learning is divided into three parts: supervised Learning (SL), unsupervised learning (UL), and reinforcement learning [2]. Deep Learning (DL), which uses multi-layered neural networks, is used in all three parts of machine learning. The difference between Deep Reinforcement Learning (DRL) and other machine learning methods is using the agent to perform the task in the environment. A very interesting way to segment MRI images and classification tasks is to use Deep Reinforcement Learning methods. The use of additional agents can also be used to classify and segment images.
References
[1] “Automated Cardiac Diagnosis Challenge.” https://www.who.int/news-room/fact-sheets/detail/cardiovascular-diseases-(cvds).
[2] C. Chen et al., “Deep learning for cardiac image segmentation: A review,” arXiv, 2019, doi: 10.3389/fcvm.2020.00025.
[3] M. H. Hesamian, W. Jia, X. He, and P. Kennedy, “Deep Learning Techniques for Medical Image Segmentation: Achievements and Challenges,” J. Digit. Imaging, vol. 32, no. 4, pp. 582–596, 2019, doi: 10.1007/s10278-019-00227-x.
[4] M. Samieiyeganeh, R. W. B. O. K. Rahmat, F. B. Khalid, and K. A. Bin Kasmiran, “An overview of deep learning techniques in echocardiography image segmentation,” Journal of Theoretical and Applied Information Technology, vol. 98, no. 22. pp. 3561–3572, 2020.
[5] V. François-Lavet, P. Henderson, R. Islam, M. G. Bellemare, and J. Pineau, “An introduction to deep reinforcement learning,” Foundations and Trends in Machine Learning, vol. 11, no. 3–4. pp. 219–354, 2018, doi: 10.1561/2200000071.
[6] O. Bernard et al., “Deep Learning Techniques for Automatic MRI Cardiac Multi-Structures Segmentation and Diagnosis: Is the Problem Solved?,” IEEE Trans. Med. Imaging, vol. 37, no. 11, pp. 2514–2525, 2018, doi: 10.1109/TMI.2018.2837502.
[7] A. S. Lundervold and A. Lundervold, “An overview of deep learning in medical imaging focusing on MRI,” arXiv. 2018.
[8] C. A. Miller et al., “Quantification of left ventricular indices from SSFP cine imaging: Impact of real-world variability in analysis methodology and utility of geometric modeling,” J. Magn. Reson. Imaging, vol. 37, no. 5, pp. 1213–1222, 2013, doi: 10.1002/jmri.23892.
[9] M. Khened, V. A. Kollerathu, and G. Krishnamurthi, “Fully convolutional multi-scale residual DenseNets for cardiac segmentation and automated cardiac diagnosis using ensemble of classifiers,” Med. Image Anal., vol. 51, pp. 21–45, 2019, doi: 10.1016/j.media.2018.10.004.
[10] Q. Zheng, H. Delingette, N. Duchateau, and N. Ayache, “3-D Consistent and Robust Segmentation of Cardiac Images by Deep Learning With Spatial Propagation,” IEEE Trans. Med. Imaging, vol. 37, no. 9, pp. 2137–2148, 2018, doi: 10.1109/TMI.2018.2820742.
[11] A. Suinesiaputra et al., “from Cardiac MRI: A Collation Study.” pp. 88–97, 2012.
[12] C. Petitjean et al., “Right ventricle segmentation from cardiac MRI: A collation study,” Medical Image Analysis, vol. 19, no. 1. pp. 187–202, 2015, doi: 10.1016/j.media.2014.10.004.
[13] X. Zhuang et al., “Cardiac Segmentation on Late Gadolinium Enhancement MRI: A Benchmark Study from Multi-Sequence Cardiac MR Segmentation Challenge,” no. Mi, 2020, [Online]. Available: http://arxiv.org/abs/2006.12434.
[14] C. Petitjean and J. N. Dacher, “A review of segmentation methods in short axis cardiac MR images,” Medical Image Analysis, vol. 15, no. 2. pp. 169–184, 2011, doi: 10.1016/j.media. 2010.12.004.
[15] L. M. C. Leon, K. C. Ciesielski, and P. A. V. Miranda, “Efficient Hierarchical Multi-Object Segmentation in Layered Graphs,” Math. Morphol. - Theory Appl., vol. 5, no. 1, pp. 21–42, 2021, doi: 10.1515/mathm-2020-0108.
[16] H. Wei, W. Xue, and D. Ni, “Left ventricle segmentation and quantification with attention-enhanced segmentation and shape correction,” ACM Int. Conf. Proceeding Ser., pp. 226–230, 2019, doi: 10.1145/3364836.3364881.
[17] I. Symposium and B. Imaging, “SEGMENTING THE LEFT VENTRICLE IN CARDIAC IN CARDIAC MRI : FROM HANDCRAFTED TO DEEP REGION BASED DESCRIPTORS Daniela O . Medley , Carlos Santiago , and Jacinto C . Nascimento Institute for Systems and Robotics , Instituto Superior T ´ ecnico , Lisbon , Por,” no. Isbi, pp. 644–648, 2019.
[18] W. Wang, Y. Wang, Y. Wu, T. Lin, S. Li, and B. Chen, “Quantification of Full Left Ventricular Metrics via Deep Regression Learning with Contour-Guidance,” IEEE Access, vol. 7, pp. 47918–47928, 2019, doi: 10.1109/ACCESS.2019.2907564.
[19] W. Bai, W. Shi, C. Ledig, and D. Rueckert, “Multi-atlas segmentation with augmented features for cardiac MR images,” Medical Image Analysis, vol. 19, no. 1. pp. 98–109, 2015, doi: 10.1016/j.media.2014.09.005.
[20] S. Moradi et al., “MFP-Unet: A novel deep learning based approach for left ventricle segmentation in echocardiography,” Phys. Medica, vol. 67, no. 110, pp. 58–69, 2019, doi: 10.1016/j.ejmp.2019.10.001.
[21] E. Shelhamer, J. Long, and T. Darrell, “IEEE Transactions on Pattern Analysis and Machine Intelligence Fully Convolutional Networks for Semantic Segmentation.” [Online]. Available: http://www.ieee.org/publications_standards/publications/rights/index.html.
[22] R. P. K. Poudel, P. Lamata, and G. Montana, “Recurrent fully convolutional neural networks for multi-slice MRI cardiac segmentation,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10129 LNCS. pp. 83–94, 2017, doi: 10.1007/978-3-319-52280-7_8.
[23] Y. Jang, Y. Hong, S. Ha, S. Kim, and H. J. Chang, “Automatic segmentation of LV and RV in cardiac MRI,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 10663 LNCS, pp. 161–169, 2018, doi: 10.1007/978-3-319-75541-0_17.
[24] J. Duan et al., “Automatic 3D Bi-Ventricular Segmentation of Cardiac Images by a Shape-Refined Multi- Task Deep Learning Approach,” IEEE Trans. Med. Imaging, vol. 38, no. 9, pp. 2151–2164, 2019, doi: 10.1109/TMI.2019.2894322.
[25] A. S. Fahmy, H. El-Rewaidy, M. Nezafat, S. Nakamori, and R. Nezafat, “Automated Analysis of Myocardial Native T1 Mapping Images Using Fully Convolutional Neural Networks,” Journal of Cardiovascular Magnetic Resonance, vol. 21, no. 7. pp. 1–12, 2019.
[26] M. Kendirci, S. Nowfar, and W. J. G. Hellstrom, “The impact of vascular risk factors on erectile function,” Timely topics in medicine. Cardiovascular diseases, vol. 9. p. E11, 2005.
[27] E. Ermis et al., “The relationship between erectile dysfunction and the Atherogenic Index of Plasma,” Int. J. Impot. Res., vol. 32, no. 4, pp. 462–468, 2020, doi: 10.1038/s41443-019-0167-2.
[28] C. Zotti, Z. Luo, A. Lalande, and P. M. Jodoin, “Convolutional Neural Network with Shape Prior Applied to Cardiac MRI Segmentation,” IEEE J. Biomed. Heal. Informatics, vol. 23, no. 3, pp. 1119–1128, May 2019, doi: 10.1109/JBHI.2018.2865450.
[29] J. Patravali, S. Jain, and S. Chilamkurthy, “2D-3D fully convolutional neural networks for cardiac MR segmentation,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10663 LNCS. pp. 130–139, 2018, doi: 10.1007/978-3-319-75541-0_14.
[30] S. Jahangard and M. Shahedi, “U-Net Based Architecture for an Improved Multiresolution Segmentation in Medical Images,” pp. 1–22, 2020.
[31] F. B. M. Suah, “Preparation and characterization of a novel Co(II) optode based on polymer inclusion membrane,” Analytical Chemistry Research, vol. 12. pp. 40–46, 2017, doi: 10.1016/j.ancr.2017.02.001.
[32] P. F. Christ et al., “Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9901 LNCS. pp. 415–423, 2016, doi: 10.1007/978-3-319-46723-8_48.
* Corresponding Author