Next Article in Journal
Using Fast Frequency Hopping Technique to Improve Reliability of Underwater Communication System
Next Article in Special Issue
Correlation between LAA Morphological Features and Computational Fluid Dynamics Analysis for Non-Valvular Atrial Fibrillation Patients
Previous Article in Journal
Real-Time Haze Removal Using Normalised Pixel-Wise Dark-Channel Prior and Robust Atmospheric-Light Estimation
Previous Article in Special Issue
Cross-Domain Data Augmentation for Deep-Learning-Based Male Pelvic Organ Segmentation in Cone Beam CT
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Application of Image Fusion in Diagnosis and Treatment of Liver Cancer

1
Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
2
Harvard Medical School, Boston, MA 02115, USA
*
Author to whom correspondence should be addressed.
Submission received: 3 January 2020 / Revised: 1 February 2020 / Accepted: 5 February 2020 / Published: 9 February 2020
(This article belongs to the Special Issue Computer-aided Biomedical Imaging 2020: Advances and Prospects)

Abstract

:
With the accelerated development of medical imaging equipment and techniques, image fusion technology has been effectively applied for diagnosis, biopsy and radiofrequency ablation, especially for liver tumor. Tumor treatment relying on a single medical imaging modality might face challenges, due to the deep positioning of the lesions, operation history and the specific background conditions of the liver disease. Image fusion technology has been employed to address these challenges. Using the image fusion technology, one could obtain real-time anatomical imaging superimposed by functional images showing the same plane to facilitate the diagnosis and treatments of liver tumors. This paper presents a review of the key principles of image fusion technology, its application in tumor treatments, particularly in liver tumors, and concludes with a discussion of the limitations and prospects of the image fusion technology.

1. Introduction

Medical imaging equipment has developed rapidly in the last decade, with widespread usage in clinical diagnosis and treatment. Two main imaging modes are employed that utilize different principles and equipment. The first is the anatomical imaging mode that mainly provides anatomical information with high resolution. The X-ray-based method, computed tomography (CT), which falls into the category of anatomical imaging, is the first technique developed for the noninvasive acquisition of images within the human body. CT is particularly effective for imaging tissues with large differences in density. At present, whole-body scans can be performed with the latest generation of CT systems, including multi-slice detectors that allow precise visualization, even for very small vessels. Magnetic resonance imaging (MRI) uses radio waves and magnets to generate body tissue images. Compared to CT, MRI uses nonionizing electromagnetic radiation, and appears devoid of exposure-related hazards. The technique employs radiofrequency (RF) radiation in the presence of carefully controlled magnetic fields to produce high-quality, cross-sectional images of the body in any plane. Using MRI, high spatial resolution can be effectively used to identify soft tissue within the human body [1,2].
The functional imaging mode mainly provides functional metabolic information. One such method is single photon emission computed tomography (SPECT). SPECT imaging instruments provide three-dimensional (tomographic) images of the distribution of radioactive tracer molecules introduced into the body which is generated from multiple 2D images of the body at different angles [3]. Another widely used method in this mode is positron emission tomography (PET). PET is a nuclear medicine functional imaging technique used to observe metabolic processes in tissue of organs, as an aid to disease diagnosis [4].
The main difference between SPECT and PET is the decay mechanism of the radiotracers used: while SPECT measures photons of Gamma decay from a tracer nuclide, the PET scan uses 0.511-kev annihilation photons that are created when positrons, which are emitted from radiotracers, come to rest and meet with free electrons in organs. At a relatively low resolution, both SPECT and PET can be applied to reflect functional and metabolic information. Ultrasonography (US), which also falls into this category, is a technique for detecting the scattering and reflection of sound at an ultra-frequency level that allows the effective imaging of muscle, soft tissue, blood vessel and bone surfaces. US equipment is easy to use, and produces real-time images. However, the depth penetration of US is limited by several factors, such as the medium through which the ultrasound travels.
All these available image techniques have different strengths and weaknesses. Therefore, an optimal combination of these methods can allow a simultaneous expression of information from various aspects of the human body within a single image. Such an image can accurately reflect internal structure and function, in turn providing physiological and pathological information. These image-gathering and analysis procedures compose the medical image fusion process. The current review presents a chronicle of available studies on the application of image fusion in malignant tumor diagnosis, with particular focus on liver cancer. Various methods exist for a comprehensive liver cancer treatment, including surgical resection, radiofrequency ablation, tumor embolization, etc. In clinical practice, an individualized treatment plan is established, relying on patient information such as the localization of lesions, and is highly dependent upon imaging technology. On the basis of Ultrasonography, CT, MRI, PET and other imaging technologies, the key for optimizing multidisciplinary treatment for liver cancer is to improve and combine existing medical imaging technologies, which would allow the clear visualization of tumor lesions along with their characteristic indicators, such as tumor size, margin, the absence of vascular invasion, adjacent structure involvement, lymph node metastasis and distant spread.
The remainder of this article is divided into five sections. Section 2 discusses the imaging methods commonly used in liver tumor diagnosis and treatment; Section 3 provides a brief introduction of the general procedure used for medical image fusion; Section 4 presents several image fusion algorithms; Section 5 describes the application of image fusion in liver tumor imaging; and Section 6 covers our Conclusion and suggestions for further research.

2. Medical Imaging Methods

2.1. CT

An X-ray beam is used to scan internal organs up to a certain thickness. In this technique, a detector receives X-ray attenuation values of the organs in different directions on this plane, followed by obtaining the digital matrix of the tissue attenuation coefficient of a scanning layer after data transformation. The values in the matrix are converted and displayed on the fluorescent screen with different grayscale in black and white to generate CT images [5]. CT plays a critical role in clinical diagnosis and the treatment of liver cancer.

2.2. MRI

By applying a certain radiofrequency pulse to an organ in a static magnetic field, H protons in the organ tissue are excited, leading to the phenomenon of magnetic resonance. Upon termination of the radiofrequency pulse, H protons induce magnetic resonance signals during the relaxation process. After receiving magnetic resonance signals, spatial coding and image reconstruction, magnetic resonance images are generated [6]. MRI images achieve an excellent soft tissue resolution of liver and other organs, with a clear display of anatomical structures, such as vascular and biliary systems and lesion morphology, and provide valuable multi-orientation and multiparameter information, such as diffusion-weighted imaging (DWI), perfusion-weighted imaging (PWI) and MR spectra.
These data aid in determining the spatial location of the anatomical structure, and are beneficial for the metabolic analysis of liver tumors, leading to improved diagnostic value. However, the image can be affected by gastrointestinal gas and respiratory movement, and the location, puncture and ablation of lesions under the guidance of MRI all require special instruments and devices. Based on the same mechanics as MRI in terms of atomic physics, function MRI (fMRI) generates images of the metabolic activities within the anatomic structures generated by MRI scans [2].

2.3. PET

A positron that is emitted from a radionuclide-labeled compound (commonly 18FDG) annihilates with an electron resulting in two 0.511-kev photons. These two photons are emitted in nearly opposite directions (180-degree angle apart, at one particular location in liver), and are registered simultaneously by the ring detector around the patient, so that the PET can accurately locate, analyze and quantify the distribution of radioactively-labeled drugs in the body. After computer reconstruction, three-dimensional human body images are obtained [5]. Changes in the physiological and biochemical levels of liver cancer and normal liver tissue cells can be determined noninvasively, quantitatively and dynamically in vitro through C, N, O, F and other nuclide markers of glucose required for tissue metabolism, with the aim of evaluating the distribution and activity of liver cancer cells in patients. Therefore, PET is a functional molecular imaging technique with high sensitivity, and can effectively aid in characterizing the metabolism of liver cancer, detecting recurrence and evaluating the outcome of radiofrequency or microwave ablation therapy.

2.4. Ultrasonography and Contrast-Enhanced Ultrasonography

Ultrasonography is an important tool widely used in the study of the anatomy and morphology of the liver. Ultrasonography can be used to obtain sonograms of every section of the organ, which allows the detection of morphological and histological changes of organs or lesions to facilitate diagnosis. Furthermore, by injecting contrast agent (e.g., microbubbles) into patients, the contrast-enhanced ultrasonography (CEUS), in which the contrast of structures or fluid within the patient can be further enhanced, is often applied for the diagnosis of liver lesions. But, due to the operator dependence and the difficulty in detecting the lesion at the top of the liver, and small liver cancer is sometimes missed, so it needs to be integrated with other imaging methods.
All these imaging technologies provide quantitative information from different aspects of tumor lesions. The geometrical parameters of tumors, such as lesion size, capsule integrity, lesion blood flow and other morphological features, are dependent on Ultrasonography, CT or MRI imaging. On the other hand, 18F-FDG PET is the main medical imaging technology used to study the characteristics of tumor metabolism. However, limited information is available from a single method. For example, ultrasound and contrast-enhanced ultrasound have high diagnostic efficacy in detecting intrahepatic lesions and they allow dynamic visualization of lesions from multiple sections, with particular utility in the detection of small lesions less than 1 cm, but they could not generate data on the spatial structure of lesions and their spatial relationship with the surrounding hepatic artery and portal vein system. Whereas, CT or MRI provide information on the relationship between the spatial location of lesions and the surrounding structure, but do not allow dynamic observation of liver lesions from multiple angles. Therefore, the application of integrated information from a fusion of different medical imaging technologies in clinical diagnosis not only achieves the cross-validation of data from different imaging methods, but also improves the detection of tumor lesions, facilitating early diagnosis and the development of effective treatment plans.

3. General Procedure of Medical Image Fusion

Image fusion technology is mainly applied to solve the limitations of single-modal image guidance, including optical, medical and electromagnetic tracking imaging [7]. The medical fusion method contains two stages: image registration and the fusion of relevant features from the registered image.
Registration of different imaging modals is performed using external sensor coils, internal references, or anatomical markers [8]. The registration of the images requires a method to correct the spatial misalignment between the multimodal images that often results from scale changes, rotations and translations. This step matches the input images using their characteristics in order to facilitate the image fusions. The next stage is to find some rules to integrate multiple input images into one comprehensive image. The medical image can be fused by each pixel, feature extraction, region segmentation and marker point determination of anatomical structure or lesion condition. After fusion is complete, the operator interface often displays with original and overlapping cross-sectional images side-by-side. The fused image can help doctors to make accurate decisions for various diagnoses. For the efficient treatment of liver tumors, information on tumor size, location and number can be obtained by the fusion method accurately. Compared with ultrasound technology used previously, the development of new image fusion technology has greatly improved diagnostic accuracy [9]. For instance, image fusion guidance technology has been widely used in thermal ablation therapy in which two-dimensional Ultrasonography does not clearly show liver cancer lesions. This method uses high-contrast CT/MRI along with real-time guidance and the evaluation of ablation borders via Ultrasonography to demonstrate clear liver cancer lesions. More examples of medical image fusion applied in liver tumor diagnosis and treatment will be further discussed in Section 5.
In these procedures, image registration across modalities is important, and could highly impact the qualities of image fusion. As a result, we briefly introduce the medical image registration which mainly contains four steps:
  • Feature extraction: The first step of image registration is to extract image features (feature descriptor), such as feature points/edges/contours/areas/structures, from input images.
  • Feature matching: The second step of image registration is feature matching. It is used to find the correspondence between the extracted features from Step 1.
  • Determination of geometric transformation parameters: This is the most important step in the image registration procedure. Based on the correspondence between the extracted features from Step 2, a suitable geometric transformation model is selected. Then, based on a certain measurement function, the geometric transformation parameters are determined. The commonly used geometric transformation models are:
    • Rigid transformation, which is mainly a transformation for rotation and translation.
    • Similarity transformation, which is mainly a transformation for translation, rotation and scaling.
    • Affine transformation, which is mainly a transformation for translation, rotation, scaling and shearing
    • Projective transformation, which is a combination of transformations, such as translation, rotation, scaling and shearing.
  • Image resampling and registration: Then, with appropriate interpolation function, a floating image is mapped to the reference image’s coordinate space to finish the image registration (floating and reference images are defined as the input images to be registered).
Among the above steps, the optimization of geometric transformation parameters in Step 3 is crucial for the quality of image registration. A function is often used to measure the similarity between the floating and reference images. Common functions are root mean square (rms), correlation, normalized cross-correlation, gradient cross-correlation, gradient difference, image entropy, mutual information, normalized mutual information, etc. Then, this function is maximized by optimizing the transformation parameters. This converted the problem to multiparametric and multipeak optimization problems. Traditional optimization methods include the gradient descent method, conjugate gradient method and genetic algorithm, etc.
Recently, deep learning has become more and more popular for image fusions and registration [10]. One approach of deep learning used for image registration is to drive iterative optimization using deep learning.
In such a method, instead of using a traditional feature descriptor, we train deep learning to learn the feature descriptor to guide the fusion of the image. For instance, a deep learning algorithm, the convolutional autoencoder, is often used to extracted features from the images for the registration [11]. Then by optimization methods such as gradient descent, the measurement function is maximized and the image is registered. In the medical field, this kind of method has been used in the registration and fusion of CT/MR [12,13] and MR/US [14,15,16]. However, such iterative method often requires a long time, and has a difficulty to achieve efficient real-time registration.
The second approach is to pretrain the deep learning network to directly obtain the transformation parameters for the image registration. Such a deep learning network can be further divided into two categories: supervised learning and unsupervised learning. Supervised learning has been applied to register/fuse CT/US [17] and MR/US [18,19,20,21]. The training of the supervised learning network requires the ground truth. Two kinds of ground truth are generally used: ground truth data from the traditional registration method, as shown in above (i.e., step 1 to step 4) [22,23] and simulated ground truth data. For example, Eppenhof and Pluim generated image pairs using random transformations and obtained ground truth data for the image registration of CT images [24]. Similar ground truth generation methods are used for the fusion of MR images [25].
For unsupervised learning, the deep learning network is trained without the need of ground truth data. The most widely used network is Voxelmorph. This framework, which was proposed by Balakrishnan et al. [26], trains the network using a metric that quantifies image which are similar to input images. In the medical field, unsupervised learning has been used for the registration/fusion of CT/MR [27,28,29] and US/MRI [30].

4. Imaging Fusion Algorithms

There are three levels of image fusion: the pixel level, feature level and decision level. Pixel level fusion is the most basic image fusion method, which directly acts on the pixels in the image and does not need to extract features, but requires strict image registration. Feature level fusion requires extracting features in the image, such as size, edge, shape, texture information and other details. Decision-level fusion is able to extract, identify and classify valuable objects in the fusion image, and perform fusion at a higher level. For medical image fusion, both the pixel level and feature level are usually applied. Due to the particularity of liver imaging, it is necessary to combine several methods to achieve image fusion. The following are the commonly used image fusion methods.

4.1. Arithmetic Combination

The fastest fusion method is the arithmetic combination. Simple weighted fusion, also known as ‘Weighted Averaging’, is of the most simple and straightforward methods in arithmetic combination. The principle of the weighted average image fusion algorithm is to take pixel values of the original images directly and perform weighted averaging to obtain the pixel value of the fused image. Similarly, in the Simple Maximum/Minimum Method, the resultant fused image is obtained by selecting the maximum/minimum intensity of corresponding pixels from the input images. Arithmetic combination has the advantages of easy implementation and fast calculation speed. However, detailed information within the image cannot be captured, image contrast is reduced, and the edge of the image is altered with this technique, resulting in unsatisfactory fusion effects in most applications. Furthermore, due to this method requiring strict registration in advance, the fusion effect of noisy image is no ideal.

4.2. IHS

A color image can be represented by a three-primary red-green-blue (RGB) color system. However, this system does not conform to the human understanding of color. Another method of understanding color utilizes hue H, saturation S and intensity I. The hue H is determined by the dominant wavelength of the spectrum, saturation S characterizes the portion of the dominant wavelength of the spectrum and intensity I represents the brightness of the spectrum. In RGB space, three-spectral coordinate (R, G and B) coupling is strong, and changes in any component of the spectral information will alter the entire spectrum. As a result, image processing in RGB space is difficult. On the other hand, in IHS space, main spectral information is reflected in hue and saturation, while changes in intensity have a very limited effect on the spectral information, and are easy to process. The main objective in processing high resolution and multispectral images is to add details of high-resolution images while retaining spectral information. Consequently, it is easier to conduct image fusion in IHS space. Examples of transformation from RGB values to IHS values are shown in Equations (1)–(3). In the medical field, IHS is useful for the fusion of Pseudocolor image processing and fusion. Pseudocolor image converts the grayscale values to the RGB values. Its aim is to better present the details on the medical image in order to obtain a clear visualization of the images. A good example of such an image is the Pseudocolor-PET images, which images are widely used for liver lesion detection [31,32]. The fusion of Pseudocolor-PET and MRI medical image, which utilize fusion methods, such as IHS-PCA [31], IHS-wavelet transformation-based method [33,34] and IHS-salient features extraction [35], has shown to be useful for both human visualization and the objective evaluation of lesions.
[ I ν 1 ν 2 ] = [ 1 / 3 2 / 6 1 / 2   1 / 3 2 / 6 1 / 2   1 / 3 2 2 / 6 0 ] [ R G B ] ,
H = t a n 1 ( ν 2 ν 1 ) ,
S = ν 1 2 + ν 2 2 ,
here, ν 1 and ν 2 are the translation values.

4.3. Principal Component Analysis

Principal component analysis (PCA) is a technique for dimensionality reduction for a large dataset. PCA is mathematically defined as an orthogonal linear transformation method that transforms data to new coordinate systems, such that the greatest variance by a scalar projection of data lies on the first coordinate, and so on [36]. In this manner, PCA [37,38,39] helps to reduce the noise and redundant information and highlight the key feature in the dataset. PCA is widely used in various applications, including image compression, image enhancement, image coding, random noise signal removal and image rotation. For image fusion, PCA can extract the key features of the images, which highlights the similarities and differences between the input images, while reducing the noise level at the same time. Then, based on these key features, we can find the optimal weights for transferring the input image information to fused images. Here, we present an example from Miao et al. [40] We define elements of matrices I A and I B representing the gray level or color of each pixel in the input images A and B, respectively. First, the wavelet-based method is used to decompose the input images to low and high frequency components. Secondly, by using PCA, the eigenvectors of image A and B could be obtained as (XA, XB)T. Thirdly, the weight values of image A and B, w A and w B , for the low frequency part, are obtained as:
w A = X A X A + X B ;   w B = X B X A + X B ,
Fourthly, the low frequency fusion is completed as:
I F = w A I A + w B I B
At last, the high frequency fusion is achieved by maximum weight method:
I F = max { a b s ( I A ) ,   a b s ( I B ) }
Then, the fused image is obtained by combination of the new high frequency and low frequency images. In the medical field, PCA has been applied in the fusion of MRI, CT, PET and US [41,42,43]. PCA can also combine with decomposition methods, such as IHS, the pyramid method, Discrete wavelet transform [44], the Curvelet transform, Contourlet transform [45] and Non-Subsampled Contourlet transform [31,46,47,48,49,50,51,52,53,54,55,56].

4.4. Pyramid Method

The principle of the pyramid method is to decompose individual images within the fusion into a multiscale pyramid image sequence (i.e., reduce the resolution of the image in a pyramid sequence shown in Figure 1). The low-resolution image is in the upper layer and high-resolution image in the lower layer, with the upper layer image being 1/4 of the previous layer image size. The pyramid of all the images is fused to the corresponding layer using a specific rule. The synthetic pyramid obtained is reconstructed according to the inverse process of pyramid generation. Based on this theory, multiple pyramid fusion algorithms (e.g., Gaussian pyramid and Laplacian Pyramid) have been proposed with different pyramid decomposition structures, fusion rules and reconstruction methods. In the medical field, the pyramid method has been applied in fusing multimodal medical images, such as MRI/CT, PET/MRI and SPECT/MRI [57,58,59].

4.5. Wavelet Transformation Based Methods

In the field of image fusion, the wavelet transform-based method, which was initially developed for signal processing [60], is widely used as high-pass filtering. The detailed image is the result of high contrast corresponding to high values in the frequency domain. The detail of image is the result of high contrast, which corresponds to high values in the frequency domain. By DWT, we can detect these details in the image, using functions that are localized in both space and frequency. For image fusion, these detailed pieces of information from input images can then be extracted and fused into a new image using certain fusion rules, such as maximum selection, weighted average and PCA. The low frequency part of the images can then be fused in a similar way.
DWT has been widely used in CT/MRI and MRI/PET medical image fusion [61,62,63,64,65,66,67,68]. However, DWT is known to be sensitive to the translation/shift of input signals, and therefore, translation among signals may exert a negative impact on effectiveness. Contourlet transform is a two-dimensional image representation based on wavelet multiscale analysis known as Pyramidal Directional Filter Bank (PDFB) [69]. Compared with DWT, its basis functions are characterized by multiscale features, directionality, anisotropy and locality.
Such basis functions effectively represent edge and curve singularity, and allow the efficient extraction of geometric and texture information in the image to obtain a better fusion effect. The multiscale geometric analytical tool used in Contourlet transform demonstrates the excellent spatial and frequency domain localization properties of wavelet analysis, as well as the bonus of multidirectional and multiscale characteristics, good anisotropy and suitability to describe the geometric characteristics of an image [70,71]. However, for the Contourlet transform, shift invariance is lost as a result of its subsampling scheme for the multiscale partition. To overcome this difficulty, researchers have introduced the improved version of Contourlet transform, the Non-Subsampled Contourlet transform (NSCT) [72]. In the medical field, the Contourlet transform and the NSCT have also been to fuse MRI/PET and CT/MRI [70,73,74]. Another deficit of wavelet transformation is the lack of ability to represent edges and geometric structures of the image. Curvelet transform [75,76,77], a multiresolution and multi-direction pyramid that can preserve geometric regularity along edges [78], has been proposed to overcome this difficulty. Ali et al. [79] proposed a Curvelet transform (CVT)-based method for the combination of CT and MRI. However, as highlighted in other studies [80], CVT is not built directly in the discrete domain, and thus does not provide a multiresolution representation of geometry. Shearlet transform (ST) and non-subsampled Shearlet transform (NSCT) [81,82,83,84] are other sets of state-of-the-art tools with optimal use in sparse directional image representation. Based on composite wavelets, an optimal approximation of 2D functions is obtained. Compared to the Contourlet method, these methods have the advantage of directional selectivity and computational efficiency. Due to no restrictions on the number of directions for shearing, ST is used for the fusion of 2D and 3D medical images [82,83] and NSST has application in CT/MRI image fusion [85,86].

4.6. Pulse-Coupled Neural Network

For image fusion, a pulse-coupled neural network (PCNN) is often used as a feature extraction method [87,88]. As shown in Figure 2, the PCNN adopts a single layer, two-dimensional and laterally-connected neural network. The neurons are connected with the pixels in the input images directly. So, the size of PCNN is equal to the size of images. Each of the neurons is also connected with neighboring neurons, as shown in Figure 2. Image feature extraction using PCNN is an iteration process. At each iteration, each neuron receives the corresponding pixel’s color intensity as an external stimulus. The outputs of its neighboring neurons from the previous iteration are treated as an internal stimulus and are combined with the external stimulus. When the total stimulus exceeds a threshold, the neuron will pulse (or fire) to have an output intensity equal to one at the corresponding location in the output image. The threshold at the neuron will significantly increase its value after firing and it decays exponentially until the neuron fires again.
Through iterative computation, PCNN neurons produce a series of pulse outputs, which contain different features (e.g., high frequency features, low frequency features or edges) of the input images, and can be used for various image processing applications. Multichannel PCNN is proposed to process multiple feature images with a single/multiple PCNN to fuse these images [90].
Nevertheless, PCNN has its shortcomings, such as numerous parameters and the complex process of setting parameters. To date, several novel algorithms to optimize the PCNN parameters have been introduced to improve calculation speed [91,92]. In image fusion, PCNN has also been used as fusion rule for wavelet transform-based methods, such as DWT [93,94,95,96], Contourlet transform [97,98,99,100], NSCT [101,102,103,104,105], ST [106,107,108,109] and NSST [110]. In medical imaging, PCNN is widely applied to fuse CT/MRI [94,111,112,113], MRI/PET [94], MRI/SPECT [113,114], US/SPECT [111] and multiparametric MR images [84].

4.7. Fuzzy Logic Based Methods

Fuzzy Logic is a multivalued logic that allows intermediate values to be defined between conventional evaluations, such as true/false, yes/no, high/low. Fuzzy systems refer to those that are directly related to fuzzy logic. These systems are mainly composed of fuzzification, knowledge bases, fuzzy inference engines and defuzzification. Fuzzification is the conversion of the input of a system to fuzzy sets with some degree of membership anywhere within the interval using a membership function. A membership function is a curve that defines how each point in the input space is mapped to a membership value. The knowledge base stores all information on the fuzzy controller, including knowledge and required control objectives in the specific application field. These core factors determine the performance of the fuzzy controller. The function of the fuzzy inference engine is to convert the fuzzy “if-then” rule into a type of mapping according to the fuzzy logic rule. Defuzzification is the conversion of the fuzzy output quantity into clear output.
Fuzzy logic is also applicable to image fusion. In this process, local features of the image are extracted and combined with fuzzy logic to compute weights for each pixel [115,116]. The fuzzy logic-based fusion rule is often used to cope with blurry image fusion. As a fusion rule, it could be further combined with DWT [117], NSCT [35,118,119] and NSST [120] for medical imaging. In this application, DWT/NSCT/NSST was performed on source images to obtain high- and low-frequency sub-bands. Next, a logic-based fusion rule was applied for the fusion of high [35,117,118] or low [120] sub-bands, and for the enhancing of the global contrast of the image [119]. The Fuzzy logic algorithm could overcome defects of losing edge information and color distortion in DWT/NSCT/NSST to improve image contrast. Neuro-fuzzy combines artificial neural networks with fuzzy logic to generate a resulting hybrid intelligent system. The humanlike reasoning style fuzzy system is combined with the learning procedure of artificial neural networks. This approach utilizes an artificial neural network to train the parameters of the membership function, and has been used in the fusion of MRI/CT images [121,122]. Similar to fuzzy logic fusion, neuro-fuzzy logic can be combined with WT, Contourlet transform [123], NSST [124] and NSCT [125] to optimize fusion performance.

4.8. Sparse Representation and Compressive Sensing Based Methods

Recently, the sparse representation of signals has become a popular topic of research. The method assumes the input image (usually expressed as a column vector) can be represented by a linear combination of several elements (a series of column vectors), which is referred to as atoms. The atoms compose a dictionary. In this technique, the most important issue is the dictionary choice. Two methods are usually employed: (1) an analytical dictionary built by selecting a specified transformation matrix via Fourier, wavelet, Curvelet and Gabor transform, and (2) dictionary learning built based on data training. The second method effectively achieves higher accuracy in extracting complex image features, a better space representation of various features of images and good adaptability. The common procedures in this category include K-means generalized SVD (K-SVD) [126,127,128,129,130], PCA [131,132], online dictionary learning, optimal directions and adaptive sparse representation [133]. However, such multidimensional signal/image processing usually involves a large amount of data. According to the compressive sensing theory [134], the image can be compressed with a few random projections if the image is sparse in a certain transform domain and can be sparsely represented [135,136,137,138].
Experimental results show that the image fusion using compressive sensing could preserve the rich texture information of the input images while reducing the amount of data required and the complexity of the algorithm processing [135,136]. During training, the dictionary for the sparse representation, the test sample (input images), can be represented as a column vector. Similarly, we create a dictionary matrix of a column vector which represents the training samples (atoms). Compressive sensing, using random projection, can be applied to reduce the dimensions of both the test vector and the dictionary matrix. Then, representation coefficients are obtained using sparse coding techniques, such as orthogonal matching pursuit, simultaneous orthogonal matching pursuit, a joint sparse representation model, approximate sparse representation with multi-selection strategy and convolutional sparse representation. At last, the sparse representation coefficients from the input images are fused by a certain fusion rule. Finally, the fused image is reconstructed by fused sparse representation coefficients. For instance, PET/CT/MRI images are fused using the K-SVD-based learning dictionary and Orthogonal Matching Pursuit (OMP) algorithm [139]. Similarly, Sparse representation-based methods have been applied for fusions of CT/MRI and MR images [138,140,141].

4.9. Edge-Preserving Based Methods

The edge-preserving filter emerged is an effective tool for image processing applications. The overall grayscale of traditional image smoothing filtering tends to be consistent in the neighborhood to achieve a smoothing effect. It is useful for the image with the similar pixel values (grayscale). However, this assumption does not hold at the edge of the image, which contains key information, since in this region, the grayscale tends to vary significantly with space coordinates. These grayscale variations provide meaningful image information. Thus, in many applications, weakening or filtering of the edges of the image during the filtering process is not desirable. Bilateral filters [142,143] can solve the problem of low-pass filtering. A grayscale difference weighting value is introduced into the original spatial low-pass filter based on the original spatial filtering weight. The grayscale phase and spatial difference distances have similar local characteristics. This suggests that when a point differs greatly from the center point in grayscale, it is considered that the point is distant from that center point. Guided filter is a similar edge-preserving smoothing filter that applies an optimal local linear approximation to achieve the edge-preserving goal. In this method, the guiding filter guides the filter with a guiding graph G, while the filter window radius, r, and the smoothing intensity, ε, are adjustable parameters. The linear transformation of the model ensures that the appearance of the output edge only depends on the edge of the guiding graph G. After the image is smoothed by edge preservation, the large edge structure of the image is preserved, and small fluctuations corresponding to noise are smoothed out. The above characteristics of bilateral and guided filters can optimize fusion weight to ensure that the fusion is more smoothly connected and visually natural. These features are extremely useful in medical image fusion applications, such as fusion of CT and MR images [142,143].

4.10. Deep Learning(DL) Methods

The DL algorithm has strong ability of feature extraction and data representation, and has made advanced achievements in medical image processing. The application of DL techniques in the field of image fusion has emerged as an active topic in the last three years.

4.10.1. Convolutional Neural Networks

The popular deep learning model, convolutional neural networks, can provide some new way in image fusion. Convolutional Neural Networks (CNNs) are able to extract the most important features from a large number of samples. The CNN uses a system much like a multilayer perceptron that has been designed to reduce processing requirements. CNNs consist of an input layer, an output layer and a hidden layer that includes multiple convolutional layers, pooling layers, fully connected layers and normalization layers. A convolutional layer defines multiple filters as a window and subsequently scans the entire image through this window.
It can output many feature mappings after training. The output can provide a multiscale and multiangle feature, it can also provide the location information of the feature. The advantages of feature extraction by CNN can be fully utilized in pixel or feature level image fusion. Another operation in CNNs is spatial pooling (max-pooling, min-pooling), which can bring some desirable invariances, including translation, rotation and scale into the model to a certain extent. Fully connected layers act as the role of a classifier. CNN overcomes the difficulty on manually designing complicated activity level measurement and fusion rules [144]. The activity level measurement and fusion rule can be jointly generated via training a CNN model. The feasibility of CNNs used for medical image fusion have already been proposed [145]. CNN can decompose the original images to high frequency and low frequency images [146], and select the rule of regional matching to fuse the two high frequency and low frequency images to get the final fusion images. Kumar et al. [147] developed a supervised CNN to learn to merge the data from PET-CT images of lung cancer. CNN has also been applied to fuse medical images MRI/CT, MRI/SPECT, multiparametric MR images [148] and PET/MRI [149]. CNN can also be combined with a wavelet transform for the fusion of CT and MR images [150]. In this method, the wavelet transform coefficients are first obtained by decomposing the input images. The next step is to use the trained CNN model to improve the high frequency coefficient’s resolution. A similar procedure is applied to combine CNN with NSST to merge the CT and MR images [151].

4.10.2. Convolutional Sparse Representation

Sparse representation has been widely used in various image fusion. Due to the modeling burden and computational cost, traditional sparse representation has always been performed on local image patches rather than on the entire image. The concept of convolutional sparse coding (CSC) originates from the deconvolutional networks proposed by Zeiler et al. [152]. Its fundamental principle is to get an image’s convolutional decomposition with a sparsity constraint. As an image representation approach, CSC is also termed as convolutional sparse representation (CSR). On the contrast to the conventional sparse representation, the sparse representation of an entire image can be computed in the CSR model. In this CSR model, the obtained representation is single-valued and optimized over the entire image [153,154]. Liu et al. [153] introduced the CSR into the field of medical image fusion on MRI/CT. Qiu C et al. proposed a novel fusion method based on convolutional sparse representation (CSR) to fuse the mis-registered GFP and phase contrast images in biomedical image fusion [155,156].

4.10.3. Stacked Autoencoders

A standard stacked autoencoder (SAE) is formed by stacking multiple autoencoders. The autoencoder can be learned by pretraining each layer before its successor using a back-propagation algorithm. At each layer, an autoencoder is used to obtain a set of features by jointly using an encoder and a decoder [157]. To prevent learning a trivial solution, stacked sparse autoencoders and stacked denoising autoencoders [158] have been applied to improve the SAEs’ methods. SAE-based DL models have been applied to image fusion for multimodal medical image feature extraction. These extracted features can be used to design optimal fusion rules [159] and to obtain better fusion images. In this method, a multitask loss function related to image fusion quality is used to train the network.
In summary, the clinical applications of the fusion methods are shown in Table 1.

4.11. Image Fusion Indicators

Several image fusion indicators can demonstrate fusion quality. One is the fused image assessment. The other is the fused image metrics. We can obtain the assessment results by the subjective ratings, computational metrics and objective human tasks.
A number of image quality metrics [160] have been proposed, including mean square error (MSE), root mean square error (RMSE), peak signal-to-noise ratio (PSNR), mean absolute error (MAE), quality index, mutual information (MI), the Petrovic and Xydeas metric and Piella’s Quality Index [161].

5. Clinical Applications of Image Fusion in Liver Cancer Diagnosis and Treatment

5.1. Ultrasonography/MRI, Ultrasonography/CT and Ultrasonography /PET-CT

Ultrasonography, a traditional, practical and convenient imaging technology, presents major advantages in the diagnosis of liver cancer. The recently developed ultrasound-based fusion imaging technology plays an important role in early diagnosis and ensuring minimally invasive procedures [170]. Several liver tumor types exist, including benign and primary liver malignant tumors and metastatic liver cancer. Benign liver lesions include hemangioma, focal nodular hyperplasia (FNH) and hepatic adenoma. Primary liver malignancies originate from hepatocytes, intrahepatic bile duct epithelial cells, endothelial cells and connective tissue. A number of tumor types, such as FNH and hepatic adenoma, have similar manifestations in enhanced CT or dynamically-enhanced MRI, and are therefore often difficult to distinguish. As the concept of fusion imaging technology was gradually implemented in clinical practice in the 1990s, ultrasonography was initially applied to the field of image fusion to identify the types of liver tumor [171].
CEUS/CT and CEUS/MRI fusion images were successfully generated and employed to dynamically observe blood flow in lesions and the blood perfusion of tumors, facilitating the diagnosis of different types of intrahepatic lesions, and in particular, minor lesions. CEUS/CT and MRI fusion imaging clearly achieves higher diagnostic efficacy than CEUS, CT or MRI alone. At present, image fusion technology based on CEUS is mainly applied for (1) diagnosis and treatment of small liver cancer, (2) the evaluation of minimally invasive treatment methods, such as liver tumor TACE and RFA, and (3) the early diagnosis and treatment of new or recurrent liver cancers or liver metastases after surgery.

5.1.1. Utility of CEUS/MRI in Diagnosis and Treatment of Small Liver Cancer

The natural course of subclinical stage liver cancer is at least two years. Through the study of subclinical stage liver cancer, the 5-year survival rate of small liver cancer is markedly higher than that of advanced large liver cancer.
However, the clinical diagnosis rate of small liver cancer is a big problem for clinicians. In addition to screening for AFP indicators in blood samples, effective imaging examination methods urgently require application to optimize clinical practice.
Contrast-enhanced ultrasound has a high diagnostic efficacy for liver lesions, especially small lesions less than 1 cm [162]. However, Ultrasonography presents inherent limitations. First, researchers with significant clinical experience are essential for effective implementation. Second, Ultrasonography lacks high resolution, and is unable to provide information on spatial hierarchical relationships. As mentioned above, MRI images are characterized by high soft tissue resolution and multiple signals, especially for intrahepatic vessel imaging. At the same time, micro-cancerous nodules, precancerous nodules and vascular cancer thrombus display different signals on MRI sequences. Introduction of the liver-specific contrast agent, Gd-EOB-DTPA (i.e., Gadoxetate Disodium-ethoxybenzyl-diethylenetriamine pentaacetic acid), has greatly improved the image quality of enhanced MRI scans, clearly revealing the boundaries of liver cancer and micro-lesions [172].
Multistage image fusion technology of contrast-enhanced ultrasound and MRI is reported to improve the accuracy of the diagnosis of small liver cancer. Originally, conventional ultrasound and MRI were used to conduct an image fusion of the axial profile of the liver. Furthermore, upon combination of contrast-enhanced ultrasonography to observe the blood flow direction of liver lesions, data from the fusion image of contrast-enhanced ultrasound (CEUS) and magnetic resonance imaging (MRI) allowed the determination of the three-dimensional parameter index of tumor lesions in liver (size, spatial location, blood supply arteries, drainage area), resulting in increased detection rates of suspicious small liver cancer. Moreover, the main blood supply source of liver cancer, portal vein tumor thrombus and the specific hepatic arterioportal fistulas caused by tumor pathological factors were clearly displayed.

5.1.2. Utility of CEUS/MRI in Radiofrequency Ablation of Liver Cancer

Radiofrequency ablation of liver has been widely applied in the radical treatment of small liver and multinodular liver cancers due to multiple advantages of safety, simplicity and minimal invasion. The 5-year survival rate following radiofrequency ablation of small liver cancer is reported to be close to that of surgical resection [173]. Based on the advantages of CEUS/CT fusion imaging technology, local ablation of liver tumors has gradually become an important supplementary adjunct to the surgical treatment of primary liver cancer and colorectal cancer with liver metastases.
CEUS/CT fusion imaging has many advantages over conventional ultrasound-guided treatment for liver lesions that are difficult to ablate. Conventional Ultrasonography is unsuitable in the following situations: (1) the target lesion for ablation shows a similar echo signal to surrounding sclerosing nodules of liver tissue, which are difficult to identify owing to low spatial resolution, (2) some target lesions cannot be clearly displayed due to the interference of diaphragm movement or close proximity to the diaphragm, colon and gas movement, and (3) after repeated TACE or RFA treatment, the local echo of advanced liver tumor lesions is mixed, the lesion boundary is blurred, and it is difficult to identify the initial and recurrent lesions. CEUS/CT fusion imaging technology solves the above issues by not only taking into account the advantages of the real-time and continuous dynamic observation of the ablation effect, but also combining the characteristics of high-resolution-enhanced CT localization of lesions. Therefore, accurate needle placement for lesions is achieved under difficult conditions, and the flexibility of Ultrasonography is used to adjust the direction of intraoperative needle insertion to pinpoint the locations of difficult lesions. Recent reports indicate that the success rate of fusion imaging-guided radiofrequency ablation technology has increased to 93% [8]. CEUS/CT fusion imaging additionally allows multiple ablation plans for complex liver lesions based on the characteristics of enhanced CT three-dimensional imaging, the reasonable placement of needles and the needling path, along with the formulation of other specific plans to reduce the occurrence of accidental cases of the mis-penetration of the anatomical structure and vital vessels around the lesion, such as the diaphragm, colon and small intestine, thus avoiding the incidence of pneumothorax, hydrothorax, colon fistula and intestinal fistula after radiofrequency ablation. In view of the above advantages, CEUS/CT fusion imaging technology facilitates precise liver radiofrequency ablation treatment, which has a high clinical value.

5.1.3. Utility of CEUS/PET-CT in Transarterial Chemoembolization Treatment of Liver Cancer

Transarterial Chemoembolization (TACE) has been employed for the treatment of liver tumors since the 1980s. With increasing information on the mechanisms underlying liver cancer progression, clinicians are gradually realizing that surgical resection is inadequate to treat cases with a background of liver cirrhosis and the biological characteristics of polycentric or multiple liver cancer, which are responsible for the high recurrence of liver cancer after surgery [174]. In addition, pathologists have confirmed that once the size of the hepatocellular carcinoma exceeds 5 cm, the incidence of tumor lesions invading the portal and hepatic vein branches to form vascular thrombus vessels is greatly increased, which is the basis of intrahepatic dissemination and distant hematogenous metastasis. TACE therapy for liver cancer has been gradually applied for clinical treatment with minimal trauma and significant efficacy. However, clinicians are yet to establish the type of liver tumor suitable for TACE therapy, the optimal means to evaluate the efficacy of TACE therapy and the specific indicators that should be evaluated before intervention. According to clinical practice, before interventional embolization for liver cancer, the scope of lesions and the number of sub-lesions should be determined, considering the clinical difficulty of the TACE-mediated control of the intrahepatic dissemination of tumors. Methods of application of existing imaging technologies to evaluate the residual tumor range after TACE, the specific times of new rounds of TACE treatment, and outcomes of tumor necrosis after treatment, are of clinical significance.
TACE treatment is based on vascular embolization, and its effect depends on efficient blood flow into the tumor. Although enhanced CT or MRI is currently accepted as the gold standard for the imaging diagnosis of liver cancer, information provided by enhanced CT or MRI is mostly static, which is unable to help clear tissue perfusion of the tumor-bearing liver segment. Detailed images of the blood vessels furnishing liver cancer could not be clearly defined in previous analyses, and it was therefore impossible to assess whether the route of TACE into the tumor-bearing liver segment was effective, and the level of tumor necrosis after TACE. CEUS can be used to display micro-perfusion in liver tissues in real time, but its success is highly dependent on the experience of the operator, with interference from various objective factors. First, reactive congestion of liver tissue around the lesion in the early stages after interventional embolization is reported to affect measurement of the range of residual tumor lesions [163]. Second, contrast-enhanced images of the focal arterial stage were not evident when TACE was applied to intrahepatic cholangiocarcinoma with blood supply deficiency, and the detection rate of residual lesions was low. Additionally, contrast enhancement and regression were rapid in the arterial phase of tumor lesions with a rich blood supply, presenting the imaging characteristics of “fast in and fast out” [162]. CEUS may overlook residual lesions in different section scans. CEUS/PET-CT fusion imaging technology integrates the advantages of CEUS and PET-CT or enhanced PET-CT to display iodine-oil deposition, the even distribution of the embolization agent and the area of tumor necrosis in liver cancer. Therefore, the technique allows not only accurate estimation of the scope of the tumor lesion necrosis area after TACE, but also the recognition of blood flow signals in the necrotic area and a timely detection of residual lesions, which is of significant value in improving the efficacy of TACE, reducing recurrence, and ultimately improving the survival rate and the quality of life of patients. Due to the speed requirement of real-time CUES and CT/MRI fusion, arithmetic combination is usually used in the above clinical applications. Here, we list one example to show the influence of image fusion of CUES and CT/MRI images in the real clinical application. The first example is from Xu et al. [164]. They performed CT/MRI and CEUS fusion imaging to evaluate treatment responses of 157 patients. From the results, if only the US modal is used, there were 26.7% (41/157) of inconspicuous lesions observed in the enrolled patients. While if US modal is fused with CT/MRI, there were only 1.3% (2/157) of inconspicuous lesions observed. One image fusion example of CEUS and MRI is shown in Figure 3 from Xu et al. [164].

5.2. CT/MRI

5.2.1. Detection of Intrahepatic Cholangiocarcinoma

Intrahepatic cholangiocarcinoma (ICC) is the second most common hepatic malignant tumor type after hepatocellular carcinoma, which is characteristically associated with invasive growth, the occurrence of satellite foci and intrahepatic metastasis [162]. Preoperative imaging findings of intrahepatic cholangiocarcinoma foci and their sub-foci and accurate delineation of the boundaries of tumor lesions provide the basis for good surgical results. To improve the long-term survival rate of patients with bile duct cell carcinoma, the elucidation of the spatial anatomical relationships of bile duct cell carcinoma foci with intrahepatic portal vein and hepatic venous systems, and an improvement of the R0 resection rate, are necessary steps.
Currently, multistage-enhanced CT and dynamic-enhanced MRI are commonly used in clinical practice for the detection of intrahepatic cholangiocarcinoma. These two technologies have the following disadvantages: the diagnostic rate for small tumor lesions <1 cm is not high, and an accurate display of tumor boundaries when the lesion differentiation degree is low and the capsule is incomplete, is difficult. The liver-specific contrast agent Gd-EOB-DTPA-enhanced MRI scan has the advantage of clearly displaying tumor lesion boundaries and intrahepatic microscopic lesions. However, due to the length of time required for MRI, images of intrahepatic vessels are likely to contain artifacts due to respiratory non-coordination [175]. With an extensive clinical application of medical image fusion technology, researchers have circumvented the disadvantages of the above techniques via the preoperative fusion of enhanced CT/MRI imaging in ICC. Fusion imaging allowed the determination of whether or not the lesion invades important hepatic vessels as well as establishment of the anatomical relationship between cholangiocarcinoma and satellite sub-focal lesions.
In other words, the technology facilitated the quantitative evaluation of whether main and sub-focal lesions were in the same tumor-bearing liver segment, and thus the possibility of combined vascular resection. In this way, preoperative assessment could be used to effectively guide whether to perform regular segment resection, combined segment resection or enlarged lobectomy, leading to the avoidance of unexpected situations, such as postoperative liver insufficiency and liver failure.

5.2.2. Surgical Operation Assistance

Image fusion is a valuable tool for planning treatment strategies and examining pathological changes. The CT portal vein image is automatically registered with Gd-EOB-DTPA-enhanced MRI images using Mitworkbetch software prior to the operation. The CT-MRI fusion image provides detailed lesion information, in turn, improving diagnostic accuracy [176,177]. At the same time, three-dimensional models and virtual surgical images based on CT/MRI fusion image reconstruction are applied to guide the key surgical procedures. Combined with indocyanine green molecular fluorescence images, CT/MRI image fusion efficiently defines tumor boundaries and identifies hidden microscopic lesions, leading to improved surgical precision. In addition, fusion imaging is particularly useful for lesions for which anatomical images are difficult to obtain from various angles with conventional techniques, such as anatomically complex hilar lesions. Based on tumor location in combination with the distance between the tumor and intrahepatic vasculature and spatial positional relationship, the optimal virtual resection plane can be determined. Additionally, fusion imaging may be effective for therapeutic evaluation. Posttreatment changes can be easily clarified by creating a fused image prior to treatment. Moreover, since 3D images can be obtained without difficulty, we may be able to successfully simulate surgical treatment in the future. Fusion imaging further allows patients to visually understand the disease process.
Intraoperative bleeding is a critical aspect of liver surgery, and an important factor affecting the success of surgery and the postoperative recovery of patients [178]. Enhanced preoperative CT/MRI fusion imaging can assist surgeons to better understand information related to intrahepatic vascular alignment, portal vein alignment variation, location of main hepatic vein branches and spatial distance from the tumor lesions. Thus, identification of important vessels and anatomical markers surrounding the lesion can be improved by the assessment of enhanced CT/MRI image fusions before the operation, which is vital in reducing intraoperative bleeding and operation times and accelerating postoperative recovery. Queisner et al. [179] conducted a series of clinical studies to evaluate the efficiency of contrast-enhanced CT/MRI image fusion technology in hepatectomy for different anatomical locations. Data from their study suggest that most preoperative surgical planning schemes based on image fusion are similar to the actual operative procedure conducted following the exploration of the liver. Therefore, routine preoperative contrast-enhanced CT/MRI image fusion could provide a valuable guide for planning surgical procedures, leading to the improvement of surgical treatment outcomes.
For the fusion of CT and MRI in real clinical applications, the common methods are Arithmetic combination [177], the PCA-wavelet transformation-based method [47,48,49,53,54,56,117] and the Pyramid method [58,59]. Here, we show one real clinical example for the assessment of the cryoablation margin using MRI-CT fusion imaging in hepatic malignancies [180]. From their study, it is shown that MRI-CT fusion imaging was achieved successfully in 46 (97.9%) of 47 lesions, and was useful for evaluating the Minimal ablative margin (MAM) of cryoablation in hepatic malignancies. An example of the fused MRI-CT images from Chen et al. [180] using Arithmetic combination is shown in Figure 4.

5.3. PET/CT & PET/MRI

PET/CT is the most widely used fusion imaging technique in clinical diagnosis. However, due to its imaging principles, PET/CT is less effective in locating tumor lesions in spatial resolution, and can only show a specific standard uptake value (SUV) range, which fails to accurately evaluate liver tumor position and its adjacent relationship in anatomical space. PET/MRI fusion technology has greater advantages in the assessment of tumor morphology, function and metabolic imaging than PET/CT [165,166]. Firstly, based on the advantages of diffusion-weighted imaging (DWI), perfusion imaging PWI and MR spectrum, MRI is far superior to CT in the functional imaging of human soft tissue organs. Secondly, the PET/MRI imaging system does not impose an ionizing radiation burden on the patient or operator. Thirdly, on CT images, signals of some abdominal and pelvic lesions may be disrupted by peristaltic bowel, poor bladder filling or uterine translocation. This type of interference is often unavoidable in the imaging process and affects the observation of diseased organs, which can be effectively overcome by the hybrid technology conditions of PET/MRI [167].
Clinical findings suggest that PET/MRI has an advantage in regular postoperative follow-up for patients at high risk of liver metastases from colorectal cancer with regard to the monitoring of tumor recurrence, especially in distinguishing inflammatory tissue around the surgical area of rectal cancer lesions, distant liver metastasis and the clinical TNM stage. With regard to adjacent tissue, taking into consideration both the metabolism of 18F-FDG in postoperative inflammatory tissues of rectal cancer and DWI images in enhanced MRI, morphologic and functional imaging can effectively discriminate whether the newly formed mass in the rectal cancer area is an inflammatory scar or recurrent tumor tissue [181]. In terms of N-staging, 18F-FDG activity is not specific to cancer, since it has been observed in macrophages involved in inflammatory and infectious diseases. Cancer patients with acute inflammatory or infectious diseases also display high SUV signals on PET/CT images, which makes it impossible to determine whether lymph nodes with high metabolic signals present an inflammatory lesion or neoplastic metastasis that affects the N stage of correctly diagnosed cancer patients. Owing to the significant benefits of the MRI in soft tissue imaging of lymph nodes, PET/MRI is obviously superior to PET/CT in distinguishing internal lymph node structures. Taking the morphological, functional and metabolic features of suspected lymph nodes into consideration, PET/MRI can effectively distinguish tissue structures, such as fat hilum, margin and necrotic area within lymph nodes, which allows determination of the tumor metabolism of lymph node tissue with high suspicion of metastasis, and thus the differentiation of malignant from benign lymph nodes. With regard to M-staging, PET/MRI, which can distinguish metastatic lesions less than 1 cm, is of greater diagnostic value in patients with suspected liver metastasis of rectal cancer. Reiner reported a higher diagnostic rate of PET/MRI relative to enhanced CT/PET fusion [182]. Simultaneously, PET/MRI in the diagnosis of primary liver cancer can clearly distinguish whether the portal and hepatic vein systems display any important pathological features of tumor thrombus involvement, which can provide a foundation for clinical decisions of subsequent treatment. For the fusion of PET-MRI for clinical applications, the wavelet transformation-based method [66,73,74,130], IHS-PCA [31] and deep learning methods [151,154] are generally used. On the other hand, wavelet transformation-based methods [52,57,168,169] and deep learning [147] are generally applied for the fusion of PET-CT. Here, we show an example in Figure 5, which compares the accuracy of the fused image of PET/MRI and single modal MRI in the correct identification of a patient with liver lesions [183]. The data from this study indicate that the fusion of PET/MRI can increase the identification rate of the liver malignant lesion from 94.4% to 100%.

6. Discussion on the Limitations and Prospects of Medical Image Fusion Technology

In the modern clinical practices, physicians have a higher demand on the accuracy and efficiency of a visually-aided medical diagnostic system. The image fusion techniques can efficiently process and combine the information from different image devices, which plays an important role in the precise positioning of tumors, the early diagnosis and treatment of cancer. With the advance of modern computer systems and medical imaging equipment, image fusion technology will be further developed, bringing a new revolution to clinical diagnosis. The research trend is to develop new algorithms which will make the registration of multimodal medical images more accurate, the fusion more efficient, and thus they will eventually achieve the purpose of improving the diagnostic effect. In the following, we present several challenges and the research trend in this topic:
The extensive application of different medical image modalities has played a globally recognized role in the diagnosis of the liver cancers [184]. However, these modalities still have several flaws. As a result, one research challenge and focus is to further improve the single modalities. These may include the reduction of the rising cost of these medical images, decreasing the patient’s exposure time to radiation, while maintaining the image quality [185,186]. In addition, current clinical application of image fusion is still limited to merge the medical images from two independent medical devices. As a result, the patient needs to receive multiple examinations. Consequently, image fusion is more expensive than single image technology, which limits its application in the clinical diagnosis and treatment of tumors.
To reduce the examination cost and the risk of additional radiation, it is ideal to develop devices which can perform multimodalities exams at the same time while maintaining high image qualities.
Development of efficient medical registration technologies is also very important. One topic is to address the significant alignment errors caused by patient breathing and motion compensation. In order to optimize linkage of the fused image, data from the unified respiratory phase should be used for registration, and the patient should remain in the same scanning position to the greatest possible extent, particularly for organs that move with the respiratory phase. Automatic registration will be the development direction of image fusion volume navigation technology, aiming to optimize the registration process and the fusion accuracy. In addition, improvement of the available algorithms in terms of accuracy and faster registration processes may promote the utility of fusion imaging. New algorithms and methods may be introduced to take into account organ movement caused by changes in breathing or position. An ideal system is based on automatic registration using complex electromagnetic tracking and computer-aided imaging algorithms, without effects on external reference points or anatomical landmarks selected by the user. These features would allow wider usage of this technology by individuals with less experience in image fusion. With further development of follow-up physical diagnosis technologies, improved fusion imaging may be widely applied in clinical practice to achieve an early diagnosis and treatment of tumors in the future.
Another interesting research topic is to reduce the computational time of the registration/fusion algorithm, or to speed up the fusion procedure. The relatively large computational time puts limitation of the implementation of several fusion algorithms in specific clinical applications based on the requirement of these medical studies. For example, as discussed in Section 5.1, for the real-time fusion of US and MRI or CT for liver lesion diagnosis, current clinical application can only use fusion results by a simple arithmetic combination due to its fast processing speed. Several research directions could be useful to overcome this challenge, such as (i) applying high performance computation, which utilizes parallel computing, to obtain efficient image fusion process; (ii) using a pretrained deep learning network for high speed image fusion processing.
As shown in Section 4.10, the deep learning method becomes more and more popular for the registration/fusion of medical images. However, the robustness and availability of the dataset still constrain the usage of such a method in clinical image fusion. For instance, the size of available data is often very limited due to the privacy of these clinical data. Secondly, the need for medical experts to label the available dataset is very time- and finance-consuming. At last, the quality of the data, especially pathological data, cannot be guaranteed. To solve the problem of data shortage, researchers have proposed and applied data augmentation in medical image processing, which increase the diversity and size of the dataset without obtaining new data. The simplest data augment function such as random image rotations or nonlinear deformations are easy to implement, but lack the ability to emulate real variations. The more advanced methods, such as few-shot/one-shot learning [187] and attribute-guided augmentation (AGA) [188], could produce a wide variety of realistic new images for the deep learning-based image fusions with little supervised data samples. Unsupervised learning is another important research direction to overcome the small dataset challenge. For instance, stack autoencoders is one of the popular feature extraction unsupervised learning algorithms used in the image registration/fusion process. A newly developed learning module, Spatial Transformer (ST) [189], can make explicit use of the data’s spatial information and can be inserted into CNNs. This makes CNNs invariant to translation, scaling, rotation and common distortions without additional training, and thus to be able to register medical images without training datasets. Another relative new and popular algorithm, Generative Adversarial Networks (GANs) [190], creates a generative network and a discriminative network at the same time. The network can receive end-to-end training and learn representative features in a completely unsupervised way which provides a research direction in deep learning-based image registration/fusion.
As discussed in the previous sections, all of the existing image fusion techniques possess their own strength or weakness. With efficient combination of different image fusion methods, the advantages of different fusion methods could be combined for higher image qualities, while the weakness of these methods could be avoided.
At the same time, a new and more efficient algorithm still needs to be developed to improve the quality and visualization of the fusion image, while reducing the errors due to resolution difference between images, image noise and the dimension difference between images. Noise effects due to signal noise could affect the image fusion process in a negative way. As a result, an efficient denoising algorithm would be useful to address the signal noise from the medical image in order to enhance the quality of fused images.
At last, many articles to date have documented case reports with only a small number of patients. Larger multicenter randomized studies, including cost–benefit analyses and clinical impact studies, are required to further evaluate the efficacy of medical fusion technology.

Author Contributions

Conceptualization, C.L.; methodology, C.L.; investigation, C.L.; data curation, C.L.; Writing—Original draft preparation, C.L.; Writing—Review and editing, A.Z. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Rinck, P. Magnetic resonance: A critical peer-reviewed introduction. In The Basic Textbook of the European Magnetic Resonance Forum; BoD: Norderstedt, Germany, 2014. [Google Scholar]
  2. Katti, G.; Ara, S.A.; Shireen, A. Magnetic resonance imaging (MRI)—A review. Int. J. Dent. Clin. 2011, 3, 65–70. [Google Scholar]
  3. Nuclear Medicine. Available online: https://www.nibib.nih.gov/science-education/science-topics/nuclear-medicine (accessed on 1 November 2019).
  4. Valk, P.E.; Bailey, D.L.; Townsend, D.W.; Maisey, M.N. Positron Emission Tomography; Springer: London, UK, 2003. [Google Scholar]
  5. Prince, J.L.; Links, J.M. Medical Imaging Signals and Systems; Pearson Prentice Hall: Upper Saddle River, NJ, USA, 2006. [Google Scholar]
  6. Hounsfield, G.N. Computed medical imaging. Med. Phys. 1980, 7, 283–290. [Google Scholar] [CrossRef] [PubMed]
  7. Maybody, M.; Stevenson, C.; Solomon, S.B. Overview of navigation systems in image-guided interventions. Tech. Vasc. Interv. Radiol. 2013, 16, 136–143. [Google Scholar] [CrossRef] [PubMed]
  8. Lee, M.W. Fusion imaging of real-time ultrasonography with CT or MRI for hepatic intervention. Ultrasonography 2014, 33, 227–239. [Google Scholar] [CrossRef] [PubMed]
  9. Lee, D.H.; Lee, J.M. Recent advances in the image-guided tumor ablation of liver malignancies: Radiofrequency ablation with multiple electrodes, real-time multimodality fusion imaging, and new energy sources. Korean J. Radiol. 2018, 19, 545–559. [Google Scholar] [CrossRef] [PubMed]
  10. Haskins, G.; Kruger, U.; Yan, P. Deep learning in medical image registration: A survey. arXiv 2019, arXiv:1903.02026. [Google Scholar] [CrossRef] [Green Version]
  11. Shen, D. Image registration by local histogram matching. Pattern Recognit. 2007, 40, 1161–1172. [Google Scholar] [CrossRef]
  12. Cheng, X.; Zhang, L.; Zheng, Y. Deep similarity learning for multimodal medical images. Comput. Methods Biomech. Biomed. Eng. Imaging Vis. 2018, 6, 248–252. [Google Scholar] [CrossRef]
  13. Ma, K.; Wang, J.; Singh, V.; Tamersoy, B.; Chang, Y.-J.; Wimmer, A.; Chen, T. Multimodal image registration with deep context reinforcement learning. In MICCAI 2017: Medical Image Computing and Computer Assisted Intervention—MICCAI 2017; Springer: Berlin/Heidelberg, Germany, 2017; pp. 240–248. [Google Scholar]
  14. Sedghi, A.; Luo, J.; Mehrtash, A.; Pieper, S.; Tempany, C.M.; Kapur, T.; Mousavi, P.; Wells, W.M., III. Semi-supervised deep metrics for image registration. arXiv 2018, arXiv:1804.01565. [Google Scholar]
  15. Haskins, G.; Kruecker, J.; Kruger, U.; Xu, S.; Pinto, P.A.; Wood, B.J.; Yan, P. Learning deep similarity metric for 3D MR–TRUS image registration. Int. J. Comput. Assist. Radiol. Surg. 2019, 14, 417–425. [Google Scholar] [CrossRef] [Green Version]
  16. Wright, R.; Khanal, B.; Gomez, A.; Skelton, E.; Matthew, J.; Hajnal, J.V.; Rueckert, D.; Schnabel, J.A. LSTM spatial co-transformer networks for registration of 3D fetal US and MR brain images. In Data Driven Treatment Response Assessment and Preterm, Perinatal, and Paediatric Image Analysis; Springer: Berlin/Heidelberg, Germany, 2018; pp. 149–159. [Google Scholar]
  17. Sun, Y.; Moelker, A.; Niessen, W.J.; van Walsum, T. Towards robust ct-ultrasound registration using deep learning methods. In Understanding and Interpreting Machine Learning in Medical Image Computing Applications; Springer: Berlin/Heidelberg, Germany, 2018; pp. 43–51. [Google Scholar]
  18. Hering, A.; Kuckertz, S.; Heldmann, S.; Heinrich, M.P. Enhancing label-driven deep deformable image registration with local distance metrics for state-of-the-art cardiac motion tracking. In Bildverarbeitung für die Medizin 2019; Springer: Berlin/Heidelberg, Germany, 2019; pp. 309–314. [Google Scholar]
  19. Hu, Y.; Gibson, E.; Ghavami, N.; Bonmati, E.; Moore, C.M.; Emberton, M.; Vercauteren, T.; Noble, J.A.; Barratt, D.C. Adversarial deformation regularization for training image registration neural networks. In Medical Image Computing and Computer Assisted Intervention—MICCAI 2018; Springer: Berlin/Heidelberg, Germany, 2018; pp. 774–782. [Google Scholar]
  20. Hu, Y.; Modat, M.; Gibson, E.; Li, W.; Ghavami, N.; Bonmati, E.; Wang, G.; Bandula, S.; Moore, C.M.; Emberton, M.; et al. Weakly-supervised convolutional neural networks for multimodal image registration. Med. Image Anal. 2018, 49, 1–13. [Google Scholar] [CrossRef] [PubMed]
  21. Yan, P.; Xu, S.; Rastinehad, A.R.; Wood, B.J. Adversarial image registration with application for MR and TRUS image fusion. In International Workshop on Machine Learning in Medical Imaging; Springer: Berlin/Heidelberg, Germany, 2018; pp. 197–204. [Google Scholar]
  22. Cao, X.; Yang, J.; Zhang, J.; Wang, Q.; Yap, P.-T.; Shen, D. Deformable image registration using a cue-aware deep regression network. IEEE Trans. Biomed. Eng. 2018, 65, 1900–1911. [Google Scholar] [CrossRef] [PubMed]
  23. Fan, J.; Cao, X.; Yap, P.-T.; Shen, D. BIRNet: Brain image registration using dual-supervised fully convolutional networks. Med. Image Anal. 2019, 54, 193–206. [Google Scholar] [CrossRef] [PubMed]
  24. Eppenhof, K.A.; Pluim, J.P. Pulmonary CT registration through supervised learning with convolutional neural networks. IEEE Trans. Med. Imaging 2018, 38, 1097–1105. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  25. Ito, M.; Ino, F. An Automated Method for Generating Training Sets for Deep Learning based Image Registration. In Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2018), Funchal, Madeira, Portugal, 19–21 January 2018; pp. 140–147. [Google Scholar]
  26. Balakrishnan, G.; Zhao, A.; Sabuncu, M.R.; Guttag, J.; Dalca, A.V. VoxelMorph: A learning framework for deformable medical image registration. IEEE Trans. Med. Imaging 2019, 38, 1788–1800. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  27. Cao, X.; Yang, J.; Wang, L.; Xue, Z.; Wang, Q.; Shen, D. Deep learning based inter-modality image registration supervised by intra-modality similarity. In International Workshop on Machine Learning in Medical Imaging; Springer: Berlin/Heidelberg, Germany, 2018; pp. 55–63. [Google Scholar]
  28. Ferrante, E.; Oktay, O.; Glocker, B.; Milone, D.H. On the adaptability of unsupervised CNN-based deformable image registration to unseen image domains. In International Workshop on Machine Learning in Medical Imaging; Springer: Berlin/Heidelberg, Germany, 2018; pp. 294–302. [Google Scholar]
  29. Liu, Q.; Leung, H. Tensor-based descriptor for image registration via unsupervised network. In Proceedings of the 2017 20th International Conference on Information Fusion (Fusion), Xi’an, China, 10–13 July 2017; pp. 1–7. [Google Scholar]
  30. Sun, L.; Zhang, S. Deformable mri-ultrasound registration using 3d convolutional neural network. In Simulation, Image Processing, and Ultrasound Systems for Assisted Diagnosis and Navigation; Springer: Berlin/Heidelberg, Germany, 2018; pp. 152–158. [Google Scholar]
  31. He, C.; Liu, Q.; Li, H.; Wang, H. Multimodal medical image fusion based on IHS and PCA. Procedia Eng. 2010, 7, 280–285. [Google Scholar] [CrossRef] [Green Version]
  32. Daneshvar, S.; Ghassemian, H. MRI and PET image fusion by combining IHS and retina-inspired models. Inf. Fusion 2010, 11, 114–123. [Google Scholar] [CrossRef]
  33. Alipour, S.H.M.; Houshyari, M.; Mostaar, A. A novel algorithm for PET and MRI fusion based on digital curvelet transform via extracting lesions on both images. Electron. Phys. 2017, 9, 4872–4879. [Google Scholar] [CrossRef] [Green Version]
  34. Haddadpour, M.; Daneshvar, S.; Seyedarabi, H. PET and MRI image fusion based on combination of 2-D Hilbert transform and IHS method. Biomed. J. 2017, 40, 219–225. [Google Scholar] [CrossRef]
  35. Yang, Y.; Que, Y.; Huang, S.; Lin, P. Multimodal sensor medical image fusion based on type-2 fuzzy logic in NSCT domain. IEEE Sens. J. 2016, 16, 3735–3745. [Google Scholar] [CrossRef]
  36. Ghassemian, H. A review of remote sensing image fusion methods. Inf. Fusion 2016, 32, 75–89. [Google Scholar] [CrossRef]
  37. Ho, P.-M.; Wong, T.-T.; Leung, C.-S. Compressing the illumination-adjustable images with principal component analysis. IEEE Trans. Circuits Syst. Video Technol. 2005, 15, 355–364. [Google Scholar]
  38. Fowler, J.E. Compressive-projection principal component analysis. IEEE Trans. Image Process. 2009, 18, 2230–2242. [Google Scholar] [CrossRef] [PubMed]
  39. Ng, S. Principal component analysis to reduce dimension on digital image. Procedia Comput. Sci. 2017, 111, 113–119. [Google Scholar] [CrossRef]
  40. Qiguang, M.; Baoshu, W. A novel image fusion algorithm using FRIT AND PCA. In Proceedings of the 2007 10th International Conference on Information Fusion, Québec City, QC, Canada, 9–12 July 2007; pp. 1–5. [Google Scholar]
  41. Yang, J.; Han, F.; Zhao, D. A block advanced PCA fusion algorithm based on PET/CT. In Proceedings of the 2011 Fourth International Conference on Intelligent Computation Technology and Automation, Shenzhen, China, 28–29 March 2011; Volume 2, pp. 925–928. [Google Scholar]
  42. Nandi, D.; Ashour, A.S.; Samanta, S.; Chakraborty, S.; Salem, M.A.; Dey, N. Principal component analysis in medical image processing: A study. Int. J. Image Min. 2015, 1, 65–86. [Google Scholar] [CrossRef]
  43. Vidhyavathi, R. Principal component analysis (PCA) in medical image processing using digital imaging and communications in medicine (DICOM) medical images. Int. J. Pharm. Biol. Sci 2017, 8, 598–606. [Google Scholar]
  44. Ouahabi, A. Signal and Image Multiresolution Analysis; John Wiley & Sons: Hoboken, NJ, USA, 2012. [Google Scholar]
  45. Ahmed, S.S.; Messali, Z.; Ouahabi, A.; Trepout, S.; Messaoudi, C.; Marco, S. Nonparametric denoising methods based on contourlet transform with sharp frequency localization: Application to low exposure time electron microscopy images. Entropy 2015, 17, 3461–3478. [Google Scholar] [CrossRef] [Green Version]
  46. Zhao, P.; Liu, G.; Hu, C.; Huang, H.; He, B. Medical image fusion algorithm based on the laplace-PCA. In Proceedings of the 2013 Chinese Intelligent Automation Conference, Yangzhou, China, 23–25 August 2013; pp. 787–794. [Google Scholar]
  47. Krishn, A.; Bhateja, V.; Sahu, A. Medical image fusion using combination of PCA and wavelet analysis. In Proceedings of the 2014 International Conference on Advances in Computing, Communications and Informatics (ICACCI), New Delhi, India, 24–27 September 2014; pp. 986–991. [Google Scholar]
  48. Bhateja, V.; Krishn, A.; Sahu, A. An improved medical image fusion approach using PCA and complex wavelets. In Proceedings of the 2014 International Conference on Medical Imaging, m-Health and Emerging Communication Systems (MedCom), Greater Noida, India, 7–8 November 2014; pp. 442–447. [Google Scholar]
  49. Benjamin, J.R.; Jayasree, T. Improved medical image fusion based on cascaded PCA and shift invariant wavelet transforms. Int. J. Comput. Assist. Radiol. Surg. 2018, 13, 229–240. [Google Scholar] [CrossRef]
  50. Mane, S.; Sawant, S. Image fusion of CT/MRI using DWT, PCA methods and analog DSP processor. Int. J. Eng. Res. Appl. 2014, 4, 557–563. [Google Scholar]
  51. Rajinikanth, V.; Satapathy, S.C.; Dey, N.; Vijayarajan, R. DWT-PCA image fusion technique to improve segmentation accuracy in brain tumor analysis. In Microelectronics, Electromagnetics and Telecommunications; Springer: Berlin/Heidelberg, Germany, 2018; pp. 453–462. [Google Scholar]
  52. Guruprasad, S.; Kurian, M.Z.; Suma, H.N. Medical Multi-Modality Image Fusion of Ct/Pet with Pca, Dwt Methods. J. Dent. Mater. Tech. 2013, 4, 677–681. [Google Scholar]
  53. Bhateja, V.; Krishn, A.; Sahu, A. Medical image fusion in curvelet domain employing PCA and maximum selection rule. In Proceedings of the Second International Conference on Computer and Communication Technologies; Springer: Berlin, Germany, 2016; pp. 1–9. [Google Scholar]
  54. Moin, A.; Bhateja, V.; Srivastava, A. Weighted-PCA based multimodal medical image fusion in contourlet domain. In Proceedings of the International Congress on Information and Communication Technology; Springer: Berlin/Heidelberg, Germany, 2016; pp. 597–605. [Google Scholar]
  55. Srivastava, A.; Bhateja, V.; Moin, A. Combination of PCA and contourlets for multispectral image fusion. In Proceedings of the International Conference on Data Engineering and Communication Technology; Springer: Berlin/Heidelberg, Germany, 2017; pp. 577–585. [Google Scholar]
  56. Al-Azzawi, N.; Wan Abdullah, W.A.K. Improved CT-MR image fusion scheme using dual tree complex contourlet transform based on PCA. Int. J. Inf. Acquis. 2010, 7, 99–107. [Google Scholar] [CrossRef]
  57. Amolins, K.; Zhang, Y.; Dare, P. Wavelet based image fusion techniques—An introduction, review and comparison. ISPRS J. Photogramm. Remote Sens. 2007, 62, 249–263. [Google Scholar] [CrossRef]
  58. Du, J.; Li, W.; Xiao, B.; Nawaz, Q. Union Laplacian pyramid with multiple features for medical image fusion. Neurocomputing 2016, 194, 326–339. [Google Scholar] [CrossRef]
  59. Sahu, A.; Bhateja, V.; Krishn, A. Medical image fusion with Laplacian pyramids. In Proceedings of the 2014 International Conference on Medical Imaging, m-Health and Emerging Communication Systems (MedCom), Greater Noida, India, 7–8 November 2014; pp. 448–453. [Google Scholar]
  60. Femmam, S.; M’Sirdi, N.; Ouahabi, A. Perception and characterization of materials using signal processing techniques. IEEE Trans. Instrum. Meas. 2001, 50, 1203–1211. [Google Scholar] [CrossRef]
  61. Qu, G.; Zhang, D.; Yan, P. Medical image fusion by wavelet transform modulus maxima. Opt. Express 2001, 9, 184–190. [Google Scholar]
  62. Singh, R.; Khare, A. Fusion of multimodal medical images using Daubechies complex wavelet transform–A multiresolution approach. Inf. Fusion 2014, 19, 49–60. [Google Scholar] [CrossRef]
  63. Zheng, Y.; Essock, E.A.; Hansen, B.C.; Haun, A.M. A new metric based on extended spatial frequency and its application to DWT based fusion algorithms. Inf. Fusion 2007, 8, 177–192. [Google Scholar] [CrossRef]
  64. Singh, R.; Vatsa, M.; Noore, A. Multimodal medical image fusion using redundant discrete wavelet transform. In Proceedings of the 2009 Seventh International Conference on Advances in Pattern Recognition, Kolkata, India, 4–6 February 2009; pp. 232–235. [Google Scholar]
  65. Xue-Jun, W.; Ying, M. A medical image fusion algorithm based on lifting wavelet transform. In Proceedings of the 2010 International Conference on Artificial Intelligence and Computational Intelligence, Sanya, China, 23–24 October 2010; Volume 3, pp. 474–476. [Google Scholar]
  66. Bhavana, V.; Krishnappa, H. Fusion of MRI and PET images using DWT and adaptive histogram equalization. In Proceedings of the 2016 International Conference on Communication and Signal Processing (ICCSP), Melmaruvathur, India, 6–8 April 2016; pp. 795–798. [Google Scholar]
  67. Bengueddoudj, A.; Messali, Z.; Mosorov, V. A novel image fusion algorithm based on 2D scale-mixing complex wavelet transform and Bayesian MAP estimation for multimodal medical images. J. Innov. Opt. Health Sci. 2017, 10. [Google Scholar] [CrossRef] [Green Version]
  68. Yang, Y.; Park, D.S.; Huang, S.; Yang, J. Fusion of CT and MR images using an improved wavelet based method. J. X-ray Sci. Technol. 2010, 18, 157–170. [Google Scholar] [CrossRef]
  69. Do, M.N.; Vetterli, M. Contourlets: A directional multiresolution image representation. In Proceedings of the International Conference on Image Processing, Rochester, NY, USA, 22–25 September 2002; Volume 1. [Google Scholar]
  70. Yang, L.; Guo, B.; Ni, W. Multimodality medical image fusion based on multiscale geometric analysis of contourlet transform. Neurocomputing 2008, 72, 203–211. [Google Scholar] [CrossRef]
  71. Huang, H.; Feng, X.; Jiang, J. Medical image fusion algorithm based on nonlinear approximation of contourlet transform and regional features. J. Electr. Comput. Eng. 2017, 2017. [Google Scholar] [CrossRef] [Green Version]
  72. Bhatnagar, G.; Wu, Q.J.; Liu, Z. Directive contrast based multimodal medical image fusion in NSCT domain. IEEE Trans. Multimed. 2013, 15, 1014–1024. [Google Scholar] [CrossRef]
  73. Shabanzade, F.; Ghassemian, H. Combination of wavelet and contourlet transforms for PET and MRI image fusion. In Proceedings of the 2017 Artificial Intelligence and Signal Processing Conference (AISP), Shiraz, Iran, 25–27 October 2017; pp. 178–183. [Google Scholar]
  74. Bindu, C.H.; Prasad, K.S. MRI–PET Medical Image Fusion Technique by Combining Contourlet and Wavelet Transform. In Proceedings of the Third International Conference on Trends in Information, Telecommunication and Computing, Kochi, India, 3–4 August 2013; pp. 145–151. [Google Scholar]
  75. Starck, J.-L.; Candès, E.J.; Donoho, D.L. The curvelet transform for image denoising. IEEE Trans. Image Process. 2002, 11, 670–684. [Google Scholar] [CrossRef] [Green Version]
  76. Raju, C.; Reddy, T.S.; Sivasubramanyam, M. Denoising of remotely sensed images via curvelet transform and its relative assessment. Procedia Comput. Sci. 2016, 89, 771–777. [Google Scholar] [CrossRef] [Green Version]
  77. Zhang, B.; Fadili, J.M.; Starck, J.-L. Wavelets, ridgelets, and curvelets for Poisson noise removal. IEEE Trans. Image Process. 2008, 17, 1093–1108. [Google Scholar] [CrossRef] [Green Version]
  78. Ma, J.; Ma, Y.; Li, C. Infrared and visible image fusion methods and applications: A survey. Inf. Fusion 2019, 45, 153–178. [Google Scholar] [CrossRef]
  79. Ali, F.E.; El-Dokany, I.; Saad, A.; Abd El-Samie, F. A curvelet transform approach for the fusion of MR and CT images. J. Mod. Opt. 2010, 57, 273–286. [Google Scholar] [CrossRef]
  80. Wang, L.; Li, B.; Tian, L.-F. Multi-modal medical image fusion using the inter-scale and intra-scale dependencies between image shift-invariant shearlet coefficients. Inf. Fusion 2014, 19, 20–28. [Google Scholar] [CrossRef]
  81. Guo, K.; Labate, D.; Lim, W.-Q.; Weiss, G.; Wilson, E. Wavelets with composite dilations and their MRA properties. Appl. Comput. Harmon. Anal. 2006, 20, 202–236. [Google Scholar] [CrossRef] [Green Version]
  82. Miao, Q.; Shi, C.; Xu, P.; Yang, M.; Shi, Y. A novel algorithm of image fusion using shearlets. Opt. Commun. 2011, 284, 1540–1547. [Google Scholar] [CrossRef]
  83. Wang, L.; Li, B.; Tian, L. Multimodal medical volumetric data fusion using 3-D discrete shearlet transform and global-to-local rule. IEEE Trans. Biomed. Eng. 2013, 61, 197–206. [Google Scholar] [CrossRef] [PubMed]
  84. Xia, J.; Chen, Y.; Chen, A.; Chen, Y. Medical image fusion based on sparse representation and PCNN in NSCT domain. Comput. Math. Methods Med. 2018, 2018. [Google Scholar] [CrossRef] [PubMed]
  85. Gupta, D. Nonsubsampled shearlet domain fusion techniques for CT–MR neurological images using improved biological inspired neural model. Biocybern. Biomed. Eng. 2018, 38, 262–274. [Google Scholar] [CrossRef]
  86. Liu, X.; Mei, W.; Du, H. Structure tensor and nonsubsampled shearlet transform based algorithm for CT and MRI image fusion. Neurocomputing 2017, 235, 131–139. [Google Scholar] [CrossRef]
  87. Wang, Z.; Ma, Y.; Gu, J. Multi-focus image fusion using PCNN. Pattern Recognit. 2010, 43, 2003–2016. [Google Scholar] [CrossRef]
  88. Wang, Z.; Wang, S.; Zhu, Y.; Ma, Y. Review of image fusion based on pulse-coupled neural network. Arch. Comput. Methods Eng. 2016, 23, 659–671. [Google Scholar] [CrossRef]
  89. Xiao-Bo, Q.; Jing-Wen, Y.; Hong-Zhi, X.; Zi-Qian, Z. Image fusion algorithm based on spatial frequency-motivated pulse coupled neural networks in nonsubsampled contourlet transform domain. Acta Autom. Sin. 2008, 34, 1508–1514. [Google Scholar]
  90. Wang, Z.; Ma, Y. Medical image fusion using m-PCNN. Inf. Fusion 2008, 9, 176–185. [Google Scholar] [CrossRef]
  91. Huang, C.; Tian, G.; Lan, Y.; Hao, Y.; Cheng, Y.; Peng, Y.; Che, W. A new Pulse Coupled Neural Network (PCNN) for Brain Medical Image Fusion empowered by Shuffled Frog Leaping. Front. Neurosci. 2019, 13. [Google Scholar] [CrossRef]
  92. Xiang, T.; Yan, L.; Gao, R. A fusion algorithm for infrared and visible images based on adaptive dual-channel unit-linking PCNN in NSCT domain. Infrared Phys. Technol. 2015, 69, 53–61. [Google Scholar] [CrossRef]
  93. Yuan, Y.; Jiang, M.; Gao, W. Image fusion based on MPCNN and DWT in PCB failure detection. Comput. Model. New Technol. 2014, 18, 128–132. [Google Scholar]
  94. Liu, Z.; Yin, H.; Chai, Y.; Yang, S.X. A novel approach for multimodal medical image fusion. Expert Syst. Appl. 2014, 41, 7425–7435. [Google Scholar] [CrossRef]
  95. Zou, B.; Wang, M.; Zhang, J.; Zhang, L.; Zhang, Y. Improving spatial resolution for CHANG’E-1 imagery using ARSIS concept and Pulse Coupled Neural Networks. In Proceedings of the 2012 19th IEEE International Conference on Image Processing, Orlando, FL, USA, 30 September–3 October 2012; pp. 2125–2128. [Google Scholar]
  96. Feng, W.; Bao, W. A new technology of remote sensing image fusion. Telkomnika 2012, 10, 551–556. [Google Scholar] [CrossRef] [Green Version]
  97. Lin, Y.; Song, L.; Zhou, X.; Huang, Y. Infrared and visible image fusion algorithm based on Contourlet transform and PCNN. In Proceedings of the Infrared Materials, Devices, and Applications, International Society for Optics and Photonics, Beijing, China, 11–15 November 2007; Volume 6835, p. 683514. [Google Scholar]
  98. Yang, S.; Wang, M.; Jiao, L.; Wu, R.; Wang, Z. Image fusion based on a new contourlet packet. Inf. Fusion 2010, 11, 78–84. [Google Scholar] [CrossRef]
  99. Yang, S.; Wang, M.; Jiao, L. Contourlet hidden Markov Tree and clarity-saliency driven PCNN based remote sensing images fusion. Appl. Soft Comput. 2012, 12, 228–237. [Google Scholar] [CrossRef]
  100. Wang, X.; Chen, L. Image fusion algorithm based on spatial frequency-motivated pulse coupled neural networks in wavelet based contourlet transform domain. In Proceedings of the 2010 the 2nd Conference on Environmental Science and Information Application Technology, Wuhan, China, 17–18 July 2010; Volume 2, pp. 411–414. [Google Scholar]
  101. Wang, N.; Ma, Y.; Zhan, K.; Yuan, M. Multimodal Medical Image Fusion Framework Based on Simplified PCNN in Nonsubsampled Contourlet Transform Domain. J. Multimed. 2013, 8, 270–276. [Google Scholar] [CrossRef]
  102. Wang, N.; Ma, Y.; Wang, W.; Zhou, S. An image fusion method based on NSCT and dual-channel PCNN model. J. Netw. 2014, 9, 501–506. [Google Scholar] [CrossRef] [Green Version]
  103. Wang, N.Y.; Wang, W.L.; Guo, X.R. A new image fusion method based on improved PCNN and multiscale decomposition. Adv. Mater. Res. 2014, 834, 1011–1015. [Google Scholar] [CrossRef]
  104. Ma, L.J.; Zhao, C.H. An effective image fusion method based on nonsubsampled contourlet transform and pulse coupled neural network. Adv. Mater. Res. 2013, 756, 3542–3548. [Google Scholar] [CrossRef]
  105. Xu, L.; Du, J.; Li, Q. Image fusion based on nonsubsampled contourlet transform and saliency-motivated pulse coupled neural networks. Math. Probl. Eng. 2013, 2013. [Google Scholar] [CrossRef]
  106. Geng, P.; Wang, Z.; Zhang, Z.; Xiao, Z. Image fusion by pulse couple neural network with Shearlet. Opt. Eng. 2012, 51. [Google Scholar] [CrossRef]
  107. Geng, P.; Zheng, X.; Zhang, Z.-G.; Shi, Y.-J.; Yan, S.-Q. Multifocus image fusion with PCNN in Shearlet domain. Res. J. Appl. Sci. Eng. Technol. 2012, 4, 2283–2290. [Google Scholar]
  108. Cheng, S.; Qiguang, M.; Pengfei, X. A novel algorithm of remote sensing image fusion based on Shearlets and PCNN. Neurocomputing 2013, 117, 47–53. [Google Scholar] [CrossRef]
  109. Ma, Y.; Zhai, Y.; Geng, P.; Yan, P. A novel algorithm of image fusion based on PCNN and Shearlet. Int. J. Digit. Content Technol. Its Appl. 2011, 5, 347–354. [Google Scholar]
  110. Kong, W.; Zhang, L.; Lei, Y. Novel fusion method for visible light and infrared images based on NSST–SF–PCNN. Infrared Phys. Technol. 2014, 65, 103–112. [Google Scholar] [CrossRef]
  111. Das, S.; Kundu, M.K. NSCT-based multimodal medical image fusion using pulse-coupled neural network and modified spatial frequency. Med. Biol. Eng. Comput. 2012, 50, 1105–1114. [Google Scholar] [CrossRef]
  112. Nobariyan, B.K.; Daneshvar, S.; Foroughi, A. A new MRI and PET image fusion algorithm based on pulse coupled neural network. In Proceedings of the 2014 22nd Iranian Conference on Electrical Engineering (ICEE), Tehran, Iran, 20–22 May 2014; pp. 1950–1955. [Google Scholar]
  113. Zhao, Y.; Zhao, Q.; Hao, A. Multimodal medical image fusion using improved multi-channel PCNN. Biomed. Mater. Eng. 2014, 24, 221–228. [Google Scholar] [CrossRef]
  114. Zhang, H.; Luo, X.; Wu, X.; Zhang, Z. Statistical modeling of multi-modal medical image fusion method using C-CHMM and M-PCNN. In Proceedings of the 2014 22nd International Conference on Pattern Recognition, Stockholm, Sweden, 24–28 August 2014; pp. 1067–1072. [Google Scholar]
  115. Javed, U.; Riaz, M.M.; Ghafoor, A.; Ali, S.S.; Cheema, T.A. MRI and PET image fusion using fuzzy logic and image local features. Sci. World J. 2014, 2014. [Google Scholar] [CrossRef]
  116. Singh, H.; Raj, J.; Kaur, G.; Meitzler, T. Image fusion using fuzzy logic and applications. In Proceedings of the 2004 IEEE International Conference on Fuzzy Systems (IEEE Cat. No. 04CH37542), Budapest, Hungary, 25–29 July 2004; Volume 1, pp. 337–340. [Google Scholar]
  117. Myna, A.; Prakash, J. Fusion of CT and MRI Images Based on Fuzzy Logic and Discrete Wavelet Transform. Int. J. Comput. Sci. Inf. Technol. 2015, 6, 4512–4519. [Google Scholar]
  118. Jiang, Q.; Jin, X.; Hou, J.; Lee, S.-J.; Yao, S. Multi-sensor image fusion based on Interval Type-2 Fuzzy Sets and regional features in Nonsubsampled Shearlet Transform domain. IEEE Sens. J. 2018, 18, 2494–2505. [Google Scholar] [CrossRef]
  119. Zhou, F.; Jia, Z.; Yang, J.; Kasabov, N. Method of improved fuzzy contrast combined adaptive threshold in NSCT for medical image enhancement. BioMed Res. Int. 2017, 2017. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  120. Wang, B.; Zeng, J.; Lin, S.; Bai, G. Multi-band images synchronous fusion based on NSST and fuzzy logical inference. Infrared Phys. Technol. 2019, 98, 94–107. [Google Scholar] [CrossRef]
  121. Rao, D.S.; Seetha, M.; Hazarath, M. Iterative image fusion using neuro fuzzy logic and applications. In Proceedings of the 2012 International Conference on Machine Vision and Image Processing (MVIP), Taipei, China, 14–15 December 2012; pp. 121–124. [Google Scholar]
  122. Rao, D.S.; Seetha, M.; Prasad, M.K. Quality assessment parameters for iterative image fusion using fuzzy and neuro fuzzy logic and applications. Procedia Technol. 2015, 19, 888–894. [Google Scholar] [CrossRef] [Green Version]
  123. Devulapalli, S.; Krishnan, R. Synthesized pansharpening using curvelet transform and adaptive neuro-fuzzy inference system. J. Appl. Remote Sens. 2019, 13. [Google Scholar] [CrossRef]
  124. Hermessi, H.; Mourali, O.; Zagrouba, E. Multimodal image fusion based on non-subsampled Shearlet transform and neuro-fuzzy. In International Workshop on Representations, Analysis and Recognition of Shape and Motion FroM Imaging Data; Springer: Berlin/Heidelberg, Germany, 2016; pp. 161–175. [Google Scholar]
  125. Das, S.; Kundu, M.K. A neuro-fuzzy approach for medical image fusion. IEEE Trans. Biomed. Eng. 2013, 60, 3347–3353. [Google Scholar] [CrossRef]
  126. Aishwarya, N.; Thangammal, C.B. An image fusion framework using novel dictionary based sparse representation. Multimed. Tools Appl. 2017, 76, 21869–21888. [Google Scholar] [CrossRef]
  127. Chang, L.; Feng, X.; Zhang, R.; Huang, H.; Wang, W.; Xu, C. Image decomposition fusion method based on sparse representation and neural network. Appl. Opt. 2017, 56, 7969–7977. [Google Scholar] [CrossRef]
  128. Gao, Z.; Yang, M.; Xie, C. Space target image fusion method based on image clarity criterion. Opt. Eng. 2017, 56. [Google Scholar] [CrossRef]
  129. Wang, K.; Qi, G.; Zhu, Z.; Chai, Y. A novel geometric dictionary construction approach for sparse representation based image fusion. Entropy 2017, 19, 306. [Google Scholar] [CrossRef] [Green Version]
  130. Liu, C.; Qi, Y.; Ding, W. Infrared and visible image fusion method based on saliency detection in sparse domain. Infrared Phys. Technol. 2017, 83, 94–102. [Google Scholar] [CrossRef]
  131. Kim, M.; Han, D.K.; Ko, H. Joint patch clustering-based dictionary learning for multimodal image fusion. Inf. fusion 2016, 27, 198–214. [Google Scholar] [CrossRef]
  132. Zhu, Z.; Yin, H.; Chai, Y.; Li, Y.; Qi, G. A novel multi-modality image fusion method based on image decomposition and sparse representation. Inf. Sci. 2018, 432, 516–529. [Google Scholar] [CrossRef]
  133. Liu, Y.; Wang, Z. Simultaneous image fusion and denoising with adaptive sparse representation. IET Image Process. 2014, 9, 347–357. [Google Scholar] [CrossRef] [Green Version]
  134. Davenport, M. The Fundamentals of Compressive Sensing; Georgia Institute of Technology School of Electrical and Computer Engineering: Atlanta, GA, USA, 2013. [Google Scholar]
  135. Han, J. The Applications of Compressive Sensing in Multi-Modal Images. Ph.D. Thesis, Faculty of Science and Technology, University of Siegen, Siegen, Germany, 2015. [Google Scholar]
  136. Deng, H.; Wang, C.; Hu, Y.; Zhang, Y. Image fusion methods based on compressed sensing: Theory and application. In Proceedings of the Eleventh International Conference on Digital Image Processing (ICDIP 2019), International Society for Optics and Photonics, Zurich, Switzerland, 30–31 March 2019; Volume 11179, p. 111790S. [Google Scholar]
  137. Deka, B. Sparse representations and compressive sensing in multi-dimensional signal processing. CSI Trans. ICT 2019, 7, 233–242. [Google Scholar] [CrossRef]
  138. Patel, V.M.; Chellappa, R. Sparse representations, compressive sensing and dictionaries for pattern recognition. In Proceedings of the First Asian Conference on Pattern Recognition, Beijing, China, 28 November 2011; pp. 325–329. [Google Scholar]
  139. Srinivas, M.; Naidu, R.R.; Sastry, C.S.; Mohan, C.K. Content based medical image retrieval using dictionary learning. Neurocomputing 2015, 168, 880–895. [Google Scholar] [CrossRef]
  140. Li, H.; He, X.; Tao, D.; Tang, Y.; Wang, R. Joint medical image fusion, denoising and enhancement via discriminative low-rank sparse dictionaries learning. Pattern Recognit. 2018, 79, 130–146. [Google Scholar] [CrossRef]
  141. Li, S.; Yin, H.; Fang, L. Group-sparse representation with dictionary learning for medical image denoising and fusion. IEEE Trans. Biomed. Eng. 2012, 59, 3450–3459. [Google Scholar] [CrossRef]
  142. Wang, Z.; Ma, Y. Dual-channel PCNN and its application in the field of image fusion. In Proceedings of the Third International Conference on Natural Computation (ICNC 2007), Haikou, China, 24–27 August 2007; Volume 1, pp. 755–759. [Google Scholar]
  143. Eisemann, E.; Durand, F. Flash photography enhancement via intrinsic relighting. ACM Trans. Graph. 2004, 23, 673–678. [Google Scholar] [CrossRef]
  144. Liu, Y.; Chen, X.; Peng, H.; Wang, Z. Multi-focus image fusion with a deep convolutional neural network. Inf. Fusion 2017, 36, 191–207. [Google Scholar] [CrossRef]
  145. Sultana, S.; Robinson, A.; Song, D.; Lee, J. Multimodal Image Fusion for Prostate Biopsy and Focal Brachytherapy with Convolutional Neural Network Autosegmentation and Anatomical Landmark-Based Registration. Med. Phys. 2019, 46, E503. [Google Scholar]
  146. Xia, K.; Yin, H.; Wang, J. A novel improved deep convolutional neural network model for medical image fusion. Clust. Comput. 2019, 22, 1515–1527. [Google Scholar] [CrossRef]
  147. Kumar, A.; Fulham, M.; Feng, D.; Kim, J. Co-Learning Feature Fusion Maps from PET-CT Images of Lung Cancer. IEEE Trans. Med. Imaging 2019, 39, 204–217. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  148. Liu, N.; Song, Y. A novel super-resolution image fusion algorithm based on improved PCNN and wavelet transform. Int. Soc. Opt. Photonics 2009. [Google Scholar] [CrossRef]
  149. Hou, R.; Zhou, D.; Nie, R.; Liu, D.; Ruan, X. Brain CT and MRI medical image fusion using convolutional neural networks and a dual-channel spiking cortical model. Med. Biol. Eng. Comput. 2019, 57, 887–900. [Google Scholar] [CrossRef] [PubMed]
  150. Liang, X.; Hu, P.; Zhang, L.; Sun, J.; Yin, G. MCFNet: Multi-Layer Concatenation Fusion Network for Medical Images Fusion. IEEE Sens. J. 2019, 19, 7107–7119. [Google Scholar] [CrossRef]
  151. Yang, Z.; Chen, Y.; Le, Z.; Fan, F.; Pan, E. Multi-Source Medical Image Fusion Based on Wasserstein Generative Adversarial Networks. IEEE Access 2019, 7, 175947–175958. [Google Scholar] [CrossRef]
  152. Zeiler, M.D.; Krishnan, D.; Taylor, G.W.; Fergus, R. Deconvolutional networks. In Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, USA, 13–18 June 2010; pp. 2528–2535. [Google Scholar]
  153. Liu, Y.; Chen, X.; Ward, R.K.; Wang, Z.J. Image fusion with convolutional sparse representation. IEEE Signal Process. Lett. 2016, 23, 1882–1886. [Google Scholar] [CrossRef]
  154. Rajalingam, B.; Priya, R. Multimodal medical image fusion based on deep learning neural network for clinical treatment analysis. Int. J. ChemTech Res. 2018, 0974–4290. [Google Scholar]
  155. Wohlberg, B. Efficient algorithms for convolutional sparse representations. IEEE Trans. Image Process. 2015, 25, 301–315. [Google Scholar] [CrossRef]
  156. Qiu, C.; Peng, W.; Wang, Y.; Hong, J.; Xia, S. Fusion of mis-registered GFP and phase contrast images with convolutional sparse representation and adaptive region energy rule. Microsc. Res. Tech. 2019. [Google Scholar] [CrossRef]
  157. Lei, Y.; Yuan, W.; Wang, H.; Wenhu, Y.; Bo, W. A skin segmentation algorithm based on stacked autoencoders. IEEE Trans. Multimed. 2016, 19, 740–749. [Google Scholar] [CrossRef]
  158. Zhang, X.; Chen, G.; Wang, W.; Wang, Q.; Dai, F. Object-based land-cover supervised classification for very-high-resolution UAV images using stacked denoising autoencoders. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2017, 10, 3373–3385. [Google Scholar] [CrossRef]
  159. Zhang, Y.; Liu, Y.; Sun, P.; Yan, H.; Zhao, X.; Zhang, L. IFCNN: A general image fusion framework based on convolutional neural network. Inf. Fusion 2020, 54, 99–118. [Google Scholar] [CrossRef]
  160. Ferroukhi, M.; Ouahabi, A.; Attari, M.; Habchi, Y.; Taleb-Ahmed, A. Medical video coding based on 2nd-generation wavelets: Performance evaluation. Electronics 2019, 8, 88. [Google Scholar] [CrossRef] [Green Version]
  161. Anantrasirichai, N.; Achim, A.; Bull, D. Literature Review of Image Fusion. 2012. Available online: https://seis.bristol.ac.uk/~eexna/papers/ReviewFusion.pdf (accessed on 1 November 2019).
  162. Minami, Y.; Kudo, M. Imaging modalities for assessment of treatment response to nonsurgical hepatocellular carcinoma therapy: Contrast-enhanced US, CT, and MRI. Liver Cancer 2015, 4, 106–114. [Google Scholar] [CrossRef]
  163. Seki, T.; Tamai, T.; Nakagawa, T.; Imamura, M.; Nishimura, A.; Yamashiki, N.; Ikeda, K.; Inoue, K. Combination therapy with transcatheter arterial chemoembolization and percutaneous microwave coagulation therapy for hepatocellular carcinoma. Cancer Interdiscip. Int. J. Am. Cancer Soc. 2000, 89, 1245–1251. [Google Scholar] [CrossRef]
  164. Xu, E.; Long, Y.; Li, K.; Zeng, Q.; Tan, L.; Luo, L.; Huang, Q.; Zheng, R. Comparison of CT/MRI-CEUS and US-CEUS fusion imaging techniques in the assessment of the thermal ablation of liver tumors. Int. J. Hyperth. 2018, 35, 159–167. [Google Scholar] [CrossRef] [Green Version]
  165. Teixera, S.R.; Kohan, A.A.; Paspulati, R.M.; Rong, R.; Herrmann, K.A. Potential role of positron emission tomography/magnetic resonance imaging in gastrointestinal and abdominal malignancies: Preliminary experience. Semin. Roentgenol. 2014, 49, 321–333. [Google Scholar] [CrossRef]
  166. Kitajima, K.; Suenaga, Y.; Ueno, Y.; Kanda, T.; Maeda, T.; Makihara, N.; Ebina, Y.; Yamada, H.; Takahashi, S.; Sugimura, K. Value of fusion of PET and MRI in the detection of intra-pelvic recurrence of gynecological tumor: Comparison with 18 F-FDG contrast-enhanced PET/CT and pelvic MRI. Ann. Nucl. Med. 2014, 28, 25–32. [Google Scholar] [CrossRef] [Green Version]
  167. Jadvar, H.; Colletti, P.M. Competitive advantage of PET/MRI. Eur. J. Radiol. 2014, 83, 84–94. [Google Scholar] [CrossRef] [Green Version]
  168. Hallqvist, A.; Alverbratt, C.; Strandell, A.; Samuelsson, O.; Björkander, E.; Liljegren, A.; Albertsson, P. Positron emission tomography and computed tomographic imaging (PET/CT) for dose planning purposes of thoracic radiation with curative intent in lung cancer patients: A systematic review and meta-analysis. Radiother. Oncol. 2017, 123, 71–77. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  169. Liu, Y.; Yang, J.; Sun, J. PET/CT medical image fusion algorithm based on multiwavelet transform. In Proceedings of the 2010 2nd International Conference on Advanced Computer Control, Shenyang, China, 27–29 March 2010; Volume 2, pp. 264–268. [Google Scholar]
  170. Yang, M.; Ding, H.; Kang, J.; Cong, L.; Zhu, L.; Wang, G. Local structure orientation descriptor based on intra-image similarity for multimodal registration of liver ultrasound and MR images. Comput. Biol. Med. 2016, 76, 69–79. [Google Scholar] [CrossRef] [PubMed]
  171. Jung, E.; Schreyer, A.; Schacherer, D.; Menzel, C.; Farkas, S.; Loss, M.; Feuerbach, S.; Zorger, N.; Fellner, C. New real-time image fusion technique for characterization of tumor vascularisation and tumor perfusion of liver tumors with contrast-enhanced ultrasound, spiral CT or MRI: First results. Clin. Hemorheol. Microcirc. 2009, 43, 57–69. [Google Scholar] [CrossRef]
  172. Lee, M.W.; Kim, Y.J.; Park, H.S.; Yu, N.C.; Jung, S.I.; Ko, S.Y.; Jeon, H.J. Targeted sonography for small hepatocellular carcinoma discovered by CT or MRI: Factors affecting sonographic detection. Am. J. Roentgenol. 2010, 194, W396–W400. [Google Scholar] [CrossRef]
  173. Lencioni, R.; Cioni, D.; Crocetti, L.; Franchini, C.; Pina, C.D.; Lera, J.; Bartolozzi, C. Early-stage hepatocellular carcinoma in patients with cirrhosis: Long-term results of percutaneous image-guided radiofrequency ablation. Radiology 2005, 234, 961–967. [Google Scholar] [CrossRef]
  174. Bruix, J.; Sherman, M. Management of hepatocellular carcinoma: An update. Hepatology 2011, 53, 1020–1022. [Google Scholar] [CrossRef] [PubMed]
  175. Imbriaco, M.; De Luca, S.; Coppola, M.; Fusari, M.; Klain, M.; Puglia, M.; Mainenti, P.; Liuzzi, R.; Maurea, S. Diagnostic accuracy of Gd-EOB-DTPA for Detection Hepatocellular Carcinoma (HCC): A comparative study with dynamic contrast enhanced Magnetic Resonance Imaging (MRI) and dynamic contrast enhanced Computed Tomography (CT). Pol. J. Radiol. 2017, 82, 50–57. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  176. Inoue, Y.; Arita, J.; Sakamoto, T.; Ono, Y.; Takahashi, M.; Takahashi, Y.; Kokudo, N.; Saiura, A. Anatomical liver resections guided by 3-dimensional parenchymal staining using fusion indocyanine green fluorescence imaging. Ann. Surg. 2015, 262, 105–111. [Google Scholar] [CrossRef] [PubMed]
  177. Paolucci, I.; Sandu, R.-M.; Tinguely, P.M.P.; Kim-Fuchs, C.; Maurer, M.; Candinas, D.; Weber, S.; Lachenmayer, A. Stereotactic Image-Guidance for Ablation of Malignant Liver Tumors. IntechOpen 2019. [Google Scholar] [CrossRef] [Green Version]
  178. Katz, S.C.; Shia, J.; Liau, K.H.; Gonen, M.; Ruo, L.; Jarnagin, W.R.; Fong, Y.; D’Angelica, M.I.; Blumgart, L.H.; DeMatteo, R.P. Operative blood loss independently predicts recurrence and survival after resection of hepatocellular carcinoma. Ann. Surg. 2009, 249, 617–623. [Google Scholar] [CrossRef]
  179. Sauer, I.M.; Queisner, M.; Tang, P.; Moosburner, S.; Hoepfner, O.; Horner, R.; Lohmann, R.; Pratschke, J. Mixed reality in visceral surgery: Development of a suitable workflow and evaluation of intraoperative use-cases. Ann. Surg. 2017, 266, 706–712. [Google Scholar] [CrossRef]
  180. Chen, C.; Xu, L.; Wang, Y.; Wang, Y.; Li, G.; Huang, H.; Wang, B.; Li, W.; He, X. Assessment of the cryoablation margin using MRI–CT fusion imaging in hepatic malignancies. Clin. Radiol. 2019, 74. [Google Scholar] [CrossRef]
  181. Nensa, F.; Beiderwellen, K.; Heusch, P.; Wetter, A. Clinical applications of PET/MRI: Current status and future perspectives. Diagn. Interv. Radiol. 2014, 20, 438–447. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  182. Reiner, C.S.; Stolzmann, P.; Husmann, L.; Burger, I.A.; Hüllner, M.W.; Schaefer, N.G.; Schneider, P.M.; von Schulthess, G.K.; Veit-Haibach, P. Protocol requirements and diagnostic value of PET/MR imaging for liver metastasis detection. Eur. J. Nucl. Med. Mol. Imaging 2014, 41, 649–658. [Google Scholar] [CrossRef] [Green Version]
  183. Kirchner, J.; Sawicki, L.M.; Deuschl, C.; Grüneisen, J.; Beiderwellen, K.; Lauenstein, T.C.; Herrmann, K.; Forsting, M.; Heusch, P.; Umutlu, L. 18 F-FDG PET/MR imaging in patients with suspected liver lesions: Value of liver-specific contrast agent Gadobenate dimeglumine. PLoS ONE 2017, 12. [Google Scholar] [CrossRef] [PubMed]
  184. Li, C.; Li, R.; Zhang, W. Progress in non-invasive detection of liver fibrosis. Cancer Biol. Med. 2018, 15, 124. [Google Scholar] [PubMed] [Green Version]
  185. Vranic, J.; Cross, N.; Wang, Y.; Hippe, D.; de Weerdt, E.; Mossa-Basha, M. Compressed sensing–sensitivity encoding (CS-SENSE) accelerated brain imaging: Reduced scan time without reduced image quality. Am. J. Neuroradiol. 2019, 40, 92–98. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  186. Bilgic, B.; Goyal, V.K.; Adalsteinsson, E. Multi-contrast reconstruction with Bayesian compressed sensing. Magn. Reson. Med. 2011, 66, 1601–1615. [Google Scholar] [CrossRef] [Green Version]
  187. Zhao, A.; Balakrishnan, G.; Durand, F.; Guttag, J.V.; Dalca, A.V. Data augmentation using learned transformations for one-shot medical image segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 8543–8553. [Google Scholar]
  188. Dixit, M.; Kwitt, R.; Niethammer, M.; Vasconcelos, N. Aga: Attribute-guided augmentation. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 7455–7463. [Google Scholar]
  189. Jaderberg, M.; Simonyan, K.; Zisserman, A. Spatial transformer networks. In Proceedings of the conference on advances in Neural Information Processing Systems, Montreal, QC, Canada, 7–12 December 2015; pp. 2017–2025. [Google Scholar]
  190. Radford, A.; Metz, L.; Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv 2015, arXiv:1511.06434. [Google Scholar]
Figure 1. Representation of an image pyramid with four levels.
Figure 1. Representation of an image pyramid with four levels.
Applsci 10 01171 g001
Figure 2. Representation of an image pyramid with four levels [89].
Figure 2. Representation of an image pyramid with four levels [89].
Applsci 10 01171 g002
Figure 3. A 51-year-old male. (a,b) Contrast-enhanced computed tomography (CT) indicates a liver tumor located in segment 7. Arterial enhancement and delay washout were observed. (c) A hypoechoic lesion in Ultrasonography (US). (d) Real-time US was successfully matched with immediately acquired pre-ablation 3DUS. (e) Real-time US was matched with pre-ablation CT volume images successfully. (f) Ablation was applied under the guidance and monitoring of US. (g) Contrast-enhanced ultrasonography (CEUS) was performed and fused with pre-ablation 3DUS. The ablation zone completely covered the target tumor and the 5-mm ablative margin. (h) CEUS fused with pre-ablation CT indicated the ablation zone completely covered the target tumor and the 5-mm ablative margin. (i) Subsequent contrast-enhanced CT within three months confirmed that the tumor had been completely ablated [164].
Figure 3. A 51-year-old male. (a,b) Contrast-enhanced computed tomography (CT) indicates a liver tumor located in segment 7. Arterial enhancement and delay washout were observed. (c) A hypoechoic lesion in Ultrasonography (US). (d) Real-time US was successfully matched with immediately acquired pre-ablation 3DUS. (e) Real-time US was matched with pre-ablation CT volume images successfully. (f) Ablation was applied under the guidance and monitoring of US. (g) Contrast-enhanced ultrasonography (CEUS) was performed and fused with pre-ablation 3DUS. The ablation zone completely covered the target tumor and the 5-mm ablative margin. (h) CEUS fused with pre-ablation CT indicated the ablation zone completely covered the target tumor and the 5-mm ablative margin. (i) Subsequent contrast-enhanced CT within three months confirmed that the tumor had been completely ablated [164].
Applsci 10 01171 g003
Figure 4. A 59-year-old man with hepatocellular carcinoma (HCC) following radical resection, who was treated with percutaneous cryoablation. (a) A single tumor measuring 0.8 cm is noted in segment 5 on the portal phase of contrast-enhanced magnetic resonance image (MRI) before cryoablation. (b) The intraoperative CT image at the end of cryoablation shows a clear ablation zone. (c) The fusion image is created after automatic rigid registration combined with manual correction. The medication-assisted manipulation (MAM) is categorized as group II: MAM 0e5 mm. (d) Contrast-enhanced MRI image obtained 4 months after cryoablation shows Local Tumor Progression (LTP) [180].
Figure 4. A 59-year-old man with hepatocellular carcinoma (HCC) following radical resection, who was treated with percutaneous cryoablation. (a) A single tumor measuring 0.8 cm is noted in segment 5 on the portal phase of contrast-enhanced magnetic resonance image (MRI) before cryoablation. (b) The intraoperative CT image at the end of cryoablation shows a clear ablation zone. (c) The fusion image is created after automatic rigid registration combined with manual correction. The medication-assisted manipulation (MAM) is categorized as group II: MAM 0e5 mm. (d) Contrast-enhanced MRI image obtained 4 months after cryoablation shows Local Tumor Progression (LTP) [180].
Applsci 10 01171 g004
Figure 5. A 25-year-old female patient with a history of colorectal cancer presented multiple liver lesions after surgery. The focal nodular hyperplasia (FNH) in the right liver shows an arterial contrast-agent enhancement (A), and is still hyperintense in the liver-specific contrast phase (C). No significant 18F-FDG-uptake is seen (B,D). A second lesion in the right liver is rated as a colorectal liver metastasis due to incomplete resection. Tumor lesion is neither detectable by MRI without liver-specific contrast phase nor with liver-specific contrast phase (E,G). In fused PET/MR images (F,H) the remaining tumor tissue lesion could clearly be identified. Additional lesions near the liver hilus are adenomas with strong arterial contrast-agent enhancement (I). In the liver-specific contrast phase lesions are hypointense (K). Similar to the FNH, no significant 18F-FDG-uptake is seen (J,L) [183].
Figure 5. A 25-year-old female patient with a history of colorectal cancer presented multiple liver lesions after surgery. The focal nodular hyperplasia (FNH) in the right liver shows an arterial contrast-agent enhancement (A), and is still hyperintense in the liver-specific contrast phase (C). No significant 18F-FDG-uptake is seen (B,D). A second lesion in the right liver is rated as a colorectal liver metastasis due to incomplete resection. Tumor lesion is neither detectable by MRI without liver-specific contrast phase nor with liver-specific contrast phase (E,G). In fused PET/MR images (F,H) the remaining tumor tissue lesion could clearly be identified. Additional lesions near the liver hilus are adenomas with strong arterial contrast-agent enhancement (I). In the liver-specific contrast phase lesions are hypointense (K). Similar to the FNH, no significant 18F-FDG-uptake is seen (J,L) [183].
Applsci 10 01171 g005
Table 1. Major medical image fusion methods.
Table 1. Major medical image fusion methods.
Fusion MethodsDiagnostic ModalityReferences
Arithmetic combinationUS, MRI, PET, CT[8,63,72,162,163,164,165,166,167]
IHSMRI/PET[31,32,33,34,35]
PCAUS-PET/CT, MRI/CT[41,42,43]
Pyramid based methodsMRI, CT, PET, SPECT[57,58,59]
Wavelet transformation-based methodsCT, PET, MRI, SPECT, Ultrasonography, Multiparametric MR images[7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,52,53,54,55,56,57,66,67,68,70,71,72,73,74,82,83,85,86,117,168,169]
Pulse-coupled Neural network methodsCT, PET, MRI, fMRI, SPECT, Ultrasonography, Multiparametric MR[84,94,111,112,113,114]
Fuzzy logic-based methodsCT, PET, MRI, MRA, fMRI, SPECT, Ultrasonography[121,122,123,124,125]
Sparse representation and compressive sensing-based methodsPET, CT, MRI, Multiparametric MR[138,139,140,141]
Edge-preserving based methodsCT/MRI[142,143]
Deep LearningPET-CT, MRI-CT, MRI-SPECT Multiparametric MR[12,13,14,15,16,17,18,19,20,21,27,28,29,30,145,146,147,148,149,150,151,153]

Share and Cite

MDPI and ACS Style

Li, C.; Zhu, A. Application of Image Fusion in Diagnosis and Treatment of Liver Cancer. Appl. Sci. 2020, 10, 1171. https://0-doi-org.brum.beds.ac.uk/10.3390/app10031171

AMA Style

Li C, Zhu A. Application of Image Fusion in Diagnosis and Treatment of Liver Cancer. Applied Sciences. 2020; 10(3):1171. https://0-doi-org.brum.beds.ac.uk/10.3390/app10031171

Chicago/Turabian Style

Li, Chengxi, and Andrew Zhu. 2020. "Application of Image Fusion in Diagnosis and Treatment of Liver Cancer" Applied Sciences 10, no. 3: 1171. https://0-doi-org.brum.beds.ac.uk/10.3390/app10031171

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop