Energy Reduction with Super-Resolution Convolutional Neural Network for Ultrasound Tomography

Abstract

This study addresses the issue of energy optimization by investigating solutions for the reduction of energy consumption in the diagnostics and monitoring of technological processes. The implementation of advanced process control is identified as a key approach for achieving energy savings and improving product quality, process efficiency, and production flexibility. The goal of this research is to develop a cost-effective system with a minimal number of ultrasound sensors, thus reducing the energy consumption of the overall system. To accomplish this, a novel method for obtaining high-resolution reconstruction in transmission ultrasound tomography (t-UST) is proposed. The method involves utilizing a convolutional neural network to take low-resolution measurements as input and output high-resolution sinograms that are used for tomography image reconstruction. This approach allows for the construction of a super-resolution sinogram by utilizing information hidden in the low-resolution measurement. The model is trained on simulation data and validated on real measurement data. The results of this technique demonstrate significant improvement compared to state-of-the-art methods. The study also highlights that UST measurements contain more information than previously thought, and this hidden information can be extracted and utilized with the use of machine learning techniques to further improve image quality and object recognition.

1. Introduction

Advanced automation and control of manufacturing processes play a key role in maintaining competitiveness. While costly process equipment and production lines can be considered the heart of industrial production, control systems and information technology are its brain. They provide the flexibility to quickly adapt production processes to changing customer requirements and ensure safety and efficiency at the lowest possible resource and energy costs. Hence, the development and application of advanced process control is one of the most effective levers for immediate and long-term gross energy savings, improvement, product quality, increased process safety, and greater production flexibility, and will provide security and promote economic growth in conventional and emerging areas.

Advanced process control includes all control procedures beyond standard closed-loop control, using PID controllers and sequential control. This approach optimizes the system so that the processes themselves always remain reproducible, leading to a further increase in throughput, productivity, and product quality on the one hand and a reduction in energy and raw material expenses on the other. One of the most interesting and advanced methods used in industry is tomography [1,2,3].

Computed tomography is one of the basic diagnostic methods included in the so-called imaging tests. It is readily available, and common, and is widely used in diagnosis and inspection. Of the numerous computed tomography methods based on physical phenomena such as electrical resistivity tomography (ERT) [4], electrical capacitance tomography (ECT) [3], electrical impedance tomography (EIT) [2], radio wave tomography and localization [5], X-ray tomography [6] or magnetism tomography [7], ultrasound tomography (UST) [8] is a commonly used method in medicine and industry [9,10]. UST is an imaging technique that exploits the properties of acoustic wave transmittance through tissues and materials [11].

Ultrasound tomography is most commonly recognized under the name of “medical ultrasound” or “ultrasonography” (USG). This technique is very useful when no other type of measurement is possible such as in the case of electro-stimulation [12]. In this paper, a low-cost industrial transmission ultrasound tomography (t-UST) is described. The paper focuses on the problem of improving reconstruction quality using low-cost probes to measure the Time of Flight (ToF) between sensors [13,14,15].

An example of an industrial application (t-UST) is to locate around a pipe with flowing liquid, then it is possible to reconstruct images from the measurement data, which will show any perturbations in the flow i.e., air bubbles. This can be very useful for non-invasive leak detection. In various scenarios, it is possible to monitor multiphase flow with such a set-up, which can then be used to monitor industrial processes [13]. The UST is also an important device for energy systems such as oil and gas [16,17] (for measuring velocity and flow) and crystallization [18] (for carbon capture monitoring).

Measurement with (t-UST) is carried out with probes placed at the edge of the object. With one of the probes, ultrasonic waves are generated. Other probes that have been placed around the object record the signal received from the transmitting probe. Such a procedure continues until $N\times N-1$ (the transmitting probe simultaneously receives the reflected signal) measurements are collected, where N is the number of ultrasound drivers.

In order to image the interior of the object under study based on measurements collected from probes, the most common approach is to solve an inverse problem (IP), where an underdetermined set of linear equations is solved [19,20,21]. This can be described by the following equation:

$$Je=m$$

(1)

where J is the sensitivity matrix, e are the elements of the mesh on which we reconstruct the images, and m is the measurement.

The sensitivity matrix J is describing the relation between the physics of the measurement (ToF) and the reconstructed image representing a deviation from the mean sound speed. Note that the J is not square; thus, it can be computationally expensive to find the pseudo-inverse $J^{-1}$. Additionally, finding the inverse is impossible without additional regularization terms that modify the final solution [22].

To solve this problem, the authors decided to use a well-known method developed by Radon [23]. From the measurements of ToF, one can pick measurements that are parallel to each other for a given measurement angle.

On the one hand, this approach compresses the information gathered from measurements from $N\times N-1$ to $\frac{N}{2}\times \frac{N}{2}-1$ through the process of calculation of Radon Transform (sinogram). On the other hand, it allows for using a very fast method to find the reconstruction. This method has one downside, it heavily relies on the information from the sensors, and it is recommended to use as many as possible. Figure 1 shows why the larger number of ultrasound transducers is important. A large number of details can vanish in the case of 16 sensor sinograms.

This stands in contradiction to the assumption of cheap industrial tomography. Another drawback of using a high number of ultrasound sensors is the power composition of the system. The device presented in the paper uses measurement cards, each controlling four measurement channels. Each card consumes 5 W during measurement. If the measurements are made at a very high rate, e.g., 30 measurement frames per second, the power consumption for 64 sensor systems can be as high as 95 watts. On the other hand, measurements with a lower number of electrodes, e.g., 16 sensors, will lower the power consumption to 35 watts. This reduction in energy consumption is achieved by reducing the number of active measurement cards from 16 to 4. It is especially important if a large number of tomographs will be used in the industrial processes because the tomograph has to operate 24 h/7 and will have a big impact on energy consumption. On the other hand, a lower number of measurement channels results in a lower number of measurements.

To deal with that problem, we need to increase the number of measurements ergo increasing the resolution of the sinogram. This leads us to the methods used in the domain of neural networks used for obtaining super-resolution images from low-resolution input images, the technique used in a wide range of methods that depends on neural networks.

The use of machine learning algorithms in industrial processes and tomography is common [2,9,21,24]. The authors decide to use a very simple approach that is a fundamental method used in super-resolution technique known as Super Resolution Convolutional Neural Networks (SRCNN) developed by [25]. In the first step, a conventional neural network very similar to SRCNN was created and trained on purely simulation data. Next, the network trained on the simulation data was validated on actual ultrasonic measurements collected from the circular tank.

2. Materials and Methods

The SRCNN itself is very simple and consists of convolutional layers that play different roles in image reconstruction. Its structure is shown in Figure 2.

The network consists of three parts patch extraction and representation, nonlinear mapping, and reconstruction. This network was designed as one of the first deep networks in the study of super-resolution. It was designed to obtain high-resolution images from low-resolution images.

The justification for the use of an SRCNN-like structure in this work is that both measurement matrix and sinograms can be treated as images that are linearly coupled to each other.

The sinograms are calculated with the use of the method described by [23]. For each measurement array, we find measurements for which the propagation of the signal in the tank is parallel. Then, measurements are then saved as a row in the sinogram. The process is repeated for each projection angle until full rotation around the tank is made. In the measurement array, each row consists of measurements, where the propagation of the signal is not parallel and constructs a network of connection between the emitter probe and all other receiving probes. In sinogram, each row represents a measurement for which the emitter$\stackrel{}{\to }$receiver connection is parallel and perpendicular to the projection angle.

There are a few major differences between the solution proposed in the paper and the original SRCNN model shown in Figure 2.

First, bi-linear up-sampling of the input image is not used, and the input shape is different from the output shape. This is because at the input we use a measurement matrix and at the output, and we expect a high-resolution sinogram from the high-resolution measurement matrix.

Second, a dense layer with a sigmoid activation function to perform a mapping from measurement to the sinogram space at the output was added.

Except for these modifications, the motivation behind the network is the same which is to obtain high-resolution images of sinograms from the low-resolution measurement.

The reason behind using low-resolution measurement versus low-resolution sinogram at the input is that measurement consists of a larger amount of information needed to obtain a high-resolution sinogram, which is unavailable in the low-resolution sinogram. This claim is validated in the Results section of this paper.

The model presented in Figure 3 was trained on the simulation data that we generated using our deterministic ultrasound algorithms. The algorithms allow us to solve the forward problem (calculate measurement from the known state using Equation (1)), and solve the pseudo-inverse problem by finding the inverse matrix $J^{-1}$.

A total of 93,306 cases were generated including all single perturbation cases and around 89,000 cases having two perturbations.

The single perturbation means that one pixel in the reconstruction image (ground truth) is set to one and the rest is 0. The case of two perturbations has two pixels with 1 and the rest is zero. The forward problem was solved for each such case, and the sinogram was calculated.

The dataset created from simulations was split into train and validation subsets, where 30% of the whole set was validation data. During training, the model never saw any cases corresponding to three or more perturbations. Finally, the model was evaluated by carefully inspecting the reconstruction from the predicted sinogram using the data obtained from the real measurements from the circular tank filled with water with rigid body inclusions. These data were the test dataset used for model evaluation.

The biggest difference between the training/validation set and test set is that the training/validation dataset consists of single pixel perturbation while the test dataset consists of multiple groups of pixel patches. The evaluation dataset is thus much harder to reconstruct making it perfect as the test set. Finally, after obtaining a super-res sinogram, the Radon method is used to calculate reconstruction.

Hardware

The tomograph used to obtain the data are an ultrasound transmission tomograph developed by Netrix S.A. It uses a system of active measuring probes communicating with each other via the CAN 2.0B bus.

The tomograph has full responsibility for controlling the measurement sequence, and active measurement probes, and for capturing and collecting data. The active probe of the ultrasound tomograph uses a single piezoelectric transducer for measurement in the absorption mode. This transducer shown in Figure 4, with a resonant frequency of 40 kHz, functions as an ultrasound transmitter and receiver. External transducers can be connected to the circuit board via the SMB socket (e.g., transducers immersed in the medium or permanently attached to the tank or pipe). The probe incorporates a microcontroller with integrated signal processing and an A/C converter. Using a programmable digital potentiometer, each probe can adjust the amplification of the received signal.

The probes are designed to be placed close to each other. The power lines, communication bus, and interrupt lines, which are necessary for proper measurement of the time from transmission to reception of signals from the other probes, are therefore made with RJ-12 cables.

The measurement of the ultrasound transmission time from one probe to another is conducted by connecting all probes to another via the communication line. If a low condition occurs on this link, all probes except the transmitting probe start measuring time and stop measuring time after receiving the ultrasound signal. Each receiving probe then transmits the measurement results to the tomograph controller. The analog signal is processed by an A/D converter or comparator with programmable thresholds. Based on the information of which probe generated and which probe received the signal, the measurement values (Time of Flight) are stored in the corresponding cells of the measurement matrix.

3. Results

The results presented in this section show how the network performed on the real measurement data—first, the approach, whereas, as input, we fed a full measurement matrix that was tested and then validated by feeding the network with sinograms. In both cases, a super-resolution sinogram at the output of the network was expected.

3.1. Results of the Modified Network

Figure 5 shows the results obtained with the training data. It can be observed that the predicted sinograms and the reconstructed images are of great quality. To check the accuracy of the predicted sinogram, the squared error of the reconstruction was shown. Reconstructions are nearly ideal with a small error at the edges.

The results obtained from the network and the examples of the reconstructions from real measurements are shown in Figure 6 and Figure 7.

The top left panel of Figure 7 shows the real measurement. The top right panel of the Figure shows the reconstruction obtained with the Radon method. The bottom left panel shows the reconstruction with the data predicted with the model, and the bottom right panel shows the measurement setup. The position of the top left phantom seen in Figure 7 is not correctly recognized, but the position of 3 out of 4 phantoms is correct.

Four well-known indicators were used to assess the quality of the reconstruction: Mean Absolute Error (MAE), Mean Square Error (MSE), Peak Signal to Noise Ratio (PSNR), and structural similarity index measure (SSIM). The MSE metric was evaluated according to the equation:

$$MSE=\frac{1}{n}\sum _{i=1}^n{\left({x^{\prime }}_i-{x^{*}}_i\right)}^2$$

(2)

where n—number of pixels in the image, ${x^{\prime }}_i$—reference of i-th pixel, ${x^{*}}_i$—value of i-th reconstructed pixel.

The next measure of reconstruction quality MAE was calculated according to the equation:

$$MAE=\frac{1}{n}\sum _{i=1}^n|x_i^{\prime }-x_i^{*}|$$

(3)

Many derivative measures have been developed based on MSE, and PSNR, but all these measures have similar disadvantages regarding, among others, sensitivity to the mutual displacement of images or their rotation. In such a situation, even an image of excellent quality, but shifted only by one or a few pixels from the original, would be assessed as distorted.

The quality PSNR was calculated according to the equation:

$$PSNR=10{log}_{10}\frac{{(max{x}_i^{\prime })}^2}{MSE}$$

(4)

SSIM is an extension of the universal image quality indicator. In this method, which is sensitive to the three most typical types of distortions, i.e., brightness change, contrast change, and image structure disturbance, the reference image is scanned and evaluated based on a sliding window (usually, it is a Gaussian window with a size of 11 × 11 pixels), for which the calculated is the local image fragment quality index according to the relation (after simplification):

$$SSIM=\frac{\left(2{\mu }_x{\mu }_y+c_1\right)\left(2{\sigma }_{xy}+c_2\right)}{\left({\mu }_1^2+{\mu }_2^2+c_1\right)\left({\sigma }_1^2+{\sigma }_2^2+c_2\right)}$$

(5)

where ${\mu }_x$—average value of the reference image inside the window, ${\mu }_y$—average value of the image evaluated inside the window, ${\sigma }_1^2$—variance of the reference image inside the window, ${\sigma }_2^2$—variance of the image evaluated inside the window, ${\sigma }_{xy}$—covariance, $c_{21}$, and $c_2$—constants protecting against instability.

Equation (5), in its expanded form, is the product of three factors corresponding to the three listed types of misstatements. First, the role of the instability protection constants is to prevent division by zeros, which could occur for very dark and “flat” (solid color) portions of the image. The recommended values are $C1=0.01L$ and $C2=0.03L$, where L is the maximum allowed brightness level in the image (usually $L=255$).

By moving the window by one pixel, a map of image quality is obtained, which, after averaging, allows us to obtain a scalar quality index in the $<-1;1>$ range.

Analyzing the obtained results, it can be concluded that the SSIM is more useful for assessing the quality of images obtained using various algorithms compared to the classical mean square error. Figure 8 shows the values of this indicator’s values and the smallest values of the mean squared error.

SSIM assesses the structural similarity between two images by comparing the local patterns of pixels in the images, while PSNR calculates the ratio of the maximum possible power of an image to the power of the noise in the image, and it produces a score in decibels (dB). PSNR is a simple and widely used measure for image quality assessment that compares the pixel-by-pixel differences between two images, while SSIM is a more sophisticated measure that takes into account the structural information in the images and is more closely aligned with human perception of image quality.

Additionally, MSE, MAE, PSNR, and SSIM were calculated for each reconstruction obtained with the use of the developed network as shown in Figure 8. The results show that the network allows for a high PNSR and large SSIM values and thus indicates that it is very accurate for the simulation data. The low values of MSE and MAE indicate that there is not much error regarding the position of inclusions in original images.

As can be seen from mentioned figures, the reconstruction improved significantly concerning the original data. The two phantoms in the tank were separated, and their location and shape were more precise.

The network was able to reconstruct a sinogram from real measurements containing four phantoms, and after inverse radon transformation, the phantoms are visible in the tank. This result is beyond the possibilities of the current methods used in ultrasound tomography.

In the validation stage of the network, the limits of the possibilities of the developed network were checked, and it was found that, with the real measurements, the network works best with up to four phantoms. It is worth noting that the network during the training never saw any cases with more than two inclusions at once. Thus, the results obtained with real measurements, including three and four inclusions, exceeded expectations.

3.2. Validation of Initial Assumptions

In Section 2, it was claimed that, to achieve high-resolution reconstructions, we need to use low-resolution measurement data and not low-resolution sinograms as input. To prove the point, we trained a very similar CNN to the original model developed by [25] where low-resolution sinograms are used as an input as seen in Figure 9. The network is then trained as before on the same data set and validated on the real measurement data just as in the case of the previous model.

The results in Figure 10 clearly show that, in the case of two phantoms, the reconstructions cannot correctly indicate the location and the size of inclusions. It is expected as the low-resolution sinogram consists of a compressed and denoised version of the measurement matrix. The results obtained for other cases (one, two, three, and four phantoms) are similar and do not correctly show the inclusions in the tank.

The direct reason for worse network performance is the amount of information in the sinogram. The raw measurements include much more noise, but, at the same time, they are rich in information about the objects inside the tank. The network can easily eliminate noise in the measurement and easily achieve super-resolution of the sinograms.

4. Discussion and Summary

The article presents a modified version of the SRCNN applied to ultrasound transmission tomography. The direct advantage of using the developed model is the cost reduction and energy consumption of the ultrasound tomographic system.

The modification of the network consisted of changing the idea of the same image in the input and the output of the network. In the model presented in the paper as input, we put a full 16-sensor transmission UST measurement matrix and, as an output, we expect a high-resolution sinogram. The expected sinogram should be equal to the one that can be calculated from 64 sensor measurements. The developed convolutional network was not only able to reconstruct such sinograms but also reconstructed images. As a result, we were able to recognize the position and the size of the phantoms.

The reconstructions obtained on the test set are close in terms of PSNR and SSIM values to the results obtained in other papers devoted to image reconstruction using deep learning methods. However, the datasets for methods presented in

Table 1. The comparison table of results achieved with super resolution approach networks on similar datasets.

Method	PSNR	SSIM
iNet [26]	31.8738	0.9249
U-Net - Elipses dataset [27]	28.02	0.8766
PWLS-PCG [28]	11.3	N/A
SIT-GAN [29]	28.714	0.8814
This Work	30.390	0.999

are different. The results obtained here were compared to them as the PSNR and SSIM values are a great indicator of the reconstruction quality.

The network outperformed the classical methods [30] and allows a system with 16 sensors to behave like a system with 64 sensors. The network provides a simple method for a cheap t-UST to be applied in more complex problems with a higher frame rate than the older classical methods. This study has a second important finding. The UST measurements hide much more information than it seems. Unfortunately, this information stayed mostly hidden from the deterministic and other machine learning models until now.

Directions for further work are related to improving still image quality and object shapes. Among other things, the authors plan to use auto-encoders in the future. The developed method can be easily used in dynamic data processing consisting of stirring, and this topic will be explored in future projects.