Simulation Dataset Preparation and Hybrid Training for Deep Learning in Defect Detection Using Digital Shearography

Abstract

Since real experimental shearography images are usually few, the application of deep learning for defect detection in digital shearography is limited. A simulation dataset preparation method of shearography images is proposed in this paper. Firstly, deformation distributions are estimated by finite element analysis (FEA); secondly, phase maps are calculated according to the optical shearography system; finally, simulated shearography images are obtained after 2π modulus and gray transform. Various settings in the parameters of object, defect, load and shearing in those three steps could prepare a diverse simulation dataset for deep learning. Together with the real experimental images taken from a shearography setup, hybrid trainings of deep learning for defect detection are performed and discussed. The results show that a simulation dataset, generated without any real defective specimen, shearography system or manual experiment, can greatly improve the generalization of a deep learning network when the number of experimental training images is small.

1. Introduction

In digital shearography, when an object surface is loaded, the object defect generates deformation different from the surroundings and forms a distinct deformation gradient, which is mainly illustrated as a butterfly speckle interferogram in a shearography image. Since the butterfly pattern is identifiable, digital shearography has been wildly applied in defect detection [1,2]. For instance, the American Society of Testing and Material has taken digital shearography as a standard nondestructive testing practice for composite materials in aerospace. Moreover, digital shearography has also been used in defect testing for wood–plastic composites, rubber-to-metal structures, metal plates and tires [3,4,5,6,7,8,9,10,11].

As a part of post-image processing of shearography images, the recognition of a butterfly speckle interferogram is one of the key techniques in defect detection using digital shearography. In many popular products from commercial manufacturers, such as Dantec Dynamics (Skovlunde, Denmark) and Laser Technology Inc. (Norristown, PA, USA), areas of a butterfly speckle interferogram indicating defect locations are still drawn manually, which is laborious. In the past two decades, defect recognition methods based on wavelet transform, curvelet transform, spatial characteristics chain code, principal component analysis, derivative operation and some other tools have been proposed [11,12,13,14,15,16,17], but traditional image algorithms are sensitive to gray value change, which can be easily misled by the abundant noise in shearography images. Since defect detection is a typical binary classification problem, artificial intelligence techniques are applied to locate the butterfly pattern. As an example, artificial neural network architectures have been combined in defect detection using digital shearography in some applications [18,19,20]. Moreover, with the great achievements of deep learning in image understanding and object detection in recent years, elaborate deep learning-based algorithms have also been explored in digital shearography, especially the most popular convolutional neural networks (CNNs) [21,22,23]. Fröhlich et al. proposed a binary defect classification method using CNN for testing the integrity of a glass fiber patch and a metallic oil pipeline. About 300 shearography image samples grouped into two types of defects (major defect and minor/no defect) were trained and tested with a best accuracy of 79%. Ye et al. applied a Faster R-CNN model for bonding defect identification on a cylindrical surface. A dataset of 200 shearography images was trained and tested with a high recognition rate of large defects, but a low recognition rate of small defects. Chang et al. also used an ensemble hybrid Faster R-CNN model for tire quality assessment, where 359 tread shearography images were trained and tested with a best accuracy of 89.16%. In the above three references, the training datasets are 200 or 300 images in size, which is low for deep learning. In general, object detection accuracy is raised when the training dataset is enlarged. However, unlike datasets of faces, human beings or animals, an ideal and huge dataset of shearography images is inaccessible by open access. Oliveira et al. designed the U-net CNN given the lack of a large training dataset and analyzed the impact damage in carbon fiber-reinforced plastics (FRP). A dataset with 1905 samples was used, but its effectiveness on a dataset of several hundred images is still ambiguous [12].

Several hundred shearography images are achievable in-site, but thousands of images are hardly accessible. The preparation of a satisfied shearography dataset for training for deep learning will be time-consuming, costly and perhaps even impossible due to the lack of diverse defective specimens in-site. Therefore, the application of deep learning in defect detection using digital shearography is limited.

To minimize the effect caused by the lack of experimental shearography datasets, the simulation of shearography images for deep learning is proposed and the effectiveness on the joining of those simulated images for defect detection is discussed in this paper. The generation of simulation shearography images is numerically modeled with multiple parameters according to the production mechanism of a real experimental shearography image. No real defective specimen, shearography system or manual experiment is needed, and diverse simulated shearography images can be prepared with different settings of the model parameters. The proposed dataset preparation method will be labor-saving and efficient.

2. Deep Learning and Dataset Preparation

Deep learning has been widely used in the field of object detection. Current state-of-the-art deep learning approaches are mainly divided into two kinds. One is the two-stage approach, such as CNN-based algorithms [24,25,26,27]. The other is the one-stage approach, such as YOLO versions [28,29]. The two-stage approach firstly selects regions of interest (ROIs) by a region proposal network and then performs classification on those ROIs. The one-stage approach performs the object localization and classification by only one network. Because the one-stage approach usually runs faster, YOLOv4, a typical one-stage approach, is applied in this paper to make defect detection quick.

2.1. YOLOv4

YOLOv4, the fourth version of YOLO, is a single-stage object detection algorithm using convolutional neural networks with optimal speed and accuracy. As shown in Figure 1, it consists of a backbone, a neck and a detection head. Its backbone network takes CSPDarknet53 to train and extract features [30]. Its neck network utilizes spatial pyramid pooling (SPP) and path aggregation network (PAN) to collect feature maps at different training stages to improve the detection ability of targets of different scales [31,32]. Its detection head is composed of a YOLO layer. The YOLOv4 code utilized in this paper was downloaded from https://github.com/AlexeyAB/darknet (accessed on 30 October 2021).

2.2. Simulation Dataset Preparation

An experimental shearography image is transformed from a phase map usually calculated by phase-shifting in a real application. Here, a simulation shearography image is also transformed from a phase map, but the phase map is directly calculated according to the optical shearography system with known deformation distribution. Therefore, a simulation dataset preparation method for deep learning in defect detection using digital shearography is numerically modeled with multiple parameters in three steps: estimation of deformation distribution, simulation of phase map and transformation of shearography image.

2.2.1. Deformation Distribution

Finite element analysis (FEA), which is a numerical technique to find solutions to partial differential or integral equations of field problems, is utilized to simulate the deformation distribution of an object. Firstly, the 3D structure and material type of the object are configured. Then, the defect pattern with a specific size, number and location is designed on/in the object. Finally, with the defected object and its configuration, FEA software tool estimates the numerical deformation distribution according to the external load.

The different settings of the object (3D structure/material type), defect (size/number/location) and load (type/strength) will simulate various deformation distributions, which can meet the diversity consideration.

2.2.2. Phase Map

When the deformation distribution w(x, y) is estimated from the above, a phase map Δ(x, y) can be calculated according to the principle of digital shearography. Figure 2 is a schematic diagram of an out-of-plane Michelson-type digital shearography system. The laser is expanded with Lens 1 and then projected onto the object. The diffuse light from the object surface is divided into two beams by the Michelson device with a beamsplitter and two mirrors. Mirror 1 is tilted at an angle to produce image shearing, and Mirror 2 is usually fixed on a piezo-electric transducer (PZT) to generate phase shifting. Suppose the shearing direction is along k = (cos α, sin α)^T; the phase difference (or phase map) Δ(x, y) between before and after loading is denoted as

$$\begin{array}{l}\Delta =\frac{4\pi {\delta }_k}{\lambda }\frac{\partial w}{\partial k}\\ \hspace{1em}=\frac{4\pi {\delta }_k}{\lambda }\left(\frac{\partial w}{\partial x}\cos \alpha +\frac{\partial w}{\partial y}\sin \alpha \right)\end{array}$$

(1)

where λ is the laser wavelength and δ_k is the shearing value in the k direction.

With different settings in shearing value and shearing direction of just one simulated deformation distribution, amounts of simulated phase maps will be calculated. Therefore, shearing settings can also meet the diversity consideration.

2.2.3. Shearography Image

A digital shearography image I, with a gray value range of [0, 255], can be transformed from the phase map Δ as

$$I=Int\left[\frac{255}{2\pi }mod\left(\Delta ,2\pi \right)\right]$$

(2)

where Int(•) is an integer operation, and mod(•, 2π) is a modulus operation by 2π. Therefore, various simulated phase maps result in various simulated shearography images, which will constitute a large simulation shearography dataset for deep learning.

2.3. Hybrid Training

As mentioned, the experimental dataset of shearography images is too small for deep learning and hard to expand, so the simulated dataset is prepared. However, the effectiveness in the joining of the simulated dataset must be studied. To maintain detection accuracy in real applications, experimental samples must be trained and tested. Hybrid trainings of deep learning based on both simulated and experimental shearography images are performed and tested in this work. The simulated shearography images are obtained following the calculation described in Section 2.2, and the experimental ones are obtained from a real setup. The effectiveness of the simulation dataset in improving the detection accuracy and generalization of YOLOv4 is discussed in Section 3.

3. Tests and Results

3.1. Simulation Dataset

We designed 150 mm × 150 mm × 6 mm plates with holes (as defects) ranging from 1 mm below the top to 1 mm up the bottom in different diameters and locations. Figure 3 shows the front views of eight specimens with eight kinds of materials and eight defect patterns. Figure 3a–h are panels of aluminum with nine defects, honeycomb with two defects, PVC foam with three defects, epoxy carbon with one defect, resin epoxy with one defect, glass with two defects, E glass fiber with one defect and ABS thermoplastic with three defects, respectively. The defects are randomly located and sized with diameters from D8 mm to D30 mm. Their 3D structures were drawn in SOLIDWORKS and imported to the FEA software tool ANSYS.

Thermal, vacuum and force loadings with strength in every level were simulated in ANSYS to obtain more deformation data. Figure 4 shows two deformation distributions and shearography images of the ABS thermoplastic panel in Figure 3h with thermal loads of 30 °C and 50 °C. The distributions in Figure 4a,c are similar, but the maximum deformations are distinct. With different thermal loads, two different shearography images of 640 × 640 pixels are simulated in Figure 4b,d. Figure 5 shows those of the honeycomb panel in Figure 3b with force loads of 5N and 20N. Two different shearography images of 640 × 640 pixels are also simulated in Figure 5b,d. Apparently, when the load is in a small strength, the defect deformations are small and butterfly patterns of some defects are vague.

Besides loading settings, shearing settings can also enrich the simulation dataset. Figure 6 shows simulated shearography images of 640 × 640 pixels in four shearing directions, i.e., horizontal, vertical and two diagonals. The butterfly patterns are distinct in the different shearing directions.

With all the settings in the parameters of object, defect, load and shearing, we prepared a simulation dataset with 5117 shearography images of 640 × 640 pixels for deep learning. Figure 7 shows part of the simulated dataset. The simulated samples are rich in diversity.

3.2. Experimental Dataset

Many engineering specimens with defects were predesigned and tested using our developed shearography setup in Figure 8 to make the experimental dataset. Figure 9 shows five specimens: FRP skin paper honeycomb panel, 450 mm × 500 mm (specimen 1); carbon fiber honeycomb panel, 300 mm × 600 mm (specimen 2); FRP foam panel, 200 mm × 600 mm (specimen 3); aluminum skin aluminum honeycomb panel, 400 mm × 350 mm (specimen 4); and glass curtain wall, 360 mm × 360 mm (specimen 5). As with the simulation dataset preparation, different loads and shearings were tried and tested. An experimental dataset with 1900 shearography images was finally obtained for deep learning. Figure 10 shows five shearography images of part views of the five specimens in Figure 9a–e with one load and shearing setting. The defects are labeled in red boxes. Compared with the simulated images in Figure 7, the experimental images are quite different in noise and background. This is the exact reason that hybrid training combining both the experimental images and simulation images is designed, as opposed to training only considering the simulation images.

3.3. Tests

With the above sample images from simulations and experiments, a series of hybrid training for deep learning is performed.

Table 1. Training results with only experimental dataset.

Test	Simulation Images	Experimental Images	Accuracy on Validation Set	Accuracy on Test Set
1	0	1900 (0.2:0.4:0.4)	98.84%	99.37%
2	0	950 (0.2:0.4:0.4)	98.84%	98.09%
3	0	475 (0.2:0.4:0.4)	98.74%	98.28%
4	0	238 (0.2:0.4:0.4)	99.02%	62.38%
5	0	95 (0.2:0.4:0.4)	92.31%	48.65%

shows five training results of YOLOv4 only with the experimental dataset. From Test 1 to Test 5, the experimental samples are successively cut down to half, but randomly separated in every test as a training set, validation set and test set with proportions of 20%, 40% and 40%, respectively. Accuracies on the validation sets for all the five tests are satisfied, being always larger than 92%, but the accuracies on some test sets sharply decreased when the number of sample images was reduced. There is a wide gap in Test 4 between the accuracy on the validation set, which is 99.02%, and that of the test set, which is only 62.38%. This indicates that the YOLOv4 network trained with 238 experimental images is over-fitted and has poor generalization. Similarly, Test 5, with only 95 sample images, obtains 92.31% on the validation set and 48.65% on the test set, which shows worse generalization of the deep learning network. This is consistent with what know: the size of the dataset in deep learning should not be too small for high detection accuracy and good generalization. When the training dataset in digital shearography is large enough, defect detection using deep learning is robust and applicable. However, when the number of training images is smaller than 250 in digital shearography, which is the common situation in real engineering applications, defect detection using deep learning is unstable and inapplicable.

The simulation dataset prepared in Section 3.1 is then joined to improve the generalization of deep learning in digital shearography.

Table 2. Training results with hybrid dataset.

Test	Simulation Images	Experimental Images	Accuracy on Validation Set	Accuracy on Test Set
6	5117 (1:0:0)	1900 (0.2:0.4:0.4)	96.44%	96.00%
7	5117 (1:0:0)	950 (0.2:0.4:0.4)	87.71%	88.88%
8	5117 (1:0:0)	475 (0.2:0.4:0.4)	85.34%	90.31%
9	5117 (1:0:0)	238 (0.2:0.4:0.4)	76.25%	85.00%
10	5117 (1:0:0)	95 (0.2:0.4:0.4)	81.34%	75.46%
11	5117 (1:0:0)	1900 (0:0.5:0.5)	43.00%	43.60%

shows the results of YOLOv4 with hybrid simulation and experimental dataset. The numbers of experimental samples in Test 6 to Test 10 are the same as those of Test 1 to Test 5, respectively, and the experimental images in every test are also randomly separated as training set, validation set and test set with proportions of 20%, 40% and 40%, respectively. The extra 5117 simulated images are added to the training set. Compared with Test 1, 2 and 3, there are small drops in the accuracies on the validation set or the test set in Test 6, 7 and 8, respectively. This is caused by the appearance difference of the simulated images in Figure 7 and the experimental images in Figure 10. The joining of simulated images damages the dataset’s consistency and makes no improvement. Similar results can also be found in Test 11, where the 5117 simulated images are taken only as a training set, and the 1900 experimental images are separated only as validation and test sets with proportions of 50% and 50%. Accuracies on the validation and test sets are only 43.00% and 43.60%. The simulated shearography images are not always helpful.

Unlike Test 4 and 5, there are no obvious gaps between accuracies on the validation set and the test set in Test 9 and 10. The over-fitting problem of YOLOv4 with small training data is largely solved with the extra simulated samples. The accuracy on the test set can be improved to 85.00% with 238 experimental shearography images and to 75.46% with less than 100 (only 95) experimental shearography images. Therefore, when the experimental samples are less than 250 and are limited to enlargement in real applications, the simulation dataset can obviously improve the generalization of deep learning. The proposed simulation dataset preparation and hybrid training strategy will make deep learning more applicable for defect detection in digital shearography.

4. Conclusions

A simulation dataset preparation method of shearography images for defect detection using deep learning is proposed in this work. Diverse simulation datasets can be generated by various parameter settings without any real defective specimen, shearography system or manual experiment. Hybrid trainings of YOLOv4 with both simulation and experimental datasets show a significant improvement in the robustness of the deep learning network with extra simulated shearography images when the experimental dataset is small, but little improvement when the experimental dataset is large enough. Therefore, the joining of simulation shearography images for defect detection using deep learning is strongly recommended when the real experimental images are few and hard to enlarge. Future work will be carried out to determine how many simulation images are needed to benefit to fixed experimental images.