WELDMAP: A Photogrammetric Suite Applied to the Inspection of Welds

Abstract

This paper presents a new tool for external quality control in welds using close-range photogrammetry. The main contribution of the developed approach is the automatic assessment of welds based on 3D photogrammetric models, enabling objective and accurate analyses through an in-house tool that was developed, WELDMAP. As a result, inspectors can perform external quality control of welds in a simple and efficient way without requiring visual inspections or external tools, and thus avoiding the subjectivity and imprecisions of the classical protocol. The tool was validated with a large dataset in laboratory tests as well as in real scenarios.

1. Introduction

Welding is a very important task in the field of engineering and construction, since welds that are not properly carried out reduce the mechanical properties of joints, minimising their effective service life, and possibly promoting their collapse, which could entail drastic consequences, being that welding joints are the weakest links in assemblies [1]. For this reason, quality requirements are highly standardised (e.g., [2,3,4] and also highly specified for specific cases such as pressure vessels [5]).

Therefore, welding is a joining process that requires inspection and monitoring tasks. The visual inspection is a simple and widely extended non-destructive test modality [6], which is applied for the assessment of weld bead geometry and the detection of surface imperfections and flaws. Visual inspection tests are regulated by the standard [7].

The application of visual inspection allows for detecting defects and imperfections that have a manifestation on the surface of the weld. The taxonomy of defectology in welds is widely established by the international quality standards (e.g., [3,4]). So, inspection techniques are aimed at detecting and evaluating defects established in these norms. Visual inspection, as a non-destructive testing method, should be implemented by inspectors with training and expertise who have acquired a level of competence also established by the international standards (e.g., [8]). These inspectors are often engineers with extensive knowledge of the welding process and who have passed several exams to demonstrate that they have the right skills.

The inspector usually has the obligation to visit the welds on-site to evaluate the quality of each weld performed according to the international standards, which entails a significant economic investment [6]. In addition, the inspector must evaluate the weld at the site, often in a very short time and record the information of each weld in an analogical way or in traditional databases. To do this, it is common to use different auxiliary tools to evaluate the quality of welds such as tweezers or weld gadget kits [9]. These tools, being for manual use, generate measures with significant errors and especially result in visual fatigue for the inspector.

Currently, some authors are investigating other, more sophisticated techniques for the detection and identification of external pathologies in welds: laser systems [10,11], scanning cameras [12], hybrid laser-based systems and cameras [13,14,15,16], stereo imagery [12,17], close-range photogrammetry [9,18], and, finally, active thermography systems as supports for visual inspection, which are applied directly to the study of cracks in welds [19].

Among these techniques, close-range photogrammetry and laser scanning allow the accurate three-dimensional reconstruction of weld bead geometries in a non-invasive way. Such 3D models can be used for the inspection of external welding imperfections and defects. However, there are several differences in terms of operating principle (i.e., active versus passive sensors), working range, and cost-efficiency ratio. The reader can find more information about this comparison in [20,21]. Close-range photogrammetry has been used for the generation of 3D models that allow for the analysis of defects such as overlap and lack of fusion, among others [9]. It has also been used for the three-dimensional assessment of plaque misalignment [22]. In [9], a pioneering method of inspection based on close-range photogrammetry was established. For its part, active systems such as laser scanning have been also used for the evaluation of the external quality of welds. In [22], authors used a structured light system, which allows the accurate assessment of the weld bead according to the quality criteria. The procedure developed enables the extraction of metric information about the thickness and the deformation angle of the welds. However, to date, there is not a global solution that integrates the 3D reconstruction of welds with the automatic assessment of their external quality according to international standards.

Therefore, this paper presents a software, WELDMAP, which allows systematising the external assessment of welds based on close-range photogrammetry, integrating algorithms to implement defectology analysis in an appropriate manner in accordance with the quality standards.

After this introduction, the document has been structured as follows: Section 2 describes the methodology, highlighting the key steps for the 3D welds’ reconstruction and their defectology analysis. The experimental results are listed and discussed in Section 3. Finally, Section 4 is devoted to outlining the main conclusions and future perspectives.

2. Methods

Currently available commercial products devoted to the quality control of welds bypass the use of robust and accurate 3D models and are more focused on the assessment of internal defectology using consolidated non-destructive techniques (e.g., X-ray, ultrasounds, etc.). Considering this limitation, an in-house software, named WELDMAP, has been developed with the aim of providing a robust tool for the external inspection of welds. Based on the advantages offered by computer vision (low-cost and flexibility) and photogrammetry (accuracy and reliability), together with the different algorithms specifically developed for quality control, the information for the inspection of welds can be computed, analysed, and monitored in an efficient way. As a result, objective and accurate expert reports can be generated by the inspectors. Figure 1 outlines the main workflow codified in the software WELDMAP. The following sections will explain this software’s methodology.

2.1. Close-Range Photogrammetric Methodology

The photogrammetric pipeline is a process oriented to obtain a dense and three-dimensional point cloud of a weld’s surface through the application of different algorithms and phases, which will be detailed in the following subsections.

Extraction and Matching of Features

Since image acquisitions were not only limited to vertical but also an oblique point of view, the processing of the images must incorporate last advances and algorithms in computer vision for images matching [23] and images orientation [24,25]. Before extracting and matching features, images must be pre-processed in order to improve the radiometric contents and ease the successive feature extraction. Image pre-processing has been reported in many papers as a fundamental step, particularly in those cases where the texture quality is non-favourable [26,27,28]. Different pre-processing algorithms are available in WELDMAP, including, among others: ACEBSF [29], POHE [30], RSWHE [31], Wallis [32], etc. This step is facultative but highly suggested in order to achieve better results in the feature extraction and matching step.

WELDMAP implements diverse sets of detectors (e.g., SIFT [33], SURF [34], MSER [35], MSD [36], ORB [37], AKAZE [38], BRISK [39], etc.) and descriptors (e.g., BOOST [40], BRIEF [41], DAISY [42], FREAK [43], etc.) to let the user run and test different combinations and assess the results in different conditions. Any kind of combination is allowed in the software. For each detector–descriptor, several advance parameters can be defined by the user.

Finally, the features extracted are matched robustly using geometric constraints in order to guarantee that they match the same point. To this end, WELDMAP uses projective geometry and more precisely homography, based on epipolar geometry, to refine the matching. Considering this, cameras could be uncalibrated and the possible presence of outliers in the matching, fundamental matrix, and RANSAC [44] are used to verify the matchings from a geometrical point of view.

2.2. Orientation and Dense Model Generation

2.2.1. Images’ Orientation and Self-Calibration

The cameras’ orientation and self-calibration were performed hierarchically in three steps following the pipeline of Figure 1:

• First, the initialisation of the first image pair was carried out, selecting the best pair of images. To this end, a trifold criterion was established for selecting the initial image pair: (i) guarantee a good ray intersection; (ii) contain a considerable number of matching points; (iii) present a good matching points distribution along the image format. Note that initialising with a good image pair usually results in a more reliable and accurate reconstruction.
• Second, once the image pair was initialised, image triangulation was performed through the direct linear transformation (DLT) [45], taking the matching points and the camera pose provided by the fundamental matrix as input data. Afterwards, and considering this initial image pair as a reference, new images were firstly registered and then triangulated using again the DLT. The DLT allows us to estimate first the camera pose and then to triangulate matching points in a direct way, that is, without initial approximations and without calibration camera parameters. As a result, 2D–3D correspondences are provided together with the registration of the images.
• Third, although all the images were registered and triangulated based on DLT, this method suffers from limited accuracy and reliability, which could drift quickly to a non-convergent state. To cope with this problem, a bundle adjustment (BA) based on a collinearity condition [24] was applied with a threefold purpose: (i) compute registration and triangulation together and in a global way; (ii) consider the estimation of the inner parameters of the camera, self-calibration; (iii) obtain more accuracy and precision in the image orientation and self-calibration, using a non-linear iterative procedure supported by the collinearity condition that minimises the reprojection error.

As a result of this orientation and self-calibration stage, an integrated registration and triangulation are obtained which entails that all images are connected and related according to the camera pose (rotation matrix and translation vector), without forgetting the inner parameters of the camera (self-calibration).

2.2.2. Dense Model Generation

Once we had solved the orientation and self-calibration of the imagery dataset, a multi-view reconstruction approach [46] that combines stereo and shape-from-shading energies into a single optimisation scheme was used. Particularly, this method uses image gradients to transition between stereo-matching (which is more accurate at large gradients) and Lambertian shape-from-shading (which is more robust). Furthermore, the dense model generation approach uses an energy function that can be optimised efficiently, using a smooth surface representation based on bicubic patches [47], which allows for defining a surface per view that has continuous depth and normals.

2.3. Defectology Analysis

Imperfections are anomalies that occur in the welded joint. They are considered as a flaw when, due to their magnitude or location, they could compromise the integrity and/or functionality of the part and/ or cause the failure of the joint. The generation of accurate and reliable photogrammetric point clouds models allows the application of different analyses oriented to the extraction of characteristics and the detailed evaluation of the state of the bead (Figure 2). In particular, the defectology analysis was developed considering four different types of defects (references in [2]) and the limits established in the international standard [3]:

• Excess weld metal (H) (ref. 502): it is related to the height of the bead over the plaque, H. The following criterion (B level) is established: H ≥ 1 mm + 0.1 × T, T being the width of the bead (max h = 0.5 mm). If this value exceeds 10% of the bead width, a defectology warning will be generated.
• Linear misalignment (d) (ref. 5071): it is related to the different heights between plaques, d. The following criterion (B level) is established: d ≥ 0.1 × t, t being the thickness of the plaque. If this value exceeds 10% of the plaque thickness, a defectology warning will be generated.
• Angular misalignment (δ) (ref. 508): it is related to the different orientations between plaques, δ. The following criterion (C level) is established: δ [n_l − n_r] ≥ 2°, being n_l and n_r the normal vector of plaque left and right, respectively.
• End crater pipe (dz) (ref. no. 2025): it consists of the detection of holes or depressed areas in the weld beam, dz. The most efficient way to detect them is by analysing the transversal profiles resulting from the weld, establishing a threshold of dz < 0.2 t, t being the thickness of the plaque, with a maximum of 2 mm (D level).

All these defects were analysed using cross-sections along the weld, using the axis of the bead as a reference. Although the software allows the assessment of the weld from an automatic point of view, the photogrammetric point clouds allow for inspecting the main metric characteristics of the weld (quantitative analysis), as well as its visual appearance (qualitative analysis). For instance, the existence of possible friction-affected areas and/or corrosion zones can also be evaluated from a qualitative point of view.

2.3.1. Weld Segmentation and Axis Definition

For the segmentation and extraction of the weld axis, the algorithm classifies the points by computing a set of geometric attributes and minimising a globally regularised energy (Equation (1)). Particularly, different pointwise geometric attributes have been applied (i.e., orthogonal distance to plane, eigenvalues, elevation, etc.) accompanied by geometric relationships between characteristics and classes (energy functions) which have made it possible to classify the point clouds according to 3 classes: (i) left plaque; (ii) right plaque; (iii) bead. Classification is achieved by minimising an energy function over the input point cloud by selecting, for each point, the classification type that gives the best score. More precisely, an energy minimisation was proposed to classify the point cloud. The energy function E(x) is defined as a sum of partial data terms E_di(x_i) and pairwise geometric relationships defined by the standard Potts model [48], which integrates spatial coherence between neighbouring elements:

$$E\left(x\right)={\displaystyle \sum }_{i=1\dots N}{E}_{di}\left(x_i\right)+\gamma {\displaystyle \sum }_{i~j}{1}_{x_i\ne x_j}$$

(1)

where γ > 0 is the parameter of the Potts model which quantifies the regularisation force, i~j represent the pairs of neighbouring points, and $1_{\{.\}}$ is the characteristic function. The partial data term E_di(x_i) measures the coherence of the class x_i at the ith point. A Graph-Cut based algorithm [49] is applied to minimise the energy function and get a solution.

Finally, once the photogrammetric point clouds have been classified in three different classes, the axis of the bead is extracted based on the computation of centroids along the bead, using RANSAC algorithm (RANdom SAmple Consensus) for the robust fit of the axis [50].

2.3.2. Cross-Sections Generation and Defects Analysis

Once the bead and the plaques have been extracted, an automatic cross-sectional analysis can be performed using the extracted axis as reference. With this analysis, it is possible to locate flaws and imperfections according to the four defects previously defined (Figure 2). Cross-sections are plotted successively and automatically along the direction of the weld axis. The distance between sections could be freely chosen by the inspector according to the level of resolution required, the type of weld, the material, and the defects or imperfections to be detected and measured [9].

Finally, the four defects considered in Figure 2 can be checked for each cross-section deriving a complete analysis of the whole weld, as follows:

• The excess weld metal, H, was computed automatically, extracting the width of the bead based on the semantic classification of the bead along the axis and taking the median as a robust reference [51].
• The linear misalignment, d, was computed automatically, extracting the orthogonal distance between plaques based on the classification of the plaques along the axis and considering the median as a robust reference. The thickness of the weld, t, was provided by the manufacturer or measured directly in the 3D model.
• The angular misalignment, δ, was computed by applying principal component analysis (PCA) to those points in $P_l, P_r$ that belong to the left and right plaques previously classified. Particularly, the third eigenvector was estimated as a normal vector for each plaque, n_l, n_r and thus used for computing the angular misalignment, δ, between plaques. Here, a threshold value of 0.9 was empirically defined, so normal vectors with the lower vertical component were discarded.
• The presence of notches or pores, dz, was computed adjusting a plane to the plaques previously classified using RANSAC algorithm [50] and analysing the orthogonal distance from the cross-sections to the fitted plane. All those points that generate an orthogonal distance ≥ 2 mm were considered as notches or pores.

3. Results and Discussion

3.1. Study Cases

A dataset composed of different types of metallic welds and joints was used to validate the WELDMAP software. We try to cope with different welds typologies in order to adapt the tool to its demanded use. This dataset was generated with images acquired in different real scenarios (e.g., shipyard and industry) and in the laboratory using specific samples (Figure 3).

3.2. Results and Discussion

The results were obtained by applying the workflow outlined in Figure 1 and the defectology analysed, described in Figure 2, taking multiple RGB images as input data. These images were acquired using a photogrammetric stereoscopic arm composed of two reflex cameras (Canon EOS 700D) with two macro-lenses (EF-S 60 mm) (Figure 4). Trying to check the robustness of the feature extraction and matching process codified in WELDMAP, we used also images acquired with a compact handheld camera (Panasonic Lumix DMC-FZ38). In this case, images were acquired by varying their distances and perspective to reinforce the invariability of the detectors and descriptors.

Subsequently, the images were uploaded to WELDMAP. At each stage of the process (Figure 1) the final user can choose among different algorithms. In order to help the final user to choose among them, the platform incorporates integrated help for the different application cases.

In order to improve the feature extraction and matching (Figure 5), all input images were pre-processed based on the same algorithm: recursively separated and weighted histogram equalisation (RSWHE) [31], since RSWHE preserves the image brightness more accurately and produces images with better contrast enhancement.

All employed detectors were limited in the maximum number of keypoints (5000) and the used matching strategy was robust matching, supported by RANSAC using the fundamental matrix as a geometric test [25]. All the computations were performed by exploiting parallel and GPU capabilities of the hardware. This setup was enough for providing good matches between images, even in those cases with more geometric and radiometric variations between images (dataset coming from a compact handheld camera).

The best combination was obtained by experimenting with the different welds considered, so that the end user is offered this configuration.

For these cases, an MSD detector [36] combined with an SIFT descriptor [33] was the best combination. For the stereoscopic dataset, the SIFT, supported by the GPU [52], provided very good results. For these cases, the MSD detector [36] combined with SIFT descriptor [33] was the best combination. For the stereoscopic dataset, the SIFT supported by GPU [52] provided very good results.

Once images were connected based on matchings, an orientation and self-calibration of the different sets of images were performed combining triangulation supported by the DLT and the BA using the collinearity condition applied in an incremental way. Likewise, this registration process allowed us to revalidate the feature extraction and matching, since a wrong feature extraction and matching would entail an incorrect orientation and self-calibration of images. Finally, a dense model of the welds was provided based on the method described above. For the dataset coming from the compact handheld camera, the 3D models of welds were scaled based on a known distance using a codified target as reference. This was not the case for the stereoscopic dataset, since the point cloud are directly scaled based on the known and calibrated baseline. The following figure (Figure 6) outlines the photogrammetric 3D models of welds obtained, as well as the quality of the photogrammetric 3D models generated (

Table 1. Quality results of the photogrammetric 3D models obtained with the different cameras.

	Photogrammetric Stereoscopic Arm	Compact Handheld Camera
Resolution (mm)	0.05–0.06 mm	≤0.2 mm
Precision (mm)	0.10–0.15 mm	≤0.4 mm

).

As it was expected, higher quality can be observed in both spatial resolution and precision with the photogrammetric stereoscopic arm. This is due to the strong and efficient configuration defined with the stereoscopic setup in terms of images matching, photogrammetric reconstruction, and dense model generation. Furthermore, the constraint defined by the known baseline between cameras allowed us to work directly with metric models. This is not the case with the compact handheld camera, where the 3D models had to be scaled using artificial targets. On the contrary, it should be noted the greater flexibility and speed of acquisition with the compact handheld sensor, guaranteeing more forward speed and being especially suitable for non-expert operators in photogrammetry, since following a simple protocol of acquisition, the structure from motion approach can successfully solve the photogrammetric 3D model of the weld.

The resolution and precision achieved by photogrammetry improved substantially when compared to the ones achieved by the methods currently used, based on the visual inspection of welds using manual gadgets (e.g., fillet weld gauge, cam gauge). These analogical methods allow verifying if the fillet is acceptable, but they do not allow technicians to record the geometry of the bead. Furthermore, they do not provide submillimetric accuracy and they are highly dependent on the inspector’s skills and expertise. Alternative methods based on geotechnologies have been developed, for example in [53], where a laser-based machine vision system allowed for the monitoring and controlling of welding with a maximum error of 0.55 mm. In [54], a laser line scanner was employed to measure the surface profile of the weld bead. Despite it having a 0.015 mm spatial resolution, it only allowed for measuring a distance of 100 mm with a width of 24 mm. Besides, the classification between bead and base metals was unsatisfactory.

Afterwards, and based on these photogrammetric 3D models, a semantic classification of the welds using geometric features and an unsupervised classification strategy based on Graph Cut was applied, taking three classes as input: left plaque (red colour), right plaque (yellow colour) and bead (blue colour). The axis extraction is outlined in white colour (Figure 7).

It should be noted the accurate and successful results of the classification strategy based on geometric features and energy function. The energy model has a relatively simple formulation and provides convincing results in practice. Figure 7 shows the potential of the model on different types of welds. Trying to validate the results, we obtained an expert on welds who checked the results, inspecting the semantic 3D models visually.

Once the welds were semantically classified, the quality control of welds was performed based on the four defects defined (Figure 2), using cross-sections as reference. Since the process is fully automatic, cross-sections were generated with 2.5 mm resolution. Figure 8 and

Table 2. Results of the four defects assessed in WELDMAP software. In red colour are highlighted those defects which did not pass the threshold established. Only those cross-sections which produced defects are reported in the table. T is the width of the bead, t is the width of the weld, n_l and n_r the normal vectors of the right and left plaques, respectively.

Type of Defect	Criterion for the Quality Assessment (mm/°)	No. Weld	Positioning along Bead Axis x (mm)	Results
Excess Weld Metal	H ≥ 0.1T + 1	1	28.2	2.59 ≥ 1.98	not pass
		4	150.8	3.01 ≥ 1.75	not pass
		6	161.0	4.23 ≥ 3.98	not pass
Linear misalignment	d ≥ 0.1t	1	23.4	0.75 ≥ 0.70	not pass
		2	74.5	1.45 ≥ 1.20	not pass
		5	110.9	0.68 ≥ 0.50	not pass
Angular misalignment	δ [nl − nr] ≥ 2°	3	15.3	2.5	not pass
		3	90.6	2.7	not pass
		3	115.4	3.0	not pass
Notch or pore	dz ≥ 2 mm	1	12.3	2.3	not pass
		4	60.6	2.8	not pass
		5	116.3	3.2	not pass

show the results of the defectological analysis of the welds assessed. Only those welds which present a defect are outlined in Table 2.

In view of Table 2, the efficiency of the WELDMAP tool to detect anomalies automatically is clear. It should be noted that the tool allows you to configure the threshold of defects as well as their percentage to consider if a weld should be rejected. In this sense, it must be emphasised that the welds inspected in situ did not provide any defectology, while the welds analysed in the laboratory (no. 1, 2, 3, 4, 5, 6), which came from defective welds, were those that showed the pathologies outlined in Table 2.

4. Conclusions

In this article, an unprecedented tool has been presented that integrates the possibility of providing smart photogrammetric models based on the quality control of welds. Under the name WELDMAP, the ability to reconstruct 3D welds, both geometrically and radiometrically, allow us to perform external quality control according to normalised standards. More specifically, this article proposes the use of RGB images acquired with reflex cameras as a non-destructive and low-cost technique for the digitisation of welded joints and the automation of their quality control by means of an in-house software, successfully tested in different scenarios and with different types of welds. The results come to support its usefulness, since today these quality control tasks have been carried out visually supported by expeditious measures, and in most cases with an analogue record.

Looking ahead, it is expected to be able to work with images from other wavelengths (e.g., thermography, X-rays) and new materials (e.g., composites, fibres, etc.), so that multimodal registration techniques have to be incorporated together with the application of artificial intelligence techniques for quality inspection tasks. So, further developments will be carried out to include additional types of defects.