Integration of Thermal and RGB Data Obtained by Means of a Drone for Interdisciplinary Inventory

Abstract

Thermal infrared imagery is very much gaining in importance in the diagnosis of energy losses in cultural heritage through non-destructive measurement methods. Hence, owing to the fact that it is a very innovative and, above all, safe solution, it is possible to determine the condition of the building, locate places exposed to thermal escape, and plan actions to improve the condition of the facility. The presented work is devoted to the technology of creating a dense point cloud and a 3D model, based on data obtained from UAV. It has been shown that it is possible to build a 3D point model based on thermograms with the specified accuracy by using thermal measurement marks and the dense matching method. The results achieved in this way were compared and, as the result of this work, the model obtained from color photos was integrated with the point cloud created on the basis of the thermal images. The discussed approach exploits measurement data obtained with three independent devices (tools/appliances): a Matrice 300 RTK drone (courtesy of NaviGate); a Phantom 4 PRO drone; and a KT-165 thermal imaging camera. A stone church located in the southern part of Poland was chosen as the measuring object.

1. Introduction

The research purpose was to investigate the application of thermal and visual data integration for a heritage object in order to determine areas of energy losses. The discussed approach exploits measurement data obtained with three independent data acquisition platforms: a Matrice 300 RTK drone; a Phantom 4 PRO drone; and a KT-165 thermal imaging camera. The parish church of the Birth of the Blessed Virgin Mary in the town of Porąbka was selected as the subject of the study. The facility was built of stone, has two towers, and its history dates back 115 years. The first and most interesting sensor used for the measurement was a Zenmuse H20T thermal imaging camera mounted on DJI unmanned aerial vehicle (UAV) Matrice 300 RTK, used courtesy of NaviGate. The RGB photos were captured by two cameras on the Matrice 300 RTK, and a standard 20Mpix camera on the second UAV used for the project, which was the Phantom 4 PRO. Additionally, the device used during the measurement was the KT-165 thermal imaging camera, on the basis of which a thermal analysis of the entire object was performed. As the final product, four independent point clouds and four independent models were obtained, which were further analyzed. The first point cloud was from the RGB camera on the Phantom 4 PRO, the second point cloud was from the RGB “wide” camera on the Matrice 300 RTK, the third point cloud was from the RGB “wide” and “zoom” cameras on the Matrice 300 RTK, and the fourth point cloud was from a thermal image taken by the Zenmuse H20T camera on the Matrice 300 RTK. The obtained products were assessed, not only in terms of quality, but also their usability and the difficulty of development. Thanks to thermal analysis it was possible to determine the places of thermal escape. The thermal point cloud deserved the most attention, followed by the model obtained on the basis of thermal images from the Matrice 300 RTK. The final stage of the work was data integration, i.e., coloring the 3D model obtained from the Phantom 4 PRO drone with a thermal point cloud from the Matrice 300 RTK.

1.1. Thermography

Thermography is wide remote-sensing data acquisition and processing technology, often applying photogrammetric methods of measurement and results presentation. The thermal sensors are used in different scales and from different distances: from close-range [1,2] that can also be mounted on robots and integrated with other sensors [3], on UAV [4,5], aircrafts or satellites [6]. Thermal information can be captured by different means: by passive methods like typical IRT (infrared thermography) [7,8,9], and active methods such as pulse IRT [10,11] and lock-in thermography [12,13]. The thermal acquisition methods can also be divided according to: mono thermal sensors; multispectral cameras [5,14,15]; or multisensor systems [4,16]. The thermography is used for several objects and purposes. The most common is building thermography [17,18,19], thermal diagnostics of buildings [13,18], humanitarian mine action (HMA) [20,21,22], and photovoltaic inspection [23], heat and air conditioning loss detection [24] or, on a bigger scale, thermal monitoring of urban areas [6]. The largest area of thermography application is heritage object sensing, controlling and monitoring for preservation purposes, especially heritage buildings [9], fresco detachment sizing [1,8], moisture detection [7,19], archeological site sensing [25], and analyzing other different cultural heritage artefacts [10]. Thermography is also applied for crime scene analyses [16], wildlife [26], vineyard condition sensing [27], tree species classification [15], or even flying UAV detection [28], not to mention military applications too.

1.2. UAV Thermography Application

The development of the application of UAV technology to different remote sensing and mapping technologies can also be applied to thermography. Nowadays, UAVs are able to carry different sensors used for thermal acquisition: near-infrared cameras [29], thermal cameras [25,30,31,32,33,34,35,36], but also multispectral cameras [14,27,37,38] and multimodal platforms with RGB, thermal and depth cameras [39]. In addition, thermal sensing can be executed in complex objects with limited access, thanks to UAV flight flexibility; thermal data can be acquired from the sides without sufficient visibility from the ground view or aircraft operation height. The new applications of UAV thermal sensing are presented in several areas: archaeological sites [25,39], heritage buildings [30,33], and modern buildings [31,34], including photovoltaic inspections [23]. UAV thermography is also applied for thermographic DTM generation [25,32], and crop field sensing [37]. In particular, it is used for: precision farming [14], soil moisture [5], leaf condition monitoring [29], and other earth observations: permafrost sensing [35], volcano activity monitoring [36], and even robust person detection [40].

1.3. Thermographic and Other Mapping Data Integration

Integration of thermography with other mapping technologies is increasing its possibilities. There are many positive research results presented in world scientific papers regarding the fusion of methods and fusion of data, where one applicationis thermogram. To systematize all integration solutions, we propose two criteria of division of methods.

The first criterion is the source of the data being integrated with thermal data. There are three basic groups of integration method:

(1)

Thermographic and LiDAR data integration are the first important group of methods presented in the papers [41,42,43,44] and mobile laser scanning (MLS) [45]. Usually, the thermographic point cloud is obtained by colorizing original LiDAR data by thermogram RGB values [46]. Another method is the integration of two-point clouds: a LiDAR and thermographic point cloud as a result of dense matching [47].

(2)

Thermographic and photogrammetric data integration are the second group, in particular using RGB sensors [40,47,48,49,50,51,52], as well as smartphone sensors [53], depth cameras [54,55], RGB visual odometry data [56], multispectral [2,4,38], and hyperspectral data [16]. The presented research is another example of this group.

(3)

The third group, according to this criterion, is thermographic data integration within more complex platforms including different sensors [57,58,59,60,61,62], also mounted on robots [3].

The second criterion of division of different approaches to thermal and other mapping data integration is the method of co-registration in a common coordinate system (or external orientation) of the two datasets to be integrated. Most of the published approaches can be divided into four groups of solutions: calibrated fixed sensors, co-registration by control points and features, point cloud superimposing, and other methods.

(1)

Calibrated fixed sensors as thermal and RGB cameras have known spatial relations between both sets of images thanks to sensor calibration. The RGB images have higher resolution, and they are used for 3D model generation [48]. The accuracy of thermal texture projection on the terrestrial laser scanning (TLS) point cloud is higher [57]. There are successful examples of multi-sensor platforms, where RGB, infrared (IR) and TLS sensors were calibrated together, and a thermally colorized TLS point cloud is obtained [58]. There is even an example of a special robot, designed and developed with a laser scanner, thermal and RGB camera for automated inspection of the buildings [63].

(2)

Another solution is the co-registration of visual or LiDAR data with thermal data by specially selected or automatically found natural or marked points and other features. Points marked by coins can be used for thermal and RGB data fusion as well multispectral and RGB data [2]. Another method of marking the thermal control point is by using of circles of aluminum foil [64]. Between thermal imagery and LiDAR point clouds natural points can be used, selected a priori and traditionally measured by tachymeter [59]. Control points are also used for two-point clouds resulting from dense matching of RGB and IR images [65,66]. In spite of the points, linear features are used for co-registration. Feature extraction of visual and thermal images can be done with the Feature Accelerated Segment Test (FAST) operator and two datasets can be matched by correlation [46].

(3)

The third solution is RGB and thermal point cloud superimposing. The fast global registration (FGR) algorithm can be used for coarse registration of two-point clouds for further refinement of intrinsic and external orientation parameters of thermal images, i.e., before thermal precise texture generation [47]. The iterative closest point (ICP) algorithm is also used for visual and IR point cloud co-registration [67]. Another solution of IR data registration is based on superposition of Structure from Motion (SfM) point cloud with TLS data.

(4)

The last group of solutions are more unique approaches. For instance, the sequence of IR images are matched and GPS/INS data are refined for more precise building model thermal texturing [68]. A different example of registration of thermal data with 3D data is RANSAC-based Efficient Perspective-n-Point (EPnP) registration, which can be applied for fusion of range camera 3D data and 2D thermal data [69].

1.4. 2D and 3D Application of Thermal Data

Thermal data are applied as a thermal image, thermal texture of 3D model or thermal point cloud from Structure from Motion (SfM) or LiDAR. The data are presented and used as planar feature or spatial virtual model.

Planar features are rectified thermograms by projective transformation usually for further application: as images for automatic detection of thermal bridges in buildings [70] or for detection of pathologies in monuments [9]. Another way to obtain planar thermal images, is the generation of thermal orthoimages from thermally textured 3D models of the building [41,71].

Spatial virtual models can be an object of analysis or as products and virtual models presenting results of thermal imaging.

(1)

The objects of analysis are stereo models acquired by stereo thermal cameras, for instance, for pedestrian counting [72] or for integration with other data of robot displacement for refinement of robot (or sensor platform) position and precise 3D point cloud calculation [73].

(2)

A thermal virtual model can be a result of different processes of thermal data. The RGB data can be matched, and a 3D triangular irregular network (TIN) model can be generated and thermally textured by registered images of an RGB and IRT camera [74]. Another approach is based on matching and adjustment of thermal and visual UAV images; the resulting point cloud can be a source of thermal orthophotos [25,71]; the artificial targets are used to register a dense surface model of matched thermograms. A 3D model can be also generated from high resolution laser scanning and low-resolution thermal imaging, which are able to be successfully integrated by convolutional neural networks [75]. The other 3D thermographic objects are results of thermal texturing of existing 3D models [76] or 3D mapping of multispectral images on a TLS point cloud [57].

The application of dense matching of the thermal imagery is more common at this time. The thermal point clouds from the photogrammetric level were applied for city thermal analysis and geometrically compared with LiDAR data [77]. On a bigger scale, the mobile mapping infrared data were used for building mapping and dense point cloud generation [78]. The UAV thermal and RGB fusion leading to thermal map modeling is an important research subject [79,80]. The methods of dense matching of thermal imagery are applied to deliver information to BIM about thermal conditions of the interior of buildings [60]. The stereo matching of thermal images is used even for small objects and their dynamic thermal measurement [81]. In this context our experiments and approach are very contemporary. We present thermal and RGB data fusion by thermal 3D mapping of monumental object. The usage of combined RGB and thermal data for heritage and complex objects and the novel method of marking the control points by thermal active points is our proposal to develop the methods and fill the gap in existing knowledge.

2. Materials and Methods

2.1. Measuring Equipment

One of the newest and best flying platforms from the DJI Enterprise (Shenzhen, China) series, Matrice 300 RTK (Figure 1a) was used to perform the measurement. The device was loaned for the duration of the work, courtesy of NaviGate (Krakow, Poland). The drone is equipped with a Zenmuse H20T thermal imaging camera, a so-called hybrid, which has a thermal imaging camera, two daylight cameras and a laser rangefinder in one housing. The thermal imaging camera has two important functionalities: it allows for the targeting of a specific point and a determination of its temperature value in real time, but it can also determine the temperature difference on the tested object. The elements of the environment are transferred to the user with the help of the appropriate color scale, depending on their temperature [82]. The temperature range of the camera is from −40 to 150 degrees Celsius (High Gain) and from −40 to 550 degrees Celsius (Low Gain). The H20T camera has a 23× optical zoom and a maximum optical + digital zoom of 200×. The device allows simultaneous recording and photos from three cameras to three separate files [82]. An additional device used for the measurement was the KT-165 (Swidnica, Poland) hand-held thermal imaging camera (Figure 1c) with a spectral range of 8–14 μm. The accuracy of the reading is 2 degrees Celsius or 2% of the reading for an ambient temperature between 15 and 35 degrees Celsius and an object temperature above 0 degrees Celsius [83]. In order to compare and generate another RGB model, the DJI Phantom 4 PRO unmanned aerial vehicle (Figure 1b) was used, the most frequently selected and the most optimal device in terms of value for money. The unmanned aerial vehicle has a built-in camera with a one-inch CMOS matrix with a resolution of 20 Mpix. It is the first camera to use a new mechanical rolling shutter system [84].

2.2. Measurement Marks

Disposable, chemical, non-toxic and non-flammable hand warmers were used as markers for the thermal images. The vendor ensures about 8 to 10 h of heat release with an average temperature of 57 degrees Celsius. The maximum temperature for the heaters is circa 69 degrees Celsius while maintaining appropriate environmental conditions. Before the measurement, a check was performed to confirm how the product would behave when left in an open space and in close contact with a material that absorbs moisture and cold (in this case stone). The results turned out to be satisfactory. Despite being exposed to cold, the hand warmer was still visible on the cold surface. After this stage, the preparation of the photogrammetric control network, which also served as a thermovision control network, began. The bags with the heating substance were shaken to start the heating process by interaction with the air. In order to keep the heat longer, each hand warmer was attached to a small wooden board with a tacker. Thus, it was isolated from the stone. A measuring disc in the form of a checkerboard was glued to each element prepared in this way (Figure 2). This process was to enable precise targeting of the total station at its center. Prepared measuring points were fixed with tape on the church supports (Figure 3). In the end, 16 pieces were placed on the supports only, 5 on the ground and 2 on the church tower. To sum up, there are 23 points of the photogrammetric control network on the entire surface of the church, acting as a thermal matrix. As additional aids, 5 heaters were also scattered on the ground around the church. Their coordinates were determined, but they were only used as control points.

The next stage was the layout of the tacheometric stations around the church. The task required high precision, because at the same time it was necessary to keep the vision between the stations and to measure each point of the photogrammetric control network acting as a thermovision control network from at least two stations. Due to the use of Real Time Network (RTN) technology, the obscuration of the horizon could not be ignored. Six tacheometric stations were created from which the measurement was made (Figure 4). In addition, characteristic pickets at the points of the church, such as the edges of the walls, cornices and the tower, were measured. It was of great importance for the georeference of the created model, especially for the Z axis.

All control network was adjusted, and the errors were about ±0.003 m for XY coordinates and ±0.005 m for Z coordinates.

2.3. Data Acquisition

The most important and necessary element, without which measurement would not be possible, was to make sure that the church was heated during the winter season. Obtaining correct measurement results is possible only and exclusively with a significant temperature difference between the object and its surroundings. The data acquisition process was started from inside the church. The thermometer showed 13 degrees Celsius inside the building, while −5 degrees Celsius was recorded outside. Initially, the weather was favorable for the measurement: it was cloudy and only around noon the first rays of sunlight broke through. The measurement of the interior started from the attic, then moved lower and lower, through the choir, to the part intended for the faithful and church service: the main aisle and sacristy. In the attic, the temperature on the walls and ceiling varied between 2 and 3 degrees Celsius. It was initially concluded that the attic was relatively well insulated. Initial analysis of the photos showed that the greatest heat escape was taking place through doors and windows. Then, photogrammetric raids were carried out with two DJI drones, the Matrice 300 RTK with the Zenmuse H20T camera and the Phantom 4 PRO. The unmanned aerial vehicles took pictures from the lowest floor to the tower itself, moving around the church, climbing a certain distance up after each “round” (Figure 5). At the same time, Matrice took 3 photos: one thermal and two RGB photos: “wide” + “zoom” and saved them to three separate files.

2.4. RGB Images Analysis and Preperation

The photos taken from the measurement with the unmanned aerial vehicle provided information on all camera parameters at the time of taking the photo: aperture, exposure time, focal length, and flash llama mode. In addition, data such as file size and GPS geographic coordinates and altitude were also obtained. Before starting the development of the 3D model, it was necessary to review the photos in terms of sharpening and exposure. Overexposed photos and photos with flare (halations) have been removed. The images obtained from the cameras were neither cropped nor geometrically transformed.

2.5. Analysis of Images from the KT-165 Camera

The analysis of the photos showed that the average temperature of the walls and ceiling in the church attic is around 3 degrees Celsius. Inside the temple, and precisely in the nave, the temperature of the walls was close to 18 degrees Celsius, which was probably largely related to the heaters under the side pews (temperature around 50 degrees Celsius) (Figure 6). The church windows facing the sun had a temperature of about 16 degrees Celsius, and those facing the north were 2 degrees Celsius lower. The temperature of the entrance door was about 11 degrees Celsius (Figure 7), and the sacristy door as much as 18 degrees Celsius (due to sunlight from the outside).

2.6. Model Generation

During the measurement work, four independent datasets were obtained: RGB photos and thermal photos gathered by the Matrice 300 RTK drone raid (zoom and wide photos), photos from the Phantom 4 PRO and from the KT-165 hand-held thermal imaging camera, which were to be used to monitor thermal changes in buildings. In order to generate 3D models from each set of images, it was necessary to use dense matching software. Agisoft Metashape was chosen. Several stages of the point cloud calculation proceeded as follows: thermal images conversion, images import, photo alignment, and internal and external orientation parameter determination. In addition, the control points were measured on the photos. There are five types of accuracy to choose from: highest, high, medium, low, and lowest. The size of the search area and the size of the resulting image will depend on the choice of accuracy parameter. Another important parameter is a method of reference preselection: source, estimated, and sequential.

The data have been divided into four separate projects for model generation. All processes were repeated with all combination of accuracy and reference preselection parameters. The photos obtained from the DJI Phantom 4 PRO unmanned aerial vehicle (678 photos) were aligned and the results of the alignments are attached in

Table 1. DJI Phantom 4 PRO photos alignment results.

Accuracy	Reference Preselection	# Of Tie Points	Reprojection Error [pix]
Lowest	Estimated	30,195	4.50
Lowest	Sequential	4106	3.66
Lowest	Source	22,010	4.16
Low	Estimated	300,904	2.09
Low	Sequential	74,742	2.31
Low	Source	260,431	1.96
Medium	Estimated	612,148	0.93
Medium	Sequential	171,235	0.80
Medium	Source	591,379	0.90
High	Estimated	636,723	0.61 *
High	Sequential	180,557	0.46
High	Source	662,534	0.59
Highest	Estimated	660,190	0.40
Highest	Sequential	165,045	0.35
Highest	Source	667,900	0.39

* The variant chosen for dense matching.

: quantity of tie points and reprojection errors. In the other three projects, materials from the DJI Matrice 300 RTK drone were developed: in one, thermal images; in the second, wide camera photos (1819 photos); and in the third, wide camera photos, thickened with zoom photos (3642). The wide camera variants of photo alignment are shown in

Table 2. DJI Matrice 300 RTK wide photos alignment results.

Accuracy	Reference Preselection	# Of Tie Points	Reprojection Error [pix]
Lowest	Estimated	3930	6.49
Lowest	Sequential	9752	7.47
Lowest	Source	12,840	7.01
Low	Estimated	373,994	big error
Low	Sequential	151,782	2.73
Low	Source	385,430	big error
Medium	Estimated	669,967	big error
Medium	Sequential	309,531	1.4
Medium	Source	754,741	33.9
High	Estimated	546,063	1.11 *
High	Sequential	282,059	36.32
High	Source	741,908	big error

* The variant chosen for dense matching.

and wide + zoom variants are shown in

Table 3. DJI Matrice 300 RTK wide + zoom photos alignment results.

Accuracy	Reference Preselection	# Of Tie Points	Reprojection Error [pix]
Lowest	Estimated	10,702	3.07
Lowest	Sequential	25,480	big error
Lowest	Source	71,725	4.28
Low	Estimated	508,348	2.48
Low	Sequential	625,091	big error
Low	Source	1,151,574	big error
Medium	Estimated	no result	no result
Medium	Sequential	619,052	1.72
Medium	Source	897,047	1.76
High	Estimated	no result	no result
High	Sequential	989,969	1.52 *
High	Source	618,178	big error

* The variant chosen for dense matching.

similarly to Table 1. The variants chosen for dense matching were marked in the tables.

The most difficult task turned out to be the alignment of the thermal images (1670 photos). Trying to fit all the photos at once did not bring about the expected results. Marking the points manually did not help either. In this situation, it was decided to divide the photos into parts according to the height of the raid: the lowest story near the ground, the height of the windows, the height of the cornices, the roof, and the church towers.

Table 4. DJI Matrice 300 RTK thermal photos alignment results.

Part	Accuracy	Reference Preselection	# Of Tie Points	Reprojection Error [pix]
1	High	Estimated	7459	0.56
	High	Sequential	8069	0.58
	High	Source	8725	0.61
	Highest	Estimated	16,384	0.47 *
	Highest	Sequential	20,137	0.51
	Highest	Source	8414	0.50
2	High	Estimated	41,380	0.63
	High	Sequential	32,882	0.59
	High	Source	27,107	0.55
	Highest	Estimated	110,462	0.50 *
	Highest	Sequential	101,471	0.49
	Highest	Source	24,027	0.40
3	High	Estimated	10,304	0.49
	High	Sequential	24,031	0.48
	High	Source	21,868	0.50
	Highest	Estimated	39,684	0.45 *
	Highest	Sequential	56,546	0.39
	Highest	Source	28,875	0.39
4	High	Estimated	52,008	0.58
	High	Sequential	56,125	0.60
	High	Source	42,149	0.71
	Highest	Estimated	78,286	0.47
	Highest	Sequential	45,541	1.84 *
	Highest	Source	31,384	0.48
5	High	Estimated	821	1.18
	High	Sequential	736	1.34
	High	Source	814	1.30
	Highest	Estimated	1352	0.78
	Highest	Sequential	1933	0.86 *
	Highest	Source	4263	0.46

* The variants chosen for dense matching and merging.

shows the results of all five parts of thermal photo alignment similarly to Table 1, Table 2 and Table 3. Each group was fitted individually, and the missed photos were manually adjusted. In the end, all the parts were put together into one coherent whole using the control points. Then, the cloud was cleared of unnecessary points and on the data prepared in this way, the construction of a dense point cloud based on the elements of internal and external orientation was initiated.

The entire process of creating a dense point cloud is based on depth maps. A well-prepared point cloud is the basis for generating a mesh model. Model resolution presents the field resolution of the developed object models. Special attention should be paid to the fact that the cloud is properly cleaned, prepared, and oriented before starting the model generation. Without this step, the model will turn out to be useless after generation, as all imperfections from the cloud will be transferred to it. With a well-prepared point cloud, generating the model comes down only to running the algorithm. Several improvements can be made as part of the possible improvement of the program work result. One of them is “hole filling”. It is used to ensure the continuity of the model despite the lack of data for specific fragments. In the project, it was mainly important for the thermal imaging model, which had a lot of imperfections after being generated, and also for the small church tower in RGB. Gaps and discontinuities in the model structure are removed by interpolating the color and course of the model based on the surrounding area. Holes in the model may also appear during its cleaning because the standard algorithm tries to ensure that the product created is continuous, so each removal of model fragments will result in a hole being created. Agisoft Metashape lacks a tool to select an area of interest before photo alignment and cut out only one specific part, without disturbing the structure of the others.

Table 5. Dense clouds parameters.

Model	GSD [mm/pix]	RMSE X [m]	RMSE Y [m]	RMSE Z [m]	# Of Points
Phantom 4 PRO	3.12	0.003	0.004	0.006	487,900
“Wide” photos	3.91	0.003	0.004	0.005	546,063
“Zoom + wide” photos	3.67	0.005	0.005	0.006	712,692
Thermovision part 1		0.007	0.011	0.013	16,384
Thermovision part 2		0.044	0.041	0.041	101,471
Thermovision part 3	9.64	0.006	0.007	0.013	56,546
Thermovision part 4		0.010	0.012	0.014	78,286
Thermovision part 5		0.028	0.099	0.182	40,071

presents the parameters of individual dense point clouds. The GSD, RMSE on control points and quantity of the dense point clouds are presented.

3. Results and Discussion

3.1. Comparison of Point Clouds and Models

In order to compare between dense point clouds, they had to be exported in .obj format and loaded into CloudCompare. The first two clouds that played the greatest role in the project were selected for analysis. It was a thermal cloud and the one obtained with the DJI Phantom 4 PRO drone, which was the most detailed and accurate. At first sight, it can be stated that the thermal imaging cloud is much thinner, which results from the fact that the resolution of the near-infrared camera matrix is smaller, which naturally has to result in a reduced density and number of points. It is characterized by single discontinuities resulting from the lack of a sufficient number of solutions. A comparative analysis of both products, in terms of accuracy and precision, confirms the hypothesis about the need to use a highly accurate RGB cloud. Despite the lack of precise positioning, the spatial orientation of the color point cloud is definitely better. To be precise, the distance is within 10 cm (Figure 8). The analysis of errors on ground control points leads to the conclusion that this is not a uniform error on both clouds, and it mainly results from the thermal imaging cloud. The much lower resolution caused difficulties in the analysis and the course of the algorithm of fitting photos, which resulted in an increase in mapping errors, and as a result, the entire study. However, it made it possible to create a coherent thermographic model of the church building in Porąbka, which was only used to color the model created from RGB photos.

Taking advantage of the fact that the DJI Matrice 300 RTK drone is equipped with a H20T camera, each of the captured thermal frames has its color equivalent, in two formats. One of them is a JPEG image from a fixed focus color camera. The second, on the other hand, was created with the help of a third camera with the ability to change the zoom. At the post-production stage of the obtained data, models were created that used either only wide-angle photos or a set of all photos (including “zoom” and “wide” cameras). The purpose of the second comparison was to check whether increasing the number of photos of the same object would cause a large change in the accuracy and precision of the developed clouds. It is possible to see the biggest difference at first glance. It is the number of generated points and the final size of the created file, which in the case of a point cloud consisting of stereoscopic alignment of over 3500 high-resolution photos, was over 100 GB. This is a significant increase compared to the second product, which is considered a disadvantage. The comparative analysis shows that there are practically no deviations in the horizontal position between the two products. On one side of the church, it is a centimeter towards the thinner cloud, on the other 1.5 cm towards the opposite side (Figure 9).

Each of the compared models was obtained from a different camera. They differed primarily in resolution, but also their performance depended on the prevailing weather conditions, the surface of the photographed object, stabilization of the unmanned aerial vehicle, and the skills of its operator. Submitting RGB photos (with both the Phantom 4 PRO and Matrice 300 RTK), generating a dense cloud and then a mesh on their basis was not problematic at all compared to thermal imagery. RGB mesh models perfectly presented the entire structure, practically fully reflecting what was built by human hands 115 years previously. The models have kept the smallest details, such as the portal above the entrance door, the stained-glass windows, and the clock face on the large tower. All mesh models were generated by Agisoft Metashape. The model obtained from the photos with the Phantom 4 PRO drone presented the best quality (Figure 10). Despite the fact that it was built in the most difficult weather conditions (the sun broke through the clouds during the work), almost all elements of the building have been preserved. The difference is only visible between the south and north sides of the church. The elements that still need to be improved are the crosses on both towers. Due to the turbulence of air between the towers, it was difficult to operate a light and small machine. An attempt to photograph the crosses from a closer distance could fail and risk destroying the object. In the case of the model obtained on the basis of “wide” photos from the Matrice 300 RTK and the “wide” model thickened with “zoom” photos, a much better representation of the tower and the crosses placed on them was observed. The stained-glass windows also looked great. More photos certainly had a positive effect on refining more details, but it was not the optimal solution. Compared to RGB models, the church’s thermal imaging model fared much worse. Significant problems with matching the photos caused numerous shortcomings in the form of “holes” in the object, which could be supplemented with the use of an additional tool in the program. Nevertheless, the obtained model gives a sufficient effect for the thermal analysis of the entire object. No church tower has been fully photographed in this way. An important advantage of the study is the properly preserved structure, colors, and texture coverage (Figure 11).

3.2. Data Integration

The initial scenario assumed the use of a model made of RGB photos as a structure, which are then color-coded according to the information from the thermal imaging. This stage of the project began with selecting the best RGB dataset. During the post-processing of the data, a significant number of point clouds and mesh models in various formats were generated in order to perform the integration. The main problem, however, was further defining the procedure leading to the combination and use of the positive aspects of each of the kits. Solutions were searched for using Agisoft Metashape, CloudCompare, and MeshLab. Ultimately, the mission was successful using the last one. The first problem was that the objects loaded into the program were not in the same place, although they were both supposed to have compatible coordinate systems. It turned out that the program cannot handle data with 7 digits before the decimal point (system 2000, zone 6).

The processed data were exported (again in all formats and types) from Agisoft Metashape, taking into account the shift of the origin of the coordinate system to the point X: 6587700, Y: 5521100, and Z: 300. This operation reduced the coordinates of all points and the entire model to two- or three-digit values. As a result of this operation, the first element of integration was achieved in MeshLab. It was a set of two models of the same object with different contents, but compatible in terms of orientation and position.

Then the RGB model was stripped of color information and an attempt was made to migrate textures from one object to another. After many unsuccessful attempts (including creating a texture in Agisoft Metashape), it was decided to use a thermal imaging point cloud. After a few transformations and preparation steps, the point attributes were transferred from the cloud to the vertices of the RGB model. Interpolation was necessary. This is directly related to the fact that the RGB model was several dozen times more detailed, so the 1:1 projection was not possible, as there were no corresponding points on both objects. The operation was successful. The vertices of the RGB model were now colored according to the values of the thermal imaging (Figure 12). In this way, the thermal vision of the point cloud was integrated with the three-dimensional RGB model of the church building in Porąbka.

4. Conclusions

The integration of RGB data with thermal imaging allows for much greater use of the potential of the model created from thermograms. The thermal imaging camera in its assumption has a relatively low resolution compared to the real color imaging sensor, the resolution of which is as much as 64 times higher. This means that the target point cloud (and model) is much more accurate, and the GSD (pixel size in the field) is about 10 times smaller. Therefore, the combination of both datasets allows for very good results. Thanks to this, it is possible to carry out a more accurate and detailed thermal analysis of the facility and to determine the heat escape route from the building. Unfortunately, the solution is not cheap and requires a lot of work. First of all, to perform such complex calculations a very advanced unit of computing is required: a 12-core (24 threads) AMD Ryzen 9 3900X processor clocked at 4.4 GHz was used for data processing. In short, it is a device designed to perform a large number of parallel and heavy operations. The computer is equipped with a graphics card based on the NVIDIA RTX 2080 Super chipset. It is a card with 12 GB of built-in memory and GDDR6 transmission standard. In addition, up to 32 GB of RAM memory (random-access memory) was used. In addition, very expensive measuring equipment is required. Both the DJI Matrice 300 RTK drone and the hand-held thermal imager are quite an expensive solution. The Matrice 300 RTK unmanned aerial vehicle with a thermal imaging camera requires exceptional flying skills from the operator, as it is a very large machine that cannot be easily operated in the airspace. Additionally, the ability to process data is also not a simple challenge. During the post-processing of the obtained data, it was repeatedly analyzed whether it was possible to create a dense cloud of points and a model from such acquired photos. Regardless of this, however, the original assumption of the project was achieved. As a result of the conducted research, it has been proven that the integration of thermovision and photogrammetric data is not only possible, but also very useful. The model by itself, made of thermal images, leaves much to be desired. Only thanks to integration, can the data be put to use. Nevertheless, the advantages of such a procedure are very promising. A significant increase in the resolution, accuracy, and precision of the created model, along with leaving information about the thermal imaging parameters of the measured object, is the main advantage of using such a huge amount of data and implementing integration.

The novelty of the proposed approach is the integration of thermal and RGB data and the generation of a 3D thermal model of a very complex and tall object. Innovative active thermal signs of the photogrammetric control network were prepared and applied. In addition, obtaining data by a large drone, such as the Matrice 300 RTK, is very difficult for historic, valuable, and complicated objects. The solution is dedicated to thermal monitoring of historic buildings. Additional manual thermal imaging has been added to perform a thermal analysis of the object. Thus, the following conclusions could be drawn: no significant escape was observed in the attic and church roof thermal. This is probably related to the recent renovation of the temple’s covering and its cover additional warming. For windows, especially such large ones, it can never be guaranteed to be completely insulated, as glass is generally more susceptible to sunlight and transmits heat to the interior of the object.

The presented approach is a valuable source of information on the energetic characteristic aspects of the buildings. In this paper, the method of thermal and visual data integration is presented with the example of a heritage object, yet the method is universal, and it can be applied to any kind of object, in which areas of energy losses have to be determined. Each technology has its advantages and disadvantages, and the real art is the ability to use the positive aspects and combine them with each other to create even more advanced and complex imagery.