Machine Learning-Based Monitoring for Planning Climate-Resilient Conservation of Built Heritage

Abstract

The increasing frequency and intensity of extreme weather events are accelerating the mechanisms of surface degradation of heritage buildings, and it is therefore appropriate to find automatic techniques to reduce the time and cost of monitoring and to support their planned conservation. A fully automated approach is presented here for the segmentation and classification of the architectural elements that make up one of the façades of Palazzo Pitti. The aim of this analysis is to provide tools for a more detailed assessment of the risk of detachment of parts of the pietraforte sandstone elements. Machine learning techniques were applied for the segmentation and classification of information from a DEM obtained via a photogrammetric drone survey. An unsupervised geometry-based classification of the segmented objects was performed using K-means for identifying the most vulnerable elements according to their shapes. The results were validated through comparing them with those obtained via manual segmentation and classification, as well as with studies carried out by experts in the field. The initial results, which can be integrated with non-geometric information, show the usefulness of drone surveys in the context of automatic monitoring of heritage buildings.

1. Introduction

The current role attributed to cultural heritage as a founding value of society, coupled with the corresponding attention devoted to its protection, has given rise to a convergence of two trends. On the one hand, over the past century, there has been a growing awareness of the role played by cultural heritage in modern societies, which has in turn expanded the amount of assets to be protected. Conversely, the progressive widening of heritage recognition to ever broader categories of tangible and intangible assets necessitates the study of a wider range of processes for their documentation and protection [1].

On the other hand, awareness of the direct impact of human activities on cultural heritage has increased considerably. The very act of recognising the cultural value of a cultural asset can, in some cases, prove to be a catalyst for its destruction, as demonstrated by the impact of overtourism and deliberate destruction during conflicts and wars to dismantle the cultural identity of adversaries [2,3].

Therefore, alterations to environmental conditions caused by human activities can produce indirect, pervasive, and continuous damage [4]. Such results are determined by degradation mechanisms induced via pollutants, which have already been studied for a long time, and also by climate change effects, as revealed in recent studies [5].

Among the climate change-related factors, it is worth noticing that the increased intensity and frequency of extreme climatic events may accelerate the natural processes of alteration and degradation of the materials of artefacts that were originally designed to last in different environmental conditions. Furthermore, extreme climatic events are often highly correlated with an increase of landslide or flood risk, affecting not only cultural heritage but also large parts of the territory.

The creation of digital documentation of cultural heritage is a complex undertaking. The application of high-resolution geomatic sampling techniques has resulted in the generation of a vast quantity of spatial data in a relatively short period of time. The accuracy and quality of the data are sufficient and often higher than required in most applications. However, metric data represent only the initial material from which further processes, such as the production of CAD drawings, 3D models, GIS, BIM, and so forth, are derived. The generation of such outcomes and their subsequent interpretation in order to obtain useful information for the understanding and more effective management of cultural heritage still necessitates the intervention of specialists. Consequently, it is beneficial to identify processes that facilitate the management, processing, and interpretation of digital data, particularly in the context of large and complex buildings, archaeological areas, and cultural sites.

This study presents the initial findings of a research project investigating the potential of automated spatial data segmentation to identify the most vulnerable areas of the façades of the Palazzo Pitti in Florence, Italy. This is part of a larger project concerning the potential detachment of stone elements due to the characteristics of the material used for the construction of the building. Such an occurrence may have particularly grave consequences in a context with a high number of tourists, such as that of Palazzo Pitti. For this reason, the façades are periodically monitored through direct inspection by specialists.

In 2020, a complete survey of Palazzo Pitti (described in more detail in Section 3) was carried out using integrated geomatic techniques. The survey results were employed to generate computer-aided design (CAD) drawings like plans, elevations, and cross-sections. Subsequently, all the stone elements of the façades were manually segmented and classified, with the aim of creating a geographic information system (GIS) database to facilitate their management, monitoring, and conservation.

This paper presents a further research advancement, namely the development of a fully automatic approach for the segmentation and classification of these elements. The objective of this analysis is to facilitate the monitoring process through providing tools for a more detailed assessment of the threat of detachment of parts of the stone elements.

The study focusses on the south-west side of the main courtyard (known as “Cortile dell’Ammannati”, Figure 1 and Figure 2). However, the same method can be adapted to the entire range of rusticated façades of the Pitti Palace, even if they have slightly different finishes, as well as to other buildings with similar stonework.

Section 2 presents the role of digitalisation of cultural heritage in the broader implementation of a digital economy and its relation to European and Italian environmental policies. Section 3 presents the case study of Palazzo Pitti and the specific features of the degradation of its façades, which justifies a continuous surveillance programme. Section 4 describes the photogrammetric survey of its façades, carried out with a drone, and the elaboration of the documents and materials used as references in the subsequent sections.

Section 5 outlines the principal methodologies and the state of the art for segmenting the components of a façade. It also describes the system adopted, which is essentially based on the geometry of the components. In Section 6, the relationship between facade elements and associated risk factors is elucidated. The results obtained are presented in Section 7; Section 8 highlights strengths and weaknesses of the proposed research and outline possible future developments.

2. Digitalisation of Cultural Heritage and the Green Deal in Europe and Italy

The European Community has promoted innovation [6,7] in the cultural heritage sector for the documentation, preservation, and sharing of the European legacy. It has done this through directives, studies [8], guidelines [9], and supporting research projects and infrastructures such as Europeana [10], which is undoubtedly the widest and best known of these. The European Community’s support for digitalisation can be traced back to the Lisbon Strategy [11,12], which was developed in 2000. The action plan highlighted the potential of digitalisation to promote European culture, while acknowledging the primary economic objective of “strengthening employment, economic reform and social cohesion within a knowledge-based economy”.

At the beginning of 2020, the European Community initiated the Creative Europe [13,14] programme with the objective of providing support to cultural sectors, including those related to heritage, in order to facilitate the exploitation of the opportunities presented by the digital transition.

In addition, the Green Deal was presented to the European Parliament in 2019 [15]. This was a series of initiatives with the objectives of achieving climate neutrality by 2050 and of identifying prevention and adaptation strategies to be adopted in response to the challenges posed by climate change.

Although the Green Deal did not explicitly refer to cultural heritage, a document [16] produced in accordance with the Work Plan for Culture 2019–2022 [17] identifies threats to and critical issues for heritage arising from climate change and examines the potential contributions that cultural heritage can make to the Green Deal for Europe. The document presents an analysis of the current state of cultural heritage in Europe in relation to climate change and proposes a series of actions and good practices aimed at risk mitigation. Research and innovation play a central role, promoting an interdisciplinary approach and “more in-depth studies on the behaviour of cultural heritage materials (organic, inorganic and composite) in times of climate change, including studies based on modelling and simulation (including protection, conservation, green materials and consolidation), and studies on the compatibility of adaptation measures with cultural heritage guidelines and innovative materials monitoring (using sensors, three-dimensional documentation and AI/machine learning processing)”.

Furthermore, the Commission Recommendation on a common European data space for cultural heritage [18] encourages member states to digitise their cultural heritage between 2025 and 2030 to improve its preservation, with particular reference to heritage at risk. It recalls how “the development of advanced digital technologies, such as 3D, artificial intelligence, machine learning, cloud computing, data technologies, virtual reality and augmented reality” has created great opportunities and that “cultural heritage assets digitised in 3D can be a source of relevant knowledge on climate-related impact, adaptation and resilience (e.g., 3D allows non-destructive analysis of assets, visualisation of damages and information for restoration, conservation, etc.)”.

In July 2020, the European Council approved Next Generation EU [19], a fund with a value of EUR 750 billion to support member states in the aftermath of the COVID-19 pandemic. In Italy, the National Recovery and Resilience Plan (Piano Nazionale di Ripresa e Resilienza—PNRR) [20], a programme of reforms and investments over a 2021–2026 timeframe, was introduced in 2021 to manage Next Generation EU funds. The objectives were twofold: firstly, to address the economic and social damage caused by the pandemic; and secondly, over a longer time frame, to resolve the structural deficiencies of the Italian economy and to contribute to the digital and ecological transition. The programme, which underwent a partial modification in 2022 [21], is divided into 16 components grouped into six missions. Of these, the first, entitled “Digitalisation, Innovation, Competitiveness and Culture,” and the fourth, “Education and Research,” address the themes of cultural heritage in the context of the digital and ecological transition in greater depth.

In this context, the Ministry of Culture, through the National Digitalisation Plan (Piano Nazionale di Digitalizzazione—PND) [22], which was drafted by the Central Institute for the Digitalisation of Cultural Heritage Digital Library, intends to promote and organise the process of digital transformation in the period 2022–2026 in accordance with European directives. The process is oriented towards the preservation of collective heritage, represented by cultural assets, with the objective of making them more accessible to people and fostering the development of new professional actors and services. With these assumptions, the PND contributes to achieving some of the sustainable development goals (SDGs) of the United Nations 2030 Agenda [23], namely, quality education, decent work, economic growth, business innovation and infrastructure, and sustainable cities and communities.

The target groups are museums, archives, and other public and private cultural institutions, as well as professionals such as artists, scholars, stakeholders, etc.

The PND, which includes a number of appendices, is a highly detailed document and is divided into three sections. The first section, the vision, outlines the reference values and objectives. The second section, the strategy, lists the technologies and processes. The third section, the guidelines, provides the operational tools.

The vision is to move from digital conversion to digital transformation. This shift is not only a technological challenge but also a cultural one. It involves the creation of a digital ecosystem [24] based on the relationships between objects and between objects and people [25].

Cultural institutions should consider digitalisation as an integral aspect of their operations, rather than a standalone initiative to be implemented on an ad hoc basis. Rather than viewing digitalisation as a marginal, secondary aspect, cultural institutions should view it as an element closely connected to all the choices and actions they develop.

Among the purposes of digitalisation, the document [26] asserts that the digital copy of the original, if acquired correctly and for documentation purposes, assumes informative and cognitive value comparable to that of the asset itself, representing material characteristics and the relative state of preservation. The digital resource thus becomes an active tool of study and knowledge that enables simulations and, if consistently replicated over time, supports monitoring processes.

The CHANGES project [27], which is funded by PNRR resources within Mission 4 ‘Education and Research’, comprises nine spokes. These are characterised by horizontal activities, including research, technology transfer, training, and the dissemination of results. The project is coordinated by eight state universities and one research organisation. Specifically, Spoke 7, entitled ‘Protection and conservation of Cultural Heritage against climate change, natural, and anthropic risks’, investigates the consequences for cultural heritage caused by the impact of climate change, which accelerates the effects of natural and anthropic risks. The investigation has been conducted with a multi-scalar approach, including territory, historical cities, architecture, and mobile cultural artefacts. This approach allows the consideration of different scenarios and the addressing of both general and specific issues. Spoke 7 selected artefacts of varying scales for the assessment of their state of conservation and exposure to risks. Among these, Palazzo Pitti was also chosen for assessing the potential for reusing the spatial data acquired during the survey campaign described in more detail below and on which this study is based. The considerations encompass the containment of the effects of degradation, which may be exacerbated by climate change, and the potential impact on the safety of visitors to a monument that is frequented by a large number of people.

In Italy, the protection of cultural heritage from the effects of climate change has been achieved through the formulation of strategies and implementation plans that provide guidance and good practices for monitoring and management in situations of risk from natural and man-made hazards.

Following the impetus provided by the 2013 European Adaptation Strategy [28], in 2015, Italy adopted the “National Climate Change Adaptation Strategy” [29], of which the operational instrument is the “National Climate Change Adaptation Plan” (Piano Nazionale di Adattamento al Cambiamento Climatico-PNACC) [30], approved on 21 December 2023.

Both documents have a section dedicated to ‘Cultural Heritage and Landscape’. The design of protection, control, and damage-prevention strategies derives from the assessment of vulnerability and risks, the study of the heritage materials, with particular reference to built heritage, and the associated types of degradation in relation to environmental conditions. Among these factors, water plays a primary role. Intense and increasingly frequent rainfall causes significant damage to buildings, in particular through accelerating the deterioration of structures and wall surfaces. Considering the buildings in the Florentine area built with the typical “pietraforte” sandstone, the action of rainwater causes the dissolution of calcium carbonate present in the calcite veins and this mechanism can lead to the decohesion of the rock, with the complete opening of the veins and the possible detachment and falling of blocks, with possible personal injury and material loss.

The results of prediction models suggest that the dissolution of carbonate stone materials will be predominantly due to precipitation and an increased atmospheric CO₂ concentration in the near future [31].

The document, admitting its own shortcomings, outlines how the research on the impact of climate change on cultural heritage has thus far been relatively limited and notes the urgency of this issue. Furthermore, it underlines the dearth of observational data, emphasising the necessity of continuous monitoring in order to correlate the degradation of materials and systems characterising cultural heritage with climatic variables and their changes. It highlights how downscaling is necessary if one wants to predict the impact that changing climatic parameters have not only on historic centres and the landscape but also on areas more specifically related to individual artefacts.

The Extraordinary National Plan for Monitoring and Preservation of Immovable Cultural Heritage, adopted by the Italian Ministry for Cultural Heritage and Activities in 2018 [32], defines the criteria for the selection of artefacts to be monitored and the consequent conservation interventions, as well as the necessary control priorities. These are determined based on specific indices of territorial danger and the individual vulnerability of buildings, the instrumental control systems to be used, and methods for implementing safety, conservation, and protection measures. The Plan also claims that monitoring the state of cultural property in relation to risk situations has always been considered of utmost importance, which over the years, on several occasions, has resulted in attempts to implement suitable policies and tools. In order to ascertain the types and level of risk to which a heritage building is exposed, it is necessary to gain a comprehensive understanding of the artifact and to conduct regular and systematic assessments of the asset’s condition and the site in which it is situated. It also states that a “holistic” approach to data identification is required, from the global to the particular and from the context to the artefact, using high-resolution diagnostics to assess the individual structure or individual element of the structure itself. To achieve this, it is necessary to integrate various technologies and techniques like remote sensing, drones, photogrammetry, and sensors to apply to the artefact, and periodic checks on the state of structural and material deterioration.

It should be noted that, as of the present date, the only operational (and only partially updated) tool available for planning interventions on cultural heritage according to the criterion of level of risk is the Cultural Heritage Risk Map [33], which has been elaborated by the Ministry of Culture and is managed by the General Directorate for Cultural Heritage Security.

The aforementioned Italian documents discuss these topics only in their general aspects. However, it is important to remember that raw data must be interpreted and evaluated. Processing is what turns data into information, and this requires a critical approach. Therefore, it is essential that documentation and monitoring are designed and carried out by qualified specialists, possibly with interdisciplinary skills. It is essential that collected data be accompanied by metadata and paradigms that allow for the correct use of the data, their quality and accuracy, their preservation, and the traceability of acquisition and processing steps [34].

The topic also aligns with the Italian National Research Plan 2021–2027 [35], which identifies ‘Humanistic Culture, Creativity, Social Transformation, Society of Inclusion’ among the major strategic research and innovation areas for development in Italy. “Digitalisation of protection, conservation and valorisation processes” is the title of the first articulation of this strategic research area. Therefore, a current research topic focuses on objectively quantifying the presence and effects of various forms of decay and then calculating damage indices to prioritize conservation and preservation works [36]. The present work is an example of the digital transition of the building analysis and monitoring process, initiating the long-sought transition to preventive conservation, maintenance, and monitoring [37] and, from a broader perspective, supporting the transition to digital transition in the field of documentation and protection of cultural heritage.

3. Palazzo Pitti and the Conservation of Pietraforte Sandstone

Palazzo Pitti is the largest and most magnificent mansion in Florence. The original part of the building was built in 1452 on the order of Luca Pitti. According to Giorgio Vasari, it may have been designed by Filippo Brunelleschi. In 1549, Eleonora of Toledo, the wife of Cosimo de’ Medici (the future first Grand Duke of Tuscany), bought the whole property. Even though the palace was already three storeys high, it is likely that the interior was still under construction [38].

After Cosimo ascended the throne, the building grew in importance and was used not only as a residence but also as the seat of the government. In the 19th century, it was briefly the seat of the Kingdom of Italy. The palace’s current appearance is the result of a series of later extensions. The U-shaped courtyard designed by Bartolomeo Ammannati, which faces the hills behind the building, is one of the most important expansions. The courtyard was built between 1561 and 1577 and, like the rest of the building, features impressive rusticated stonework.

The façades of the courtyard were developed on three floors above ground level [39]. The architectural style employed by Ammannati involved the traditional superimposition of the three classical orders—Doric, Ionic, and Corinthian—with rustic yet regular ashlars (Figure 3).

On the ground floor, the rusticated blocks are domed and close together The column shafts and the wall were designed to accentuate the massiveness of the lower floor, which serves as the foundation of the building.

The upper floors exhibit an alternation of rustic and plain stonework. On the second floor, the rustic ashlars are squared, while on the third, they are domed. From an architectural perspective, this creates a strong and striking contrast between light and shadow. However, this type of stonework involves the presence of numerous protruding elements, from which some detachment of stone fragments can occur. This can lead to safety concerns and challenges in the conservation of the work [40,41,42].

The façades were constructed using a local kind of sandstone, called pietraforte, which was also used in many other Florentine buildings. The most important quarries for this kind of stone were located behind the Palazzo Pitti and right in the place where the courtyard was built. This sandstone is bluish-grey when freshly extracted, but over time it takes on a characteristic brown colour due to presence of iron oxide. It typically has complex stratification and calcite veins at different angles. Although the façade’s stone texture is aesthetically pleasing, it compromises its consistency. This sandstone is susceptible to spalling and the detachment of large fragments due to mineral solubilisation and thermo-hygrometric cycles. As the rustication has overhanging elements, including large projections, it is important to regularly inspect the condition of the façades and promptly secure any elements at risk.

These problems are dependent on the intrinsic characteristics of the stone and have always been present, as evidenced by the numerous gaps in the façade and the metal anchors used in the past to hold the broken blocks together (Figure 4).

However, pollution and climate change, altering the intensity and frequency of rainfall and the concentration of CO₂, have exacerbated these detachment problems, accelerating the degradation processes, and consequently increasing the building’s preservation issues and the safety risks to people.

Motivated by the above considerations, a research agreement was established in 2020 between the administration of the “Gallerie degli Uffizi”, which is responsible for the maintenance and preservation of Palazzo Pitti, and the University of Florence, to conduct a comprehensive 3D survey of the building, aiming at providing the tools for successively monitoring the building’s health. This agreement involved a comprehensive survey of all interior and exterior spaces within the complex, which comprises over 1500 rooms and houses four museums and other cultural heritage institutions. The documentation encompassed both geometric and construction characteristics, as well as information on finishes, materials, and the state of conservation. It also included details on their management, use, and current and past designations. A three-dimensional information system was designed to support knowledge of the building and its future management and maintenance [43].

The survey was conducted utilising integrated geomatic techniques. This entailed the establishment of a control network employing topographic and geodetic tools (GNSS and a total station) and the utilisation of 3D scanning and structure-from-motion (SfM) photogrammetry for the architectural survey [44]. Laser scanners were primarily used for surveying the interior spaces, while the exterior ones, including all the roofs and façades, were acquired through the integration of data derived from scanning and photogrammetry.

4. UAV Photogrammetric Survey on the Courtyard of Palazzo Pitti

The detailed survey of the façades of the courtyard of Palazzo Pitti was carried out using UAV SfM photogrammetry, designing a suitable flight pattern in order to achieve the required result in the most efficient way [45,46,47]. Among the factors to be properly taken into account in the flight plan, the geometric and exposure characteristics played a pivotal role. The dimensions of the courtyard are approximately 40 m × 50 m, with façades approximately 35 m high on three sides, while the south-east side is about 12 m high. This implies that in July, when it was possible to carry out the survey, the lighting conditions were changing rapidly during the day. Consequently, the time periods during which each façade was in shadow were calculated, and the survey was conducted primarily in the early morning or late afternoon, also in order to avoid the presence of visitors close to the flight area.

On each of the three largest façades (the fourth is only one storey high), an average of one target per column or window was placed. The coordinates of these targets were measured with a total station. In addition, some easily recognisable natural points were measured on the cornice below the roof, where it was not possible to manually place the targets. This allowed for a redundant set of control measurements, to also take into account the possibility that some targets could be damaged or removed. The elaborations presented in the following sections focus on the south-west façade of the courtyard (Figure 5), whose photogrammetric reconstruction was obtained by exploiting 18 control points (CPs) and 11 check points (ChPs).

The drone used for the survey was a DJI Phantom 4 Pro, which has the camera characteristics reported in

Table 1. DJI Phantom 4 Pro camera specifications.

Specification	Dimension (in mm)	Dimension (in Pixels)
Focal length	8.8
Sensor	13.2 × 8.8	5472 × 3648
Pixel size	0.00241

.

The main products of the photogrammetric survey were planned to be a 3D model and an orthophoto, whose spatial resolution (ground sample distance, GSD) was required to be approximately 2 mm.

Images were taken orthogonal to the façade, at an average distance of 6 m, in order to achieve the required spatial resolution (Figure 6). The terraces on the second and third floors, as well as the roof, exhibit a projection of approximately 1.5 m from the façade. Consequently, the distance between the drone and the most prominent parts of the façade was approximately 4.5 m. In these circumstances, it was necessary to operate the drone manually, as the GNSS signal was irregular or completely absent within the courtyard and at a short distance from the walls.

The camera network comprised a series of orthogonal photographs, with an 80% forward overlap and a 70% side overlap. This entailed that the strips were taken at approximately two-metre intervals. The acquisition of images was also extended beyond the eaves line to record the roof pitches. Additionally, images with an oblique axis were captured at the same distance to reinforce the camera network and to facilitate the detailed documentation of the side faces of the elements. It is important to note that the aforementioned terraces are equipped with cast-iron railings and protective nets, which would have prevented imaging the façade behind them. Consequently, the stripe configuration was implemented through pitching up and down over and under the terraces.

Each elevation terminates at a concave or convex angle to the adjacent ones. In order to ensure a safe distance from the walls, in the corners, photographs were taken along a vertical line with the axis oriented towards the intersection of the walls. These sets of images were subsequently employed in the production of the orthophotos of both the façades.

In consideration of the optimal exposure times to achieve the requisite images in the shade, in an effort to avoid motion blur, the flights were executed at a velocity of approximately 0.3–0.5 m/s. In total, approximately 3000 images were captured for the survey of the south-west façade. Although the drone was capable of saving images in the raw format (DNG), the image buffer had a limited capacity, which may have resulted in the loss of some frames, which motivated saving them in JPEG format.

Given the necessity of performing imaging in limited timeframes, flights were conducted over several days. This necessitated meticulous planning and comprehensive annotation of the metadata associated with each flight session. This was undertaken in order to prevent gaps or overlaps and to ensure the generation of consistent results.

Nowadays, several photogrammetric tools and software are available on the market. In the 2020 survey, all the photogrammetric processing tasks were performed with Agisoft Metashape Pro software. The photogrammetric model of the south-west façade exhibited a root mean square error (RMSE) of 9.4 mm for the CPs and 8.1 mm for the ChPs, values compatible with the error level of the realized topographic control network. The outcomes included the creation of (i) a digital elevation model (DEM) with a resolution of 2 cm (Figure 7), upon which the studies presented here were based, and (ii) a 2 mm GSD orthophoto (Figure 8), which served as the cartographic basis for the 2D drawings. In particular, all ashlars and stone blocks were identified manually with great precision and this information has been used for the validation of the results of the current study. (Figure 9).

5. Segmentation of the Façade Elements

Segmentation entails partitioning a set of entities into several subsets, each of which is usually characterised by some kind of intra-set homogeneous characteristics. When considering spatial data related to a 3D model, segmentation aims at grouping the original data in different regions based on certain characteristic such as colour, intensity, typology, or semantic information. Since both photogrammetry and laser scanning, the two most commonly used 3D reconstruction techniques nowadays, produce point clouds as natural output, a number of techniques have been recently investigated in order to properly segment such data, when dealing with semantic segmentation of heritage buildings in particular [48,49,50,51,52,53]. Among the different proposed approaches, it is typically possible to distinguish between those directly dealing with the point cloud, which usually either deploy neural networks [54] or exploit geometric features [55], and those based on image segmentation instead. In the latter case, the image segmentation results should be properly reprojected on the original point cloud [56].

Although the approaches mentioned above aim to deal with the point clouds produced for quite generic heritage buildings, some simplifications may be introduced when considering certain specific cases, such as when dealing with only a building’s façade, for instance. A dominant plane and its normal direction can usually be identified when considering a façade; hence, a 2D representation of the façade can be easily obtained via projecting the façade elements along the normal direction onto the dominant plane. The obtained representation, being either an orthophoto or a DEM, depending on the specific kind of operation that has been implemented, is clearly not completely equivalent to the original data, but the information loss, in this specific case, is often quite marginal. Several approaches have been tested for orthophoto segmentation, usually based on the use of proper convolutional neural networks [57,58], as well as DEM [59]; the latter approach usually aims at exploiting geometric information often including spatial derivatives, normals [60], or computations of level sets [61]. These approaches are often restricted to brick segmentation, often on masonry walls [57,58,59,62,63].

Motivated by the above observations, this work, quite similarly to [59,64], implemented geometry-based segmentation directly on the façade DEM (Figure 7). Since the main goal of this work is to monitor the detachment of portions of ashlars and the assessment of the riskiest elements, the purpose of the implemented segmentation procedure was principally that of automatically segmenting the ashlars. The DEM segmentation workflow was as follows:

• Preliminary operations. The DEM may include areas outside of the region of interest and/or areas not covered by ashlars. A simple depth threshold-based procedure was used in order to determine the DEM regions sufficiently close to the ideal wall surface to be considered ashlars. In our implementation, this corresponded to limiting the analysis to points within a 1.1 m depth interval;
• Depth variation-based edge detection. This step aimed at discriminating the potential object boundaries as those associated with depth variations in the DEM. A simple thresholding mechanism was used to determine the norm of the DEM gradient with such an aim, i.e., let $z=f\left(x,y\right)$ be the DEM depth value corresponding to the $\left(x,y\right)$ coordinates and $\nabla f(x,y)$ be the value of the gradient in $\left(x,y\right)$, which can be estimated, for instance, as follows:

$$\nabla f(x,y)=\left[\begin{array}{cc}\frac{\partial f}{\partial x}& \frac{\partial f}{\partial y}\end{array}\right]\approx \left[\begin{array}{cc}\frac{f\left(x+1,y\right)-f\left(x-1,y\right)}{2}& \frac{f\left(x,y+1\right)-f\left(x,y-1\right)}{2}\end{array}\right],$$

(1)

where, for simplicity of notation, $x$ and $y$ are assumed to represent the location of the considered point in the DEM expressed in pixels.

Then, the following thresholding was applied to the norm of the gradient on all the $\left(x,y\right)$ positions in the DEM:

$$\left|\nabla f(x,y)\right|>\overline{∆z},$$

(2)

where $\overline{∆z}$ is a properly set depth threshold. An example of the thresholding results is shown in Figure 10. It is worth noticing that several more complex alternatives can be considered in spite of the described depth-variation-based edge detection approach, e.g., edge detection based on the Harris filter [65]. Nevertheless, in this work, other options have not yet been considered because of the simplicity of the proposed method and the reasonably consistent results obtained in the case study.

• Connected components computation. Connected regions [66], of black pixels, i.e., not previously identified as potential edges, were determined in the binary image resulting from the previous step (e.g., Figure 10), considering 8 connectivity, i.e., black pixels were in the same region if either they had an edge or a corner in common.
• Optimization of object boundaries. Object boundaries derived from the depth-thresholding mechanism summarized in Equation (2) may be inaccurate, in particular when objects have a rounded shape, as visible in Figure 11a,b. This typically results in partially incomplete segmented ashlars (or, more in general, objects). Consequently, an optimization procedure was implemented in order to reduce the above-mentioned issue. With such an aim, first, the normal to the DEM surface in the neighbourhood area of each object border was computed, i.e., on the pixels identified as potential edges using Equation (2) (white pixels in Figure 10). Then, all the points of each border were moved to the corresponding closest local maxima of the angular difference between the normal to the surface and the $z$ direction. Finally, a piecewise linear fit was computed on the derived object boundary in order to reject the outliers, if any. The result of this procedure for the region in Figure 11a is shown in Figure 11c.

6. Ashlar Classification and Risk Assessment

Risk assessment in buildings and infrastructures is a very important and popular topic, attracting the interest of researchers from different fields [67,68,69]. Nevertheless, the specific peculiarities of the façades of Palazzo Pitti require the development of ad hoc procedures in order to properly deal with their monitoring and maintenance.

The risk of detachment of portions of stone blocks is related to a number of factors, including the progress of lamination, the presence of discontinuities, cracks, and fractures, material loss and colouration, and erosion and exfoliation phenomena, as well as the geometric parameters described in detail below. In addition to material and decay analysis, non-destructive techniques (NDTs) have been applied for effective assessment of the façade’s conservation: sclerometric measurements through the Schmidt hammer test to evaluate the stone surface strength and ultrasonic velocity testing to evaluate the consistency of masonry and allow the identification of internal defects like fractures, voids, and detachments [70]. Monitoring campaigns are periodically carried out on the various façade of the Palazzo Pitti, including the removal of detached parts of stone or the consolidation of blocks that need it; the last monitoring campaign was carried out in February 2022.

A first step in the risk assessment related to the detachment of ashlars has already been investigated [71] by an interdisciplinary group, including geologists, restorers, and geomatic experts. That work presented a method for managing both geometric data and results obtained through NDTs in an integrated manner. With regard to the geometric data, a virtual inspection was conducted manually on the same dataset used in the present study. A number of key aspects were considered, some of which were derived from the 3D survey output, while others were inferred by the virtual inspection of high-resolution, 3D-oriented images. Materials, forms of decay, and previous restoration works were also mapped.

On a test area, the co-occurrence of all of those key factors was considered and related to the shape model of the façade, and further analysis was limited to overhanging elements.

The findings were corroborated through comparing the stone elements that were identified as presenting criticalities with those selected by expert geologists and restorers in a field investigation, thus validating the proposed approach as a valuable preliminary analysis prior to fieldwork, capable of reducing the time it takes and therefore its costs.

Despite the outcomes of such comparison proving the potential of the adopted approach, the main limitation was clearly the significant need for human intervention in the data processing and analysis. The present work aims to overcome the limitations of the manual approaches described in the aforementioned paper to ascertain the viability of deriving pertinent information to assess the risk of detachment of portions of stone elements through a more comprehensive and extensively automated analysis of the shape model, while the other previously identified factors influencing the criticalities of the blocks are, at present, overlooked. Introducing the discarded factors in the automated analysis will be considered in the future development of this work.

Analyses of a building conducted with a traditional approach require interpretation and, therefore, involve the work of specialized personnel according to study and “training” processes that can now be at least partially computerized. Courses in architectural history basically use methods analogous to machine learning: images are proposed, from different viewpoints, of buildings with some relationship to each other; comments on the images correspond to the training information. The architectonic order is a formalized code—yet, always declined in different ways—and its recognition in the elevation requires critically applying a knowledge model. The architectural order can then be broken down into its sub-elements, as well as the elements composing a general building, which can be identified and organised into more or less complex structures, depending on the objectives of the study, organising them, if necessary, into classes and subclasses [72,73,74,75]. In the masonry of Palazzo Pitti, there is not always a correspondence between the architectural decoration, apparently subdivided in three overlapping orders characterised by ashlars of different shapes and surface finish, and the façade’s structure. In many cases, a single architectural element consists of different stone blocks; in others, the same block is carved to represent different architectural elements. Consequently, although at first glance the rustic order appears to consist of homogenous elements, on a closer inspection it is possible to distinguish the stone blocks from the ashlars (Figure 12). A single stone block used for masonry construction constitutes the minimum spatial reference for diagnostic and structural analysis.

In this study, on the other hand, the segmentation was aimed at identifying the individual ashlars, and subsequent analyses were concentrated on those elements. In accordance with this consideration, only a selection of the objects segmented as described in Section 5 was used for further analysis, in particular in order to (i) distinguish between different ashlar categories based on their shapes and (ii) compute a geometric-based ashlar detachment risk map. The rationale mostly motivating the procedure presented in this section is that, from a merely geometric point of view, the most protruding elements in the façade are those that are supposed to be associated with the highest level of (geomentry-only-based) risk. Hence, the above-mentioned restriction of object selection was implemented, considering only areas where protruding elements were present and, on the other hand, considering only objects within a certain area range, in order to select (almost) only ashlars.

Then, an unsupervised geometry-based classification of the segmented objects was performed using K-means [76]. It is worth noticing that segmentation and classification techniques can be used to support building information modelling (BIM) generation; e.g., semantic segmentation may be deployed, for instance, as a tool to support a scan-to-BIM workflow. Despite such potential of an automatic segmentation and classification procedure, the goal here was merely that of grouping the ashlars into categories according to their shapes and exploiting such information in the computation of the geometry-based risk assessment.

The use of unsupervised classification was motivated by the need of reducing as much as possible the human intervention in the analysis procedure, hence avoiding the need for creating an ad hoc training dataset. Nevertheless, supervised classification will also be considered in future investigations in order to check the potential performance of this kind of approach. Among the unsupervised approaches, K-means was chosen for its simplicity and limited computational complexity. Given a set of n vector data, where each data sample ${\mathit{x}}_i$ is assumed to be a d-dimensional vector, K-means aims at partitioning the n samples into K clusters by minimising the functional $V$ given by the sum of the intra-cluster squared distance with respect to the cluster mean ${\mathit{\mu }}_j$:

$$V=\arg min{\sum }_{j=1}^K{\sum }_{i\in C_j}{\left|{\mathit{x}}_i-{\mathit{\mu }}_j\right|}^2,$$

(3)

where $C_j$ is the j-th cluster. It is well known that the outcome of the algorithm depends on the values {${\mathit{\mu }}_j$}_j=1,…,K used to initialize it. Nevertheless, such dependence can be remarkably reduced via running several instances of the algorithm with different initialization conditions, as was carried out in this work.

In the application considered in this work, each vector ${\mathit{x}}_i$ was associated with a different object, i.e., a different ashlar, and K-means was used to automatically group the ashlars into different categories based on their shape. K-means output clearly depends on the specific vectors used to describe the object’s geometric characteristics. Taking into account the specific characteristics of the ashlars in the considered façade, the following geometric features were considered as input for the segmented objects for the K-means clustering algorithm:

• object area,
• width (along the x direction),
• height (along the y direction),
• depth variability, computed as the median absolute deviation (MAD) of the depths in the DEM area associated with the segmented object,
• variability of the horizontal angle of the surface normal with respect to the z axis, computed as the MAD of the horizontal angles for all the object pixels.

The values of the geometric features reported above were centred and normalized before being inputted in the K-means algorithm, in order to avoid an implicit different weighting between such factors. Nevertheless, future investigations will be dedicated to determining whether the use of different weighting criteria may lead to better classification results.

Then, a geometry-based risk index was computed for each of the segmented objects, based on how much it protruded. In accordance with the previously presented motivations, this index related to only the protruding volume of the considered ashlar. Let us consider the example in Figure 13a, where the protruding volume has to be computed for the blue ashlar; the red one is the one below it. In the computation implemented here, the upper surface of the red one is extended along the y direction up to the top border of the blue ashlar, hence defining the green solid shown in Figure 13b. The protruding volume is hereafter considered as the subtraction of the green parallelepiped volume from the blue ashlar one. The computation of such volumetric difference is achieved as follows:

• partitioning with a grid the projection of the blue ashlar surface on the x–y plane. A natural choice for such partitioning is clearly to consider grid cells equal to the DEM ones;
• for the grid cell of index (h,k), determining the corresponding parallelepiped of basis area A_hk and size z_hk along the z direction given by the difference of z depth between the blue and green boxes corresponding to such a position. Then, the volume for the parallelepiped is V_hk = A_hk × z_hk, and the protruding volume is computed as $\sum _{h,k}(A_{hk}\times z_{hk})$.

The computation of the protruding volume presented above can easily be generalized to the irregular shapes of the ashlars. It is worth noting that when ashlars have a rounded shape, usually approximately circular on any section on the y = $\overline{k}$ plane, the cell representation may not provide a good fit to the real ashlar shape close to the two sides of the ashlar. The ashlar section being approximately circular, the circle radius r was computed as half of the ashlar width on the x direction. The area on the x–y plane of the ashlar slice in correspondence with y = $\overline{k}$ can be approximated as πr²/2. Extending the volume computation to this case and its generalizations is also trivial.

Assuming a constant value for the stone density, i.e., 2.6 g/cm³ [77], the protruding volume was converted to an approximate mass value. Hence, the following metrics were computed for each segmented object:

(a)

the approximate mass m of the protruding volume;

(b)

the linear density, that is m/Δx, where the mass m is defined as above, while Δx is the object width along the x direction;

(c)

the weight of the protruding volume above each cell, i.e., for each cell in a segmented object, only the portion of the protruding mass above the cell was considered.

7. Results

The segmentation procedure presented in Section 5 was applied to the DEM of the south façade of the courtyard, obtaining the results shown in Figure 14.

In order to validate the obtained segmentation, a comparison was carried out with the manually identified stone elements shown in Figure 15, which were drawn with CAD tools on the orthophoto and were used here as a reference.

Since the main interest is in determining a geometrical vulnerability map for the protruding ashlars in the façade, the analysis of the segmentation results presented here and the subsequent vulnerability assessment reported here was limited to the area shown in Figure 16.

Table 2. Automatic segmentation vs. manually identified stone elements.

Accuracy	F1	OverSeg [%]	Median IoU [%]	MAD IoU [%]
0.89	0.94	11.8	71.2	7.9

provides a numerical assessment of the implemented automatic segmentation procedure, reporting the following performance metrics: accuracy, F₁-score, the over segmentation index (OverSeg), median intersection over union (IoU), and the median absolute deviation of the IoU values. Accuracy and F₁-score are defined as follows:

$$\mathrm{a}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{TP+TN}{TP+TN+FP+FN},$$

(4)

$${\mathrm{F}}_1=2\frac{precision\times recall}{precision+recall},$$

(5)

where TP are the true positives, TN the true negatives, FP the false positives, FN the false negatives, precision = $\frac{TP}{TP+FP},$ recall = $\frac{TP}{TP+FN}$. The OverSeg index, indicated as a percentage, is defined as the number of real stone elements that are segmented in more than one object divided by the total number of real stone elements automatically detected. The IoU between a ground truth stone element and an automatically segmented object at least partially overlapped with it is equal to the overlapping area between the two divided by area of their union. It is worth noting that when a real stone element was segmented into more than one object, only the automatically segmented object with largest overlap with the real one was counted as TP, whereas the others were counted as FP. Figure 17 shows the IoU distribution limiting the analysis to only the correctly matched elements (true positives).

The accuracy and F₁ values in Table 2 can be considered sufficiently high to be quite satisfying. However, the median IoU was not as high as the accuracy and F₁. This value of the median IoU can be explained through the relatively large number of over-segmented stone elements (and a few of under-segmented ones), as shown in the OverSeg value in Table 2 and the left side of the IoU distribution in Figure 17. Figure 17 also shows that very few stone elements were identified with more than 90% IoU; this can probably be explained by differences mostly in the object borders.

From a visual inspection of the façade, the presence of ashlars of different shapes is quite clear. For instance, in Figure 3 and Figure 6, it is possible to distinguish at least the following categories: quite large rounded and squared stone elements on the half-columns, smaller squared elements on the sides of the half-columns, capitals, and less protruding elements on the arches and keystones. In accordance with this observation and with a few tests, K-means algorithm was executed on the segmented objects, setting K = 5 and running 1000 instances of the algorithm with different initial conditions. The obtained classification is shown in Figure 18. Given the obtained classification results, the five identified classes can be quite well described as follows:

• Rounded large column elements (yellow);
• Squared large column elements (blue);
• Squared side column elements (green);
• Arch elements (orange);
• Rounded large column elements (light blue).

Regarding the light blue elements in Figure 18, it is clear that they are concentrated on the bottom left of the façade and that such a class is clearly a duplication of the first one. Nevertheless, the algorithm dedicated a different class to these elements because of some incorrect segmentations, leading to the formation of larger objects (under-segmentation issue) with shapes different from those described in the other classes. It is also worth noting that air conditioning systems installed on the façade (Figure 19) were also classified, mostly in the “arch elements” class. Although a few elements in Figure 18 were identified as wrongly classified (usually because of segmentation errors), the obtained classification appears overall to be quite reasonable.

In accordance with the indexes introduced at the end of Section 6, the following figures show the assessed mass of the protruding volume for each segmented object (Figure 20), the corresponding linear density, normalized to its maximum value within the test area (Figure 21), and the determined mass of the protruding volume over each point of the detected objects (Figure 22).

Figure 20 allows identification of the stone elements associated with the largest protruding mass. Although this can give a general indication of risky locations, it is clearly largely correlated with the stone size. The normalized linear density, shown in Figure 21, allows partial reduction of the correlation with the stone size, while still highlighting the most protruding regions. Finally, Figure 22 shows that the overhanging mass index, normalized by its maximum value on the façade, may be useful to identify certain of the locations within the stones that may more subject to detachment.

8. Discussion and Conclusions

8.1. Strengths and Weaknesses of the Proposed Research

The integration of knowledge into computer systems offers a promising solution to complex problems. Through considering geometric, structural, functional, and other aspects, this approach can significantly enhance the efficiency and effectiveness of conservation and preservation efforts. A pro-active approach towards monitoring has gained importance in recent times, and an objective approach based on measures and indicators has become the goal to pursue [78]. This study proposes the reuse of a 3D survey carried out using geomatic techniques and the development of methods for analysing the resulting 3D model to support the identification of possible problems relating to the conservation of the building and the safety of its visitors. It is appropriate to analyse the proposed approach’s strengths and weaknesses by comparing it with the traditional mode of operation based on macroscopic analysis and close on-site inspection by experienced operators. This involves noting the number and type of calcite veins, open discontinuities, lamination, previous restoration interventions, and registering NDT measurements.

In particular, strengths must be considered from a long-term perspective, in both economic and technological terms. This encompasses, on the one hand, the economies of scale triggered via standardised processes repeated over time, transforming analysis from a one-off event to regular monitoring. On the other hand, technological developments, particularly those that may be supported by AI, may also be considered.

The potential of the work can be outlined as follows:

-

cost effectiveness for built heritage owner–managers and limited time needed in the field, thus minimising the courtyard’s inaccessibility to visitors and the costs of an aerial platform; the last monitoring campaign, carried out in February 2022, took two operators working for seven days on the platform, plus one operator to move the crane—during that time, the courtyard could not be visited by tourists;

-

objectivity of the results produced, although this should not be confused with their reliability. Rather, reliability refers to the parameters analysed but not to the representativeness of the model adopted to describe the real levels of risk;

-

once the analysis workflow has been defined, it can be completed in a very short computing time, thus contributing to the sustainability of repeating the process at shorter time intervals;

-

the results produced as an outcome of each measurement campaign are directly comparable, if consistently realised in terms of resolution, georeferencing, etc., thus enabling effective monitoring;

-

field campaigns carried out in the past resulted in paper-based reports with obvious problems of preservation and sharing, which are overcome by the digital transition in this approach, which allows effective multidisciplinary comparison, on-line cooperative work, remote consultation, and assessment by experts.

Current limits include the following:

-

in this work, we have considered the effect of only the geometry of the examined objects, while cultural heritage conservation and restoration tasks concern the evaluation and modelling of a multitude of critical factors influencing structural or material deterioration. The more complex modelling approach proposed in [79] can be optimised by automating certain steps, as is proposed in this paper;

-

automatic classification was applied to the rusticated construction elements that make up the columns, piers, and ashlars of the arches, and the infill above them. No criteria have yet been defined for the automatic evaluation of the overhang of the blocks of the stringcourse cornices;

-

some stone elements are characterised by a complex three-dimensionality (as is the case for the capitals of the different orders, the decorations of the arch keystones, etc.), which has been neglected at this stage, having assumed the DEM as a significant shape model, thus limiting the analysis to 2.5 dimensions and depending on its resolution (4 cm/pixel).

8.2. Relation with Previous Works

Despite several works having already been proposed in the literature for identifying certain elements in building façades and for segmenting certain repetitive, quite regular, elements, such as bricks on masonry walls [57,58,59,60,61,62,63], the direct usage of such methods on quite complex façades of heritage buildings is often not so effective. This paper proposes a geometry-based segmentation method, quite similar to those in [59,64], combined with properly employed unsupervised classification that enabled distinguishing between different classes of ashlars. Despite some similarities with other segmentation methods, the obtained outcomes, i.e., the ashlar classification, have been used in a more general and original procedure, leading to the computation of three geometry-only-based metrics that can be employed in procedures to assess the detachment risk of portions of stones.

8.3. Future Research Outlook

The segmentation approach presented is based on the preliminary definition of geometric parameters characterising the shape and volume of stone elements. This research will continue investigating point cloud segmentation methods based on neural networks. Some of the critical elements for evaluating the strength of stone ashlars are also particularly minute and were not recorded in the point model, as was the case with calcite veins, for example; however, the high resolution of the images allowed them to be recognized in most cases, thus suggesting that segmentation systems based on oriented images should also be considered in the future, in order to reproject the results onto the 3D model at a later date. Indeed, further direction of the research development will have to concern the extension of critical factors from merely geometric to diagnostic ones, characterising the specific material considered and its state of preservation.

The implementation of automatic methods of analysis makes it possible to simplify monitoring activities, which are indispensable in this case study, given the high attendance by tourists. In particular, the repetition over time of survey campaigns using photogrammetry from UAVs will be able to provide information to be compared with previous states, allowing, for the first time, effective monitoring of the entire area under consideration.

8.4. Conclusions

Although considering a simplified approach with respect to the possible complexity of a multicriterial analysis, the proposed research has achieved significant results in terms of automating the segmentation and classification process, allowing the programming of monitoring not only of the envelope of the façade surface but also of its individual elements. For the time being, the factor assumed to prevail with respect to the danger of stone detachment is the geometric one: the greater the overhang (and thus the weight of the protruding portion) of the stone block with respect to the one below, the more damaging the effect of its fall could be.

In conclusion, the research presented here contributes to the long-term goal of supporting the management of a historic monumental building and allows the prioritisation of conservation and preservation responses following periodic monitoring campaigns conducted on the façades of Palazzo Pitti, identifying the stone elements presenting greater criticality.