Digital Twin Sensors in Cultural Heritage Ontology Applications

Abstract

This paper concerns the extension of the Heritage Digital Twin Ontology introduced in previous research to describe the reactivity of digital twins used for cultural heritage documentation by including the semantic description of sensors and activators and all of the process of interacting with the real world. After analysing previous work on the use of digital twins in cultural heritage, a summary description of the Heritage Digital Twin Ontology is provided, and the existing applications of digital twins to cultural heritage are overviewed, with references to reviews summarising the large production of scientific contributions on the topic. Then, a novel ontology named the Reactive Digital Twin Ontology is described, in which sensors, activators, and the decision processes are also semantically described, turning the previous synchronic approach to cultural heritage documentation into a diachronic one. Some case studies exemplify this theory.

1. Introduction

Digital twins are increasingly used in all research and application fields.

As discussed in recent reviews on the use of this term in different domains [1], this concept captures the complexity of digital equivalents of the real world both in industrial and research applications. It is sometimes present as a generic term to designate a digitised model of real artefacts or, more specifically, a structured information system about reality. Also, regulatory authorities are using this terminology. Notable examples are offered by the European Commission, which has introduced the idea of the Digital Twin of the Earth [2] to indicate a global environmental model but is also using this notion rather loosely in its research programmes, calling for research projects that deal with “digital twins” in different domains to simulate research activities and conduct them in a digital framework.

The idea of introducing digital twins in land planning and built environment activities dates back to the Cambridge National Digital Twin project, which stated general principles for their application in this domain, the Gemini Principles [3].

As concerns cultural heritage, the term digital twin made its appearance in reference to digitised 3D models of heritage artefacts as a synonym of digital replica, suggesting an original literary rather than technical use of the term. The first applications of 3D visualisation in cultural heritage date back to the last decade of the 20th century, especially in archaeology, and were motivated by the need to illustrate the supposed pristine appearance of remains, replacing drawings and maquettes previously used for this purpose with visual reconstructions of the past. A similar approach rapidly extended to document the shape of artefacts, monuments, and sites and has now produced a huge amount of heritage 3D models of extremely different quality and detail. Looking at heritage assets in 3D rather than by 2D images has become a straightforward practice, although still needing more work, as demonstrated by the EU Recommendation on this topic [4] calling for activities incentivising the creation of new ones.

In time, it has become apparent that taking into account only the shape of heritage assets is not sufficient for advanced model use, both in research and in practice. Any serious use requires background documentation which needs to be stored, linked to the 3D model, and made available for inspection.

Work on this topic started within the 3D-COFORM project (2008-2012) [5,6,7] and continued with various significant contributions, especially for heritage conservation and restoration [8,9]. In some of these applications, the additional information concerning, for example, materials used in different parts or conservation interventions has been attached as a text to the 3D model or to some of its regions. Although availing a completely different 3D technology, similar applications have been developed adopting a CAD approach to the creation of the 3D model. In this case, the heritage-related documentation is managed by extending an existing information system, called BIM (Built Information Modelling), widely used in the building industry to specify information about materials, services, and processes concerning the construction of a new built asset. BIM is semantically defined by the IFCs (Industry Foundation Classes), which have been extended in various way to incorporate concepts typical of cultural heritage. The resulting application, called HBIM where H stands for Heritage, has a widespread use especially for documenting monuments and historic buildings. A 2023 paper [10] surveys a very large number of such applications. Within HBIM models, digital twins have been envisaged as a way of introducing heritage-related concepts [11,12]. However, the data management software managing HBIM suffers from the limits of the data management software available for BIM, which privileges the analysis of individual assets modelled but is much less efficient in investigations across different buildings modelled in this way, for example those built with the same material (wood, bricks, etc.).

Both such approaches suffer from their “original sin”, i.e., having a 3D model as the root of the documentation tree, where any other information is just a leaf. Thus, information structures pertain to an individual heritage asset that makes further data processing a cumbersome task: comparisons among different assets, e.g., to search for construction materials across non-visual documentation, is impractical, as no advanced database technology can assist in this job because the data are not properly organised. On the contrary, well-structured heritage documentation organizes data in classes of which the information pertaining to an individual asset is an instance. Classes are interrelated by properties. For example, “wood” would be an instance of the class “Construction Material”, linked to the class “Heritage Asset” by a property “is made of”. This enables us to search for all assets with wood as the construction material and listing the corresponding instances, i.e., all of the buildings made of wood. Such a semantic approach enables the creation of large digital documentation systems which allow us to manage knowledge about heritage assets and the activities carried out on them, especially conservation and preservation ones. In this way, a huge amount of research results can be made available and searchable by scientists and practitioners. This availability complies with the FAIR principles as it not only supports findability and access but also makes them interoperable via the compliance to standards and ultimately allows for their reuse. The purpose of the present paper is to show how the digital twin paradigm is suitable to contribute to this objective and improve the effectiveness of data organisation in the heritage domain.

Classes and properties are organised in ontologies, and for cultural heritage, the standard one is the CIDOC Conceptual Reference Model (CRM), ISO 21127 [13]. CIDOC stands for the committee which originally started the CRM definition, now managed by a Special Interest Group. Besides organising the information related to cultural heritage, the CRM presents an important feature, extensibility, i.e., the possibility of specialising concepts, i.e., the classes, and relationships, i.e., the properties, to fit better to the data description in a particular application subdomain. The resulting models are interoperable with any other one compatible with the CRM by establishing a correspondence (a mapping) between couples of corresponding classes (and properties) in each model, either directly to each other when feasible, or via the CRM class/property which is a common superclass/super-property of both. Thus, the CRM base—the universal set of heritage-related concepts—has developed several extensions, such as CRMarchaeo for archaeological excavation, CRMsci for scientific observations, CRMdig for digital activities and components, and so on, offering a complete framework to organize knowledge about heritage. It also encompasses concepts related to intangible heritage by means of classes already proposed for narration. Both classes and properties are identified by a literal mark common to all in the ontology, followed by a progressive number; for example, E followed by a number (e.g., E1) denotes the CRM classes and P followed by a number (e.g., P9) is used for properties. It must be noted that extensions to the CRM benefit from inheriting the robustness of this reference model, including, among others, the consistency of the ontology which derives directly from being an extension of the standard one. For example, a building might be, at the same time, an asset and a place, but the risk of partial overlap of the two classes has already been resolved within the CRM and the solution extends to all its extensions, such as ours. Therefore, concerning the construction of the RHDTO and the control of its internal consistency, we have not relied on any specific tool such as Protégé or OntoStudio, but rather on the conceptual tools and methodologies provided by the CIDOC CRM. In particular, by aligning the RHDTO with the CIDOC CRM, we were able to carry out conceptual checks at each step of the development and for each new entity introduced to ensure that the model remains consistent, coherent, and interoperable with the CIDOC CRM framework and with other ontologies and systems that use the CIDOC CRM as their core model. To achieve this alignment, we have followed a systematic and iterative process of mapping and refinement, in which we have identified and investigated the relevant classes and properties in the CRM and other related ontologies and defined new classes and properties in the RHDTO only when actually required. We have extensively used CIDOC CRM scope notes, examples, and formal definitions to ensure that the RHDTO’s classes and properties are well structured and consistently linked with any other entity in the CRM ecosystem.

In the development of the RHDTO model, we have exclusively focused on the conceptual design of the ontology without restricting our investigation to its implementation in a specific tool or language. This is because the RHDTO is intended to be a high-level ontology that provides a comprehensive framework for modelling the dynamic and complex nature of cultural heritage entities and risk management operations and, as such, is designed to be agnostic to any particular tool or format. Of course, once the general conceptualisation is formalised, the ontology can be implemented by means of different technologies. For example, the Heritage Digital Twin Ontology (HDTO), which we already released and which we are extending here, has already been expressed in the RDFS and the data have been encoded using the Turtle syntax, this being a particularly suitable compact and human-readable format for the RDF to describe the objects of our domain and to be queried and reasoned over by using standard Semantic Web technologies, such as SPARQL and OWL reasoners. This is what we plan to do with the RHDT extension as well, using the RDFS or any other suitable format, when the model is tested and evaluated and has reached the necessary stability and maturity to be released and used.

Regarding the potential computability of the ontology, during the development phase, we have verified the completeness and expressivity of the extension by ensuring that it provides all of the features for modelling the multifaceted nature of all cultural heritage and risk management entities, while at the same time keeping the introduced classes and properties to the minimum necessary number. However, the actual computability of the RHDTO, in terms of its performance evaluation, adaptability to services, and scalability, requires further investigation and testing during the implementation phase.

In the present paper, as well as in the previous ones, we aim to describe the logical structure of the ontology as a solution to the digital documentation needs of the heritage sector. A rich and extensive semantic structure may improve cultural heritage documentation, which is indeed possible, although ineffective, also with much simpler systems, but the richness and flexibility of more advanced ones, such as the CRM and its extensions—including the present one—allow us to cover the required documentation extension. A rich documentation allows us to use the data for many digital applications, from valorisation to the focus of our work, conservation, and preservation. A digital twin holistic paradigm appears to us as the most effective framework for this purpose.

The above-mentioned critical points of 3D-based systems are among the reasons why in 2022 and 2023, we proposed a novel ontology for heritage information as a CRM extension. It is based on the Heritage Digital Twin (HDT), a holistic approach to heritage information where the 3D graphical component is just one element [14,15]. In this ontology, named HDTO, all of the documentation is organised according to a compatible extension of the basic CRM model, and thus, information from other documentation systems organised according to the CRM can be straightforwardly imported into it. Moreover, the HDTO enables us to document the intangible component of tangible heritage assets, and in the particular case of intangible heritage, such as the one enlisted in the UNESCO list [16], it is the only existing one providing complete coverage; for intangible heritage, the visual documentation may consist of video or audio recordings or may even be totally absent. Three-dimensional models have also been used to document aspects of intangible heritage (see [17] for a review of such applications), but they are usually less suitable as the starting point of the related documentation which includes stories, traditions, and other immaterial content is very often unrelated to the shape. However, we will thoroughly explore the complexity of documenting intangible heritage in future work.

The HDTO has been used to set up the cloud-based knowledge base (KB) created in 4CH [18], an EU-funded project designing a Competence Centre for the Conservation of Cultural Heritage. Documentation in the 4CH KB includes the relevant information about heritage assets, from the visual and 3D data to the results of scientific analyses, conservation activities, and historical research. The description on the 4CH KB implementation is fully documented in the above-mentioned technical report, which also addresses aspects concerning its scalability to very large amounts of data and their processing. As described in the related report [18], the system has been implemented on the INFN-CNAF supercomputer based in Bologna, Italy, planned to become Leonardo, one of the most powerful ones in Europe. This machine can effectively allow for fast data processing regardless of their size and amount by providing computational resources in real time as required. The applicability of real-time calculations in the knowledge base depends on several factors, such as the size and complexity of ontology-encoded information, the volume and velocity of the data, the computational resources and algorithms available, and the specific requirements and constraints of the services and applications. We also recognize that the knowledge base may contain a large and diverse number of subclasses and instances, which may pose challenges and trade-offs in terms of the accuracy, completeness, and timeliness of the calculations. In this regard, we have designed the RHDT Ontology to be highly expressive while also minimising redundancy and complexity, which should help to improve the efficiency of data processing. To address these challenges, we also plan to adopt a multi-strategy approach at the implementation stage, which combines different techniques and tools for ontology engineering, data management, and reasoning, such as modularisation, approximation, sampling, and parallelisation. Thus, we believe that real-time calculations are feasible for the types of data and decisions required by the knowledge base extended according to the RHDTO, such as the detection and diagnosis of critical events, the prediction and simulation of future scenarios, and the optimisation and adaptation of the system’s behaviour.

On the other hand, data aggregation from heritage institutions is a process that may require time to establish cooperation agreements that go beyond the digital technicalities, such as those concerning access rights, ownership of the data, and other legal aspects. In our opinion, preparing the technical framework—as our work on the ontology aims to do—is however a necessary step in this direction. It also allows us to open a discussion with heritage institutions and professionals, the data providers, which may lead to fine-tuning and improving the data organisation framework.

Organising heritage data management is per se important, especially in the present time in which the data overflow risks are becoming unmanageable. It also has important impacts on research and management. Aggregating heritage-related documentation, possibly stored in different repositories, within a holistic approach as the one we propose may have different uses. For example, it may enable us to improve the valorisation of heritage assets and the enjoyment by the public by providing a one-stop shop for all of the content required to build scientifically based explanations, also availing approaches such as those pertaining to virtual, augmented, and extended reality. However, this is not our goal. Our work focuses on the conservation, preservation, and when necessary, restoration of heritage assets. In this regard, a rich documentation is absolutely required, and a wide knowledge base allows for comparisons with similar conditions and interventions on other assets. Indeed, heritage is affected by many different risks, both individual—i.e., related to individual assets—and systemic. The latter include, for example, the effects of catastrophic events such as floods, fires, landslides, and earthquakes, as well as human actions such as vandalism, crime, and war. Sensors and IoCT play a crucial role in documenting the availability of automated reactions to conservation risks, and thus, their introduction in the ontology is a significant step forwards in the semantic description of heritage assets.

While the previous version of the HDTO provided the first step in organising the documentation of heritage assets into a static system, with the present improved version, we aim to move towards a dynamic connection between the real thing and its digital replica. The RHDTO can therefore provide several benefits since it establishes a consistent framework for representing heterogeneous information and ensure that data are captured and stored in a format ready to be used by a wide range of services and applications. In particular, the ontology is fundamental for integrating data from a variety of sources, including historical and material records, scientific studies, and sensors, as well as data on the general architecture of the system and its functioning, information with different structures and formats. The ontology is thus fundamental to create the complex knowledge graph necessary to make these data interoperable, an operation not feasible when using databases or other non-semantic tools. Since the RHDTO is designed as a system with different components and data, the ontology is therefore essential to orchestrate this heterogeneous mass of information and allow the system to work at its best.

The HDTO does not (yet) consider the dynamic and interactive aspects connecting a digital twin to reality; timewise, it is static, just incorporating in each documentation item a timestamp that may be used to reconstruct the diachronic evolution of the asset. In both of our previous papers mentioned above, the authors acknowledged that this approach was just a first step in the definition of a full-fledged digital twin, still lacking the modelling of interactions with reality-based information.

In the present paper, and in other forthcoming related ones, we will try to move beyond this limitation and to develop an extended documentation system where interaction with reality may be documented and put to work in a continuous interchange of information and process activation between the real world and its equivalent in the digital universe. Such extension is motivated by the need to reflect either one of the features embedded in the digital twin concept, its reactivity to inputs coming from reality and its capacity to produce corresponding outputs and real-world actions. Besides this theoretical necessity, we believe that a reactive heritage digital twin may better model heritage assets which are immersed in a continuous changing landscape, are affected by phenomena happening in it, and with their own changes, contribute to compose its evolution. The new semantic model proposed here extends but in no way supersedes the previous version, which conserves its usefulness in the many cases in which this dynamic perspective is not required. The HDTO is, anyway, the substrate on which the digitally simulated reactions take place, and the new ontology named the Reactive Heritage Digital Twin Ontology (RHDTO) is an extension incorporating all of the previously defined classes and properties.

Due to the complexity of such model extension, we will proceed by taking steps with separate contributions, the present one being dedicated to sensors and activators. This stepwise approach will also allow for a more focused discussion within the research community, hopefully producing improvements to be incorporated in future versions of the ontology. We will often use very simple examples to keep the reader’s attention on the ontological modelling rather than on the technical aspects of the sensor and to facilitate the model’s explanation to non-technical heritage professionals who play a very important role in use-inspired [19] research in this domain, but of course, the ontology works also in more complicated cases, as will be illustrated in the sequel.

Finally, and to conclude this long introduction, we will not address implementation aspects in this paper. For the existing knowledge base, the previous version of the ontology was written in the RDF and implemented as described in the already mentioned technical report [18]. The implementation was preceded by the logical description of the ontology [14,15] which was very favourably received by the heritage community. The implementation was then reviewed by a commission of experts and considered an excellent result of the related funded project. We will follow the same approach with the extended ontology presented here, stimulating a discussion and receiving suggestions for improvement as far as the novel dynamic approach to heritage documentation presented here is concerned. Then, we will test its effectiveness on case studies (the example presented in this paper may be one of them), in parallel with collecting data from the producers/owners, i.e., cultural heritage institutions, finally implementing the system in the RDF as required.

2. A Summary of the HDTO

To keep this paper self-contained, we summarize below the main features of the HDTO. For full details, please refer to the above-mentioned papers [14,15] introducing the concept of the Heritage Digital Twin (HDT).

According to [20], “a digital twin is a virtual representation of a physical system (and its associated environment and processes) that is updated through the exchange of information between the physical and virtual system”. Another definition in [21] describes the digital twin as made across five dimensions: physical entities, virtual models, services, data, and connections.

Both our papers [14,15] acknowledged that what they proposed was only half way in the work needed to develop a full-fledged concept of digital twins related to cultural heritage, as they addressed only three of the above-mentioned aspects of the digital twin: physical entities (renamed as real-world entities to also encompass intangible aspects), virtual models, and data, while services and connections were temporarily set apart to be considered in future work, of which the present paper is an initial part. This led to a formal semantic definition of the HDT and of the related ontology, the HDTO. The main reason for this partial move was to start setting up an overarching data infrastructure where all of the parts of the virtual model include the relationships to a well-organised data system. The HDTO is thus conceived as an extension of the CRM, enabling semantic interoperability with all of the (many) existing information systems for cultural heritage documentation which adopt this standard or compatible systems, such as the EDM (Europeana Data Model) [22] and its forthcoming extension expected to be used in the data space for cultural heritage. Our above-mentioned papers were well received by the research community, having been extensively quoted in the literature and the first one [14] being the recipient of the Data 2022 Best Paper Award.

The most important innovation proposed in the HDTO is to define the concept of the Heritage Digital Twin (HDT) in a holistic perspective, i.e., as the complex of all available digital information concerning a (real world) heritage asset, either movable (e.g., museum exhibits), immovable (e.g., monuments and sites), or even intangible (e.g., traditions). In the resulting data organisation, all of the data, such as reports, documents, datasets, and visual representations (2D, 3D, or 4D), or any other related digital data, are linked with each other within the HDT by appropriate properties. Thus, the HDT becomes a full-fledged digital alter ego of real-world heritage assets.

As presented in the two papers [14,15] mentioned above, the HDTO aims to encapsulate the dual aspects of cultural heritage, both tangible and intangible, and the cultural events in which cultural entities are involved throughout their history, offering specific classes and properties for the dynamic documentation and analysis of their interactions. Moreover, the HDTO is pivotal in structurally organising data to document digital twin systems, knowledge bases, and other similar operational platforms. Being machine-readable and actionable, the HDTO functions as an internal “language” of the digital twin system, ensuring smooth interaction among its various digital components. The HDTO classes describe the more common high-profile entities, such as places, agents, physical objects, events, and temporal entities, thereby ensuring high-level interoperability in multiple domains. New classes and properties are defined to describe more specific concepts such as digital twins, cultural entities, 3D models, and others, having no exact match in the CRM ecosystem. To facilitate the distinguishing of classes and properties in the text, henceforth, we will indicate their names and symbols in italic boldface.

The root class of the HDTO is HC1 Heritage Entity, comprising tangible and intangible entities of the real world regarded as valuable because of their contribution to society, knowledge, and/or culture. HC1 is a subclass of E1 CRM Entity, the root class of the CRM. Instances of HC1 Heritage Entity may refer to real assets of any nature: physical, both movable and immovable, immaterial, or born digital. Subclasses of HC1 Heritage Entity model the material and immaterial aspects that a cultural entity can have. In particular, the HC3 Tangible Aspect class models the tangible, material component of entities of the real-world, both movable (e.g., archaeological, artistic, and cultural objects) and immovable (e.g., built heritage as monuments, buildings, cities, and other complexes). HC4 Intangible Aspect comprises instead cultural events, traditions, and practices having particular social, historical, and cultural significance, including practices and expressions, memories, and oral traditions about events, things, and people. They can be independent or related to some physical object. Other classes in the HDTO model the digital documentation related to heritage entities, encompassing 3D models (rendered through the dedicated HC8 3D Model class), images, other audio–visual mediums (modelled using the HC7 Digital Visual Object class), as well as textual information made available in digital format (represented by means of the HC6 Digital Heritage Document class). Stories and storytelling, intended, respectively, as accounts of facts about cultural entities and the way in which these facts are narrated, are modelled using the narrative and narration classes provided by the Narrative Ontology (NOnt), another CRM compatible ontology for to the description of texts and their content [23]. The use of the NOnt, as well as other compatible models such as CRMsci and CRMdig, demonstrates the capability of the HDTO to integrate semantic tools defined in other contexts to represent the different facets of a digital twin. However, no classes or properties of the NOnt are used in this paper, so we refer to [15] for definitions and examples.

Heritage Digital Twins of cultural entities are rendered in the HDTO by means of the class HC2 Digital Twin used to organize and connect the digital information available in a given system and pertaining to an HC1 Heritage Entity, including digital representations and information of the effects of events that influenced or/and are related in any way to its state and activities (e.g., restorations, conservations, etc.) carried out on it. The HC2 class semantically represents the way in which the Heritage Digital Twin is implemented and the building blocks through which it operates. In addition to those inherited from the CRM, the HDTO also provides other properties, such as HP1 is digital twin of, linking a heritage entity (HC1) with its digital twin (HC2). In general, the HDTO properties are mainly intended to semantically interconnect the various pieces of information modelled through the classes. Thus, they are elements used to implement the knowledge graph of the Heritage Digital Twin. The main classes of the HDTO and the properties linking them are shown in the following diagram (Figure 1). The figure outlines the semantic structure of the HDTO, which is fully described and illustrated with diagrams in [14,15].

An additional advantage of the introduction of the HDT derives from the already mentioned modelling of historical buildings using BIM (Building Information Modelling), based on the Industry Foundation Classes (IFCs), and the HBIM (Heritage BIM) based on them. The extensible nature of the HDTO allows us to incorporate BIM and such extensions into it via mappings from/to the IFCs and the proposed HBIM extensions to the HDTO.

3. Sensors and Activators

In a simple definition, a sensor is a device that produces an output signal for the purpose of detecting a physical phenomenon. More precisely, a sensor is a device, module, machine, or subsystem that detects events or changes in its environment and sends the information to other electronics, frequently a computer processor. Thus, a sensor measures one (or more) physical quantity and sends a signal based on the measured value(s) to an activator, which automatically activates some action if required according to built-in criteria. Very simple everyday examples are thermostats, which combine a temperature sensor and an activator switching off or on an air conditioner if the temperature is within or outside a pre-selected range. The automated action may be very simple, such as the switching on/off of a warning red light in a car fuel gauge or an anti-theft alarm which detects the interruption of an electrical circuit due to the opening of a house front door and activates an alarm siren. The latter case may also consist of a movement detector (the sensor) that sends a signal to the siren activator. This second example is a bit more sophisticated as it may take into account other aspects, for example the volume of the intrusive object, and decide if it is a thief or just your cat wandering around during its human’s absence.

In any case, the model of the overall action consists of several steps:

• A sensor: measuring one (or more) physical quantity—in the above examples, they, respectively, are the temperature, the quantity of fuel in the car tank, and the amount of infrared radiation for an anti-theft movement sensor—and forwarding the measure(s) to the decider.
• A decider: comparing the sensor measurement with pre-set decision rules and deciding if some action needs to be activated. In the above examples, for the thermostat, the decision is based on a threshold, a limit temperature, and the required action is switching the air conditioning on/off; for the fuel gauge, the threshold is a minimum acceptable quantity of it, and the action is switching on the red warning light in the car dashboard; and for the anti-theft system, the action involves determining if the size of the intruder according to the infrared size is comparable with a human or is smaller, such as a cat. If the sensor signal falls in the alarm range, the decider sends an activation signal to the activator. The decision rules will often be much more complex and imply different actions according to the values measured by the sensor.
• An activator: When authorised by the decider, it commands actions in the real world, e.g., switching some device on or off, such as the air conditioning system, the car red warning light, and the alarm siren. The action may be programmed to result in something more complex than a simple “Do”; in this case, we may logically split such a complex activator into different elementary activators. For example, an anti-theft activator may command different actions such as “activate the siren” and “call police”; thus, it corresponds to two elementary activators.

In simple cases, the decider is included either in the sensor or in the activator, frequently in the latter as in the above examples. It is however a logical functionality different from the device, and thus, we will keep it separate from both.

The logic of the sensor–decider–activator system is often hard-coded in the electronics of the devices, especially for simple ones, but there is also a strong tendency towards soft-coding to support its integration with other smart devices and enable remote control via wi-fi and the internet. This has led to an increase in the purely digital component, which facilitates the inclusion of this system in a digital twin framework. In any case, both the sensor and the activator are placed in the real world, so they do not belong to the digital twin but should be included in the overall semantic description of the whole system. The decision process may be very simple, as in the above examples, or very complicated when several factors must be taken into account, requiring processing and possibly, in a forthcoming future, an AI-based decision process.

4. The Internet of Cultural Things (IoCT)

The Internet of Cultural Things, i.e., the use of the so-called Internet-of-Things (IoT) in cultural heritage applications, has been a subject of study and research in the last 10 years.

The concept of the Internet of Cultural Things (IoCT) refers to the application of Internet of Things (IoT) technologies in the context of cultural heritage and cultural assets (see [24] for a recent survey on the topic). It involves the integration of sensors, devices, and information systems within cultural contexts to monitor, preserve, study, and enhance artworks, archaeological sites, museums, monuments, and other elements of cultural heritage. In the development of digital twins in the cultural heritage sector, IoCT plays a fundamental role. In the cultural context, digital twins have been extensively used to create a digital replica of an artwork, an archaeological site, or a museum environment, allowing cultural operators to explore, preserve, and share cultural heritage in innovative ways. Sensors range from the very simple ones previously described to a network of devices measuring physical quantities and exchanging the related information with each other. Both sensors and activators may be directly linked to the digital twin or be connected in a network in which they are the nodes.

The orchestration of such IoCT nodes may envisage different ways in which devices and sensors within an IoCT network communicate with each other and with data processing systems, allowing devices and sensors within an IoCT network to effectively and efficiently collaborate to collect, transmit, simulate, and process relevant cultural data, thereby contributing to the development of digital twins and innovation in the cultural heritage sector.

Among the many studies concerning IoCT and digital twin applications in the cultural heritage domain, [25] reviews the most important applications, mainly focusing on the valorisation of cultural heritage and to assist visitors and promote tourism. Very simple examples of such applications are the sensors activating a video when somebody enters a museum room or, in general, starting an automated reaction to human presence to make the heritage experience more interactive, enjoyable, and participative for visitors. More complex ones collect measures from various devices, connected in an articulated system as described above, and use these to make decisions on actions to be performed by one or more activators.

Digital twin applications to the conservation of monuments are discussed in general in [26], while [27] describes the use of sensors to monitor the conservation conditions of historic buildings, in this case a church in Matera’s “Sassi” considered in its complex environment. Applications to movable artefacts are less frequent; among them are the one described in [28] in relation to a famous violin and the study of palaeolithic lithics presented in [29]. A semantic approach to digital twins is introduced in [30], which discusses the ontologies for the description of sensors and IoT in preventive conservation. Other related papers are [31,32,33,34].

5. The Reactive Heritage Digital Twin Ontology (RHDTO)

5.1. Introductory Notes

In this section, we introduce the new entities of the RHDT aimed at modelling the reactive elements of the digital twin, such as sensors, services, and data processing engines. As mentioned earlier, our ontology builds upon the CRM ecosystem and incorporates elements from the CRMdig, CRMsci, and CRMpe extensions. Concepts and logics from various existing ontologies, developed to model sensors and related phenomena, have been investigated to fine-tune our conceptual tools. Models such OntoSensor [31] and SensorML [32,33] provide a rich and expressive framework for representing sensors, their properties, and their relationships to other entities, such as observed properties, features of interest, and observation processes. Nevertheless, our primary goal is not only to model the phenomena related to digital objects, their nature, and their interactions, but also to develop a semantic tool for making Reactive Digital Twins operational and achieving integration and interoperability between the different types of data they store and manage.

By leveraging the CRM ontology and extending it with new classes and properties where necessary, we aim to create a comprehensive ontology that can effectively represent the dynamic and complex nature of cultural heritage and risk management, and support decision-making and risk assessment in real-world scenarios. The CRM, with its robust framework for modelling events, is particularly well suited for capturing the dynamic aspects of the Reactive Heritage Digital Twins and representing the complexity typical of this domain, including state changes detected by sensors and other devices, component interactions, and processes. Leveraging this event-centred approach allowed us to better focus on system behaviours and predict future outcomes by identifying common patterns and trends within the system’s operations.

Throughout the development process, we have thus prioritised the reuse of existing classes and properties from the CRM ecosystem, only introducing new ones when necessary to accurately represent the intended semantics of the described entities. This approach strikes a balance between ensuring interoperability and maintaining precision in describing the functionalities and components of the system, while also keeping the ontology compact and easy to understand and apply. Therefore, the resulting RHDT Ontology only introduces classes and properties to describe the specific components and events involved in the various operations performed by the system, the interaction between devices and services, and the communication with human operators.

Although our ontology is specifically designed for cultural heritage, many of the new concepts we have introduced, such as the devices, sensors, activators, data, and signals, may have broader applicability beyond this domain. In fact, we have taken care to define these entities in general terms to make them consistent with relevant models developed in other fields, such as OntoSensor and SensorML, and to facilitate interoperability and reuse in other domains.

Table 1. The CRM extensions used in this paper.

Extension Name	Used for	Reference	Class Symbol	Property Symbol
CRM	General use	[13]	E	P
CRMdig	Digital objects	[35]	D	L
CRMsci	Scientific analyses	[36]	S	O
CRMpe	Interoperability framework	[37]	PE	PP
HDTO	Heritage Digital Twins	[14,15]	HC	HP

reports the symbols used for classes and property names in the CRM and its extensions used in the present paper. For example, in the CRM, the class symbol is E and the property symbol is P. Symbols are combined with a number to identify a specific class or property, e.g., E1 or P2.

Also, in the following sections, we will use the convention of denoting the semantic elements in the text by italics–boldface.

Another extension of the CRM, proposed in [38], describes a model for the results of scientific analyses on heritage assets to be used for the data concerning them. It is not used in the present paper, but its classes and properties are relevant in applications to support the description of the condition states of artefacts, the experiments carried out to evaluate them, and their results.

5.2. Sensors

Sensors are a central element of the Reactive Digital Twin. In our ontological view, a sensor can be defined in general terms as a digital device placed on physical objects or in specific locations intended to measure and collect data about them, process it, and transmit it to the digital twin system for analysis and further processing. A sensor can measure various physical or environmental properties, such as temperature, pressure, humidity, light, sound, position, velocity, or acceleration, and can be of different types, such as analogic or digital, wired or wireless, active or passive. Building upon this general definition, we introduced the new HC9 Sensor class, specialising the D8 Digital Device of CRMdig, a general class aimed at describing instances of material items capable of processing or producing digital data.

The positioning of a sensor on a cultural object can be modelled using the new HP15 is positioned on property, allowing for the representation of the spatial relationship established between the object to be monitored (HC3 Tangible Aspect) and the sensor placed on it. For sensors positioned in a space adjacent to the cultural object, we can use the P55 has current location property of the CRM, connecting a sensor to the instance of E53 Place where the monitored object is currently located to indicate that both of these physical entities share the same space.

During its operational lifecycle, a sensor can assume a particular status representing its condition at a given time, which can be modelled using the E3 Condition State class of the CRM. Instances of this class enable the description of sensor states (such as on/off, operational/non-operational) and operation modes, reporting any faults or errors, as well as any maintenance or calibration activities performed on it. Additionally, sensors are typically operated by different kinds of software, tailored for gathering and processing acquired data, generating signals, and executing specific functions such as configuration, calibration, monitoring, and diagnostics. To represent this operational software, we employ the D14 Software class of CRMdig that was chosen due to its comprehensive semantic features allowing for the modelling of all of the software components of the Reactive Digital Twin. While the D14 class already provides the necessary features to model the software components of any device, we deemed it important to explicitly represent the relationship between the sensor and its controlling software. For this reason, a specific HP11 is operated by property was created to indicate the close interconnection existing between the sensor and the software that controls it. This also enables the link between the software, the measurement operations performed by the sensor, and the digital signals generated from them.

5.3. Measurements and Signals

In a CRM perspective, the measurement operations performed by sensors can be modelled as events. A sensor measurement event, in fact, involves the collection of data identifying various modifications of conditions on the object or in the environment under examination, within a certain spatiotemporal interval. In accordance with these features, we have defined the new HC13 Sensor Measurement class as a subclass of the S21 Measurement class of the CRMsci ontology to describe the measurement events performed by the sensors connected to Reactive Digital Twins. The L12 happened on device property of CRMsci is used to specify the sensor (HC9) on which the measurement (HC13) took place. Measured events can instead be modelled by means of the CRM E5 Event, a class particularly well suited for describing generic events of various types. The measurement (HC13) and the measured (E5) events are ontologically related through the O24 measured property of CRMsci. Instances of the CRM E55 Type class can be assigned to instances of HC13 to detail, among the other things, the detected types of risk conditions related to the cultural object, and subsequently trigger the generation of the corresponding signals. This assignment is usually carried out by means of the L17 measured thing of type property of CRMsci.

Generated signals can be modelled as digital objects since they codify a measurement taken by a sensor to be transmitted, under certain conditions, to the system for further processing or analysis. Typically, a signal is a piece of software encoded in a formal language (e.g., XML, JavaScript, etc.) and generated in response to a specific event, such as a change in sensor reading, a threshold crossing, or a specific timer. It may contain various information, including sensor identifiers, timestamps, recorded values, measurement units, and details about condition assessment quality. In our model, instances of signals are represented by means of the new HC12 Signal class, a specialisation of the D9 Data Object class of the CRMdig ontology. Signals (H12) generated in the presence of a potential risk detected by sensors can be linked to the measurement event that generated them through the L20 has created property of CRMdig. Once generated, signals are transmitted to the Reactive Digital Twin and specifically to dedicated intelligent digital agents (deciders) running as part of the Reactive Digital Twin system. The transmission of signals to deciders is recorded via the new HP12 was transmitted to property.

5.4. Deciders

The RHDTO provides a new HC10 Decider class, specialising the scope of PE1 Service of the CRMpe ontology, to describe in detail the deciders and their features. Deciders can receive and analyse input data from various sources, such as sensors, activators, or other services, and query the digital twin knowledge base to acquire knowledge concerning the linked cultural objects and their potential risks. They further process all of this information using algorithms, rules, or models and generate output instructions, such as commands, feedback, or status updates, to be sent to other components of the digital twin, to activators, or to other external devices. Moreover, deciders can also send alerts to human operators in various forms, such as email, SMS, push notifications, and visual or audible alerts, depending on the nature of the detected issue.

Ontologically, the actions performed by deciders after the decision-making process is complete can be modelled as events. Thus, we have designed a specific HC14 Activation Event class, a subclass of the CRM E5 Event, to represent them. The O13 triggered property of CRMsci is particularly useful for linking activation events to the deciders by which they were triggered. Activation events can prompt specific actions by digital agents, such as adjusting a valve or activating a pump, and/or alert human operators, informing them of the necessary countermeasures to be taken in response to the detected potential risks. Both operations are modelled by means of the new properties HP14 alerted, pointing to instances of the E39 Actor class, and HP13 activated, linking activation events with activators.

5.5. Activators and Human Interaction

Activators are other fundamental components of Reactive Digital Twins. In our RHDT Ontology, they are described as devices that enact actions upon physical objects or processes based on the instructions received by the deciders operating within the Reactive Digital Twin system. They can be of mechanical, electrical, digital, hydraulic, or pneumatic types, and can execute a range of actions, including movement, adjustment, control, or regulation. Activators may operate in distinct modes, including automatic, manual, or semi-automatic, and can be governed by various software systems, such as firmware, drivers, or applications. Moreover, they offer various degrees of precision, accuracy, and responsiveness, contingent upon their design, fabrication, and maintenance processes.

Activators interact with the Reactive Digital Twin through multiple channels, including sensors, APIs, or communication protocols, enabling the reception of commands, feedback, or status updates from the system. Given their specialised nature, we have defined a new HC11 Activator class, a subclass of D8 Digital Device of the CRMdig ontology to describe these peculiar devices. As in the case of sensors, activators may be operated by software (again encoded through the D14 Software class of CRMdig) intended for interpreting the commands and feedback coming from the Reactive Digital Twin system and controlling the activator’s actions accordingly.

As already anticipated, the E39 Actor CRM class is used to represent people and/or institutions responsible for the safety and security of cultural entities and their environment. These are the people who (may) receive alerts from deciders in order to take appropriate actions in the case of risky events. As we said, the HP14 alerted property is used to associate an instance of HC14 Activation Event with an instance of E39 Actor, indicating the alerts sent to the specified individuals or groups. This property is of particular interest since it describes the modalities of interaction and collaboration between the digital system and the real world, making particularly evident, also on the ontological level, the importance of the human component for the effective prevention and management of dangerous situations.

A schematic overview of the RHDT Ontology, its entities, and their semantic relationships is illustrated in Figure 2.

6. A Summary of the New Classes and Properties

The new classes and properties are listed below with their scope note, i.e., a short description.

6.1. Classes

6.1.1. HC9 Sensor

Subclass of D8 Digital Device of CRMdig Ontology

This class comprises specialised devices utilised for monitoring and evaluating the conditions of artifacts, structures, or environments of historical significance. These devices are designed to detect changes in parameters such as temperature, humidity, vibration, or light exposure which may impact the conservation of cultural assets. Operated by dedicated software, sensors can exist in various states, including active, standby, or alert. For example, a temperature sensor may indicate fluctuations that could endanger delicate artifacts like manuscripts or artworks. When certain predefined thresholds are surpassed, the sensor enters an alert state, signalling potential risks and prompting appropriate intervention measures.

6.1.2. HC10 Decider

Subclass of PE1 Service of CRMpe Ontology

This class comprises software components responsible for receiving signals from sensors and autonomously making decisions aimed at safeguarding cultural objects. Acting as an intelligent service within the conservation systems of the digital twin, deciders process incoming data from sensors and information stored in the digital twin knowledge base to detect factors such as environmental conditions, artifact vulnerability, and conservation protocols. Based on this analysis, it executes predefined algorithms or decision-making rules to determine appropriate actions for preserving cultural heritage. These actions could include adjusting environmental controls, activating protective measures, or triggering alerts to conservators or relevant personnel when necessary.

6.1.3. HC11 Activator

Subclass of D8 Digital Device of CRMdig Ontology

This class comprises digital devices responsible for executing actions determined by the decider of the digital twin system to safeguard cultural objects. Serving as a crucial link between decision-making and practical implementation, the activator translates directives from the decider into tangible interventions aimed at mitigating risks or optimising conservation conditions. Interventions may encompass a variety of mechanisms, including the activation of mechanical, electronic, pneumatic, or hydraulic systems, among others. Activators ensure the timely and effective execution of proactive conservation strategies, contributing to the long-term preservation of cultural heritage assets.

6.1.4. HC12 Signal

Subclass of D9 Data Object of CRMdig Ontology

This class is used to model particular data objects generated by sensors and to document specific detected conditions. Signals are typically transmitted to the digital twin, where they are processed and analysed by the system’s algorithms to generate insights into the condition and conservation needs of cultural heritage. Signals may be encoded in a specific data formatting language, facilitating efficient transmission, storage, and analysis. Utilising standardised formats ensures interoperability and compatibility among different sensor systems and conservation platforms, including the digital twin knowledge base, enabling the seamless integration of data from diverse sources for comprehensive conservation management. The incorporation and analysis of encoded signals enables the digital twin to support informed decision-making and proactive preservation strategies for cultural heritage assets.

6.1.5. HC13 Sensor Measurement

Subclass of S21 Measurement of CRMsci Ontology

This class comprises specific measurement events in which a sensor detects and quantifies a specific parameter or condition relevant to the monitoring and conservation of cultural assets. Events of this kind occur when a sensor registers changes in parameters such as temperature, humidity, light exposure, or vibration, capturing information that reflects the environmental conditions surrounding cultural objects or structures. Sensor measurement is thus an essential feature for continuously assessing the time and circumstances in which conditions of risk may affect cultural heritage, providing valuable insights into factors that may impact the preservation of artifacts or sites over time.

6.1.6. HC14 Activation Event

Subclass of E5 Event of CRM Ontology

This class serves to model actions performed by an activator to initiate specific interventions or alerts aimed at safeguarding cultural assets. Activation events occur when an activator executes directives received from the digital twin’s decider, triggering actions such as activating climate control systems, deploying protective enclosures, or alerting personnel through various communication channels, including email notifications, SMS, and other similar ones.

6.2. Properties

HP11 was operated by the following:

• Domain: HC9 Sensor;
• Range: D14 Software.

This property links an instance of HC9 Sensor to the instances of D14 Software that operate it. The software is usually a piece of code running on the sensor and responsible for controlling and managing it, for instance by configuring its settings, collecting and processing its data, and generating signals based on its measurements.

HP12 was transmitted to the following:

• Domain: HC11 Signal;
• Range: HC9 Decider.

This property associates the instances of HC11 Signal with the instances of HC9 Decider, indicating that a certain signal has been transmitted to the digital twin decider for processing. The property can be used to model the flow of data from sensors to the deciders and can be useful for tracking the status of signals and ensuring that they are properly processed by the decider. The property can also be used to model the relationship between signals and the specific decider services that process them, allowing for more fine-grained analysis and optimisation of the digital twin system.

HP13 activated the following:

• Domain: HC14 Activation Event;
• Range: HC11 Activator.

This property associates the instances of HC14 Activation Event with the instances of HC11 Activator, indicating the specific digital device activated by the digital twin system based on the decision made by the decider component regarding the actions to be taken according to the detected risk.

HP14 alerted the following:

• Domain: HC14 Activation Event;
• Range: E39 Actor.

This property associates the instances of HC14 Activation Event with the instances of E39 Actor, indicating the action of alerting human personnel. The property can be used to model the communication between the digital twin and human operators, following the decision taken by the decider component, and to describe the modalities of collaboration and decision-making between the digital system and the real world.

HP15 was positioned on the following:

• Domain: HC9 Sensor;
• Range: HC3 Tangible Aspect.

This property is used to model the spatial relationship between instances of HC9 Sensor and the physical object on which the sensors are located. The property is fundamental to document the specific case in which a sensor is physically placed on or attached to a cultural heritage physical object rather than being simply placed nearby or in the same environment as the monitored object.

7. Example: Giovanni Pisano’s Pulpit in Pistoia, Italy

To exemplify the documentation of a system using sensors, we will consider a recent paper concerning the pulpit in the church of Sant’Andrea in Pistoia (Italy), a medieval masterwork by the Italian sculptor Giovanni Pisano described in [39]. The physical system includes sophisticated sensors and takes into account previous work by a worldclass restoration centre based in Florence, the Opificio delle Pietre Dure (OPD), as described in the above-mentioned paper. The array of sensors installed for the purpose of gathering environmental and dynamic data encompassed various types, among which were humidity sensors situated on the wall in the side nave adjacent to the pulpit, and uniaxial accelerometers positioned atop the pulpit and on the ground near its base. This specific configuration facilitated meticulous monitoring of both the environmental area surrounding the pulpit and any dynamic oscillations occurring on its surface.

To demonstrate the functionality of our ontology in representing this scenario, we specifically focus on these two types of sensors, illustrating how they can be semantically described using our classes and properties and integrated into the overall description of the installed monitoring system.

The semantic modelling starts by noting that Giovanni Pisano’s pulpit is a monument and thus a physical cultural object that can be represented by instantiating the HC3 Tangible Aspect class of our RHDT Ontology. The church of Sant’Andrea in Pistoia (Italy) can be represented by an instance of the E53 Place class of the CRM, while the fact that the pulpit is housed in this church can be rendered through the P55 has current location property of the same model. Specific identifiers for both of these heritage entities can be defined and/or derived using for example the global identifiers provided by Wikidata, such as the https://www.wikidata.org/wiki/Q3925522 (accessed on 20 May 2024) URI identifying the pulpit, or the https://www.wikidata.org/wiki/Q1148335 (accessed on 20 May 2024) URI identifying Sant’Andrea’s church in the Wikidata system. The HC2 Digital Twin class can be instantiated to define the Reactive Digital Twin of the pulpit in the semantic space of our model.

As previously noted, instances of the HC9 Sensor class could be employed to represent the diverse types of the sensors used to monitor the monument. We focus our example on two instances of this class used to represent the temperature and humidity sensor installed on the walls of the church and the uniaxial accelerator sensor positioned on top of the pulpit. Regarding their specific position, the distinct placements of these two sensors are rendered either through the CRM property P55 has current location, used to indicate the positioning of the temperature and humidity sensor on the wall of the church, and the property HC15 is positioned on, used to specify the placement of the uniaxial accelerometer sensor directly on the surface of the monument. The HP11 is operated by property is utilised to establish the connection between the sensors and the corresponding instances of the CRMdig D14 Software class, employed to represent the software operating them.

An associated instance of the HC13 Sensor Measurement class is defined to specify, through the CRMdig O24 measured property, the type of event monitored by each sensor, and specifically, the seismic movements detected through variations in acceleration by the uniaxial accelerometer, and changes in the temperature and humidity parameters of the church recorded by the temperature and humidity sensor. Both of these events are modelled by means of the E5 Event class of the CRM.

The signals resulting from these measurements are represented by using the HC12 Signal class and linked to the event that generates them via the L20 has created property of CRMdig. The signal transmission to the digital twin of the pulpit (HC2) is encoded through the HP12 was transmitted to property and the integrated monitoring system that receives them by instances of the HC10 Decider class. This is the system designed to acquire the transmitted values, analyse the various physical and environmental conditions of the monument, extend the information coming from other sensors, and enrich and interact with the digital twin’s knowledge base.

The sensor system of Giovanni Pisano’s pulpit does not include activators, but just for the sake of exemplifying how the system would be described when including an activator, we have added a hypothetic second part, not present in the actual system. In this supposed system, a decider (HC10) evaluates the variations in some measures from standard values and if they exceed a threshold, it sends an email to the OPD to intervene. This process can be semantically modelled by means of the O13 triggered CRMdig property and HC14 Activation Event representing the action of email transmission by the system to the OPD competent office (E39 Actor).

Figure 3 represents the knowledge graph of the example. It illustrates the semantics of this example in a simplified way to show an application of the RHDTO to a typical monitoring scenario for a heritage asset.

8. Conclusions and Future Work

In the present paper, we have addressed the introduction of sensors and activators in the RHDTO to describe the reactions of a system modelled as the digital twin of a heritage asset to external real-world events. The impact of such events is analysed in terms of measures of one or more physical quantities made by sensors and analysed by deciders which command actions (or possibly no action) to activators.

This model does not cover all of the possible reactions of the digital twin to external events. They may be described by information collected by external systems and forwarded to the digital twin system, such as weather forecasts or earth science information and then combined with other data from the decider, for example with the geographic location of the asset, ultimately leading to actions by activators in a way similar to the sensor–decider–activator chain but with the sensor replaced by the external information. A rather famous example is the MOSE system in Venice, used to limit the effects of flooding caused by high tide and strong northern winds on the monuments of the famous city by rising mechanical barriers at the lagoon mouths on the Adriatic Sea. MOSE bases its action on forecasting high tides, which have a well-known regularity, and wind strength and direction with weather forecasts. Although questioned in light of some anticipated environmental side effects, MOSE has so far succeeded in avoiding the effects of acqua alta (literally, “high water”, i.e., high tide) on the city monuments.

Finally, another possibility is the simulation of external events, digitally modelled in the system and described in the digital twin, to be analysed by the decider to produce as the output the simulated consequences, possibly also including the simulated effects of corrective actions by activators. For example, one might ask the system what are the effects of extremely hot summers on an art gallery and how are these mitigated by air conditioning. All such simulations will need scientific models behind the scenes and the related processing programmes, which may be stored in a computer system and documented with all of the other information in the corresponding Reactive Digital Twin.