The Evolution of Interactivity, Immersion and Interoperability in HBIM: Digital Model Uses, VR and AR for Built Cultural Heritage

Abstract

Today, a building is not just a “body” or a “machine” as defined by modern architecture, but rather an immaterial entity immersed in a digital world where not only its components but also the information associated with it are accounted for. In recent decades, building information modelling (BIM) has made it possible to move from 2D CAD drawings to 3D models capable of supporting different processes and interacting with different disciplines in the AEC industry for storing, documenting and sharing heterogeneous content. It has thus become possible to direct these techniques towards built heritage to investigate new forms of communication and share heritage building information modelling (HBIM) models. This research investigates this evolution in both generative terms (scan-to-BIM process) and cultural and historical terms in order to orient BIM uses towards novel forms of interactivity and immersion between users and models. The author proposes the use of a digital process and the development of VR and AR environments based on a visual programming language (VPL) to improve access to a deeper knowledge of HBIM models and the artefacts and information contained therein.

1. Introduction: The Digital Era in Cultural Heritage

The digital era we live in is characterised by a continuous evolution of knowledge, working methods and updating processes that we apply in our daily work. The fields of information technology (IT) and system programming have increased developers’ potentials, thus supporting the current needs of professionals involved in diverse procedures and disciplines [1,2,3].

In the last decade, world-renowned skills and methods have been developed in Europe based on new technologies to preserve, manage and leverage value from cultural heritage (CH) [4]. In particular, recent developments in computer graphics are re-engineering the construction sector and facilitating the management of existing buildings and complex structures [5,6,7]. The advantages of interactive graphics allow for the creation of digital worlds that can represent and manage our reality efficiently and rapidly.

CH exists in digital forms, both tangible and intangible [8,9]. Tangible heritage includes artefacts, buildings, structures, infrastructure and archaeological sites related to the natural environment and the materials from which all of these are made. Intangible heritage includes different values related to representations, practices, historical memories, knowledge and skills that communities construct, use and transmit from a complex and varied historical evolution [10]. Accordingly, digital heritage (DH) includes a considerable amount of information obtained with different types of new tools and methods, as identified in the Strategic Research Agenda (SRA) for JPI-CH (The SRA was developed purposely with the aim of presenting cultural heritage as a holistic, integrated research area. Using the JPI CH Vision Document as a starting point, further input was requested from a wide range of stakeholders across Europe reflecting the three key facets of cultural heritage: tangible, intangible and digital. http://www.jpi-culturalheritage.eu/wp-content/uploads/SRA-2014-06.pdf (accessed on 10 July 2021)) [11,12]:

Cultural heritage exists in tangible, intangible and digital forms. Tangible heritage includes artefacts (for example, objects, paintings, archaeological finds, etc.), buildings, structures, landscapes, cities and towns, including industrial, underwater and archaeological sites. It includes their location, relationship to the natural environment and the materials from which all these are made, from prehistoric rock to cutting edge plastics and electronic products. Intangible heritage includes the practices, representations, expressions, memories, knowledge and skills that communities, groups and individuals construct, use and transmit from generation to generation. Digital heritage includes texts, databases, still and moving images, audio, graphics, software and web pages. Some of this digital heritage is created from the scanning or converting of physical objects that already exist, and some are created digitally or “born digital”. Whatever its genesis, it needs constant maintenance and management to be retained.

Furthermore, the architecture, engineering and construction (AEC) sector has been subjected to a large-scale re-engineering process [13,14]. The advent and use of the latest generation of tools, such as digital modelling and virtual and augmented reality (VR-AR), have allowed new forms of digital management during a building’s lifecycle [15,16,17,18]. Thanks also to the integration of advanced 3D survey techniques, such as digital photogrammetry (terrestrial and aerial) and laser scanning with 3D modelling, it was possible to go beyond the digital management of new buildings to investigate and define new scan-to-BIM requirements to improve the digitisation process of built heritage [19,20,21,22]. In this context, heritage building information modelling (HBIM) has enabled advanced management of complex scenarios such as heritage buildings, archaeological sites, ancient infrastructure and monuments [23,24,25,26,27]. Unlike the first digital era, when 2D computer-aided design (CAD) drawing was the primary tool for architectural representation, HBIM has helped architects, engineers and archaeologists to achieve new levels of information related to the architectural and structural components of surveyed artefacts [28]. The transition from 2D representations to 3D informative models also allowed experts to improve their analyses, thus improving the sharing of information such as material characteristics of architectural components (physical and mechanical) and descriptive fields through schedules, databases, orthophotos, structural analyses and multitemporal designs [29,30,31]. As reported, some of this digital heritage is created from scanning or converting physical objects that already exist, and some is created digitally by different 3D modelling techniques, such as non-uniform rational basis spline (NURBS) algorithms and building information modelling (BIM) based on libraries of 3D parametric objects [32,33,34].

In this context, several studies have shown how digital models, related information, schedules and databases can be shared through digital twins, favouring information sharing and minimising data loss [35,36,37,38,39,40].

In the last few years, thanks to the development of VR-AR platforms, it has also been possible to increase immersion in digital worlds with VR headsets and discover environments that can represent buildings with a high level of detail (LOD), level of information (LOI) and accuracy [41,42,43].

Consequently, in the scan-to-BIM process, the interoperability paradigm has turned out to be a key factor for developing innovative forms capable of digitally documenting, sharing and managing complex scenarios [44,45,46].

This study investigates and shows the pros and cons of HBIM through several case studies conducted in the field in the last 10 years and how, thanks to IT developments, visual programming language (VPL) and models designed for interoperability, it is possible to increase the level of human–digital model interaction.

LODs and LOIs of HBIM objects have been oriented to different BIM uses, such as the design and decay analysis of the Basilica of Collemaggio in L’Aquila (Italy), the finite element and historical phase analysis of Masegra Castle in Sondrio (Italy) and monitoring of the Azzone Visconti Bridge in Lecco (Italy). Finally, through interdisciplinary projects based on extended reality (XR) techniques and AR-VR developments, research activities show novel results in serious games and interactive virtual museums of the Basilica di Sant’Ambrogio and the Arco della Pace in Milan (Italy). Thanks to the latest generation of devices, users can interact with models that can respond to the input given on the touch display or through a joystick, favouring a progressive discovery of information, data and geometric characteristics, moving from static HBIM models to novel forms of interactivity, immersion and interoperability.

2. Materials and Methods

In the last few years, AEC professionals have witnessed an epochal change that has led to a re-engineering of their daily practices. At the same time, platforms such as Unity, Blender, Unreal Engine and Twinmotion have provided a completely open logic, allowing for the development of digital worlds capable of interacting with users through different types of devices with new commands and programming languages, from mobile phones and tablets up to the latest generation VR headsets such as the Oculus Rift and Guest. In this study, exchange formats and new levels of interoperability are defined to transfer geometric and information data in XR environments, with the goal of minimizing the adaptation–orientation of the model. Case studies of the Basilica of Collemaggio, Masegra Castle and the Azzone Visconti Bridge are reported to analyse the pros and cons from a methodological point of view.

The work flows that led to the creation of HBIM models are analysed to highlight the limits of interoperability, interactivity and immersion. Once an understanding of the potential of each workflow and, above all, the limits has been achieved, the author proposes a process that can move from static models for purely professional use to interactive XR projects aimed at a wider audience. In particular, the Basilica of Sant’Ambrogio and the Arch of Peace required a new methodological approach to transfer the informative and geometric richness of scan-to-BIM models into immersive environments characterized by interactive virtual objects (IVOs). The choice of the two case studies was dictated by the fact that both buildings represent a valid and rich field of research from a cultural, historical and artistic point of view.

Thanks to the support of the municipality of Milan, ENAC was able to obtain permits for UAV surveys and complete the terrestrial surveys made over the years. Once the complete 3D survey was obtained, it was possible to recreate the two buildings and orient them towards a scan-to-BIM-to-VR process. The collection of historical information and drawings has been integral to increasing the virtual experience and investigating new forms of immersion and sharing of data. They were oriented to the virtual story of their historical, cultural and artistic background. Thanks to the integration of 3D surveying, HBIM and XR, it has been possible to explore new forms of interaction and immersion, increasing the LOI and dissemination for different types of users (AEC experts and non-experts).

The main research objectives of this study are as follows:

• To define and share the reliability of digital models through the quality control of the grade of accuracy (GOA) and standard deviation value between HBIM objects and point-cloud data.
• To enhance different levels of interactivity and interoperability for HBIM models using visual programming languages (VPLs).
• To test and develop various forms of immersion for XR projects, in VR and AR.
• This latter phase allowed the author to define a method for transforming static models to interactive virtual objects (IVOs) to answer user input.
• The paper is structured as follows:
• The first part is dedicated to the state of the art, with a synthetic framework oriented towards the level of interactivity and immersion of BIM uses, HBIM and VR-AR for the DCH domain.
• A description of the research case studies is provided from a historical and cultural point of view to show the evolution of interactivity, immersion and interoperability for Italian HBIM projects such as the Basilica of Collemaggio in L’Aquila, Masegra Castle in Sondrio and Azzone Visconti Bridge in Lecco.
• The description of the method is based on four stages (data collection, scan-to-BIM process, information mapping and information sharing) to improve the LOI of the HBIM projects of the Arco della Pace and the Basilica of Sant’Ambrogio in Milan.
• The conclusion discusses the results and related conclusions.

Figure 1 summarizes the proposed process with the aim of explaining the main phases:

• Data collection: Section 4 shows the methods, tools and technologies used to survey buildings, highlighting how the paradigm of the complexity of built heritage is a crucial field of investigation for the subsequent phases of analysis and use of HBIM models.
• Scan-to-BIM process: The following sections describe the process of interpreting and modelling different types of data, including primary data sources (point clouds from laser scanning and digital photogrammetry) and secondary data sources (historical records, drawings, archives and as-found documentation) to obtain HBIM models with different final purposes.

Information mapping: Once the HBIM models have been created through the integration of NURBS algorithms and the application of different degrees of generation (GOGs), the verification, importing/exporting and 3D mapping phases are described, verifying the reliability of the models from a point of view that is informative to the various professionals involved in the processes.

These first phases are described through three case studies with regard to their uses, and the pros and cons in the field of interactivity, immersion and interoperability are analysed. Once the advantages and limitations are explored, the final phase is investigated through two new case studies.

• Information sharing: Computer science development and VR-AR implementation have allowed the author to increase the use of models in the field of extended reality. The key factor that made it possible to increase the levels of interactivity and immersion and reach the latest generation of devices, such as VR headsets and AR viewers, is interoperability. This is addressed through the analysis, application and implementation of exchange formats with two main objectives: (i) to reduce the loss of information between one software and another and (ii) to make the entire process as sustainable as possible (scan-to-BIM-to-XR).

3. State of the Art

3.1. Digital Transformation: Uses of BIM in Architecture, Engineering and Construction (AEC) Industry

In recent decades, one of the most innovative technologies applied to the AEC industry is BIM. The concept of BIM was theorised in the 1970s by Charles M. Eastman of the Georgia Institute of Technology as a 3D digital reconstruction of the building process that facilitates interoperability and exchange of information in the digital format [47]. In 2002, the term BIM was coined by Jerry Laiserin [48]. In his manual, it was defined as one of the “most promising developments in the architecture, engineering, and construction (AEC) industries, with BIM technology and an accurate virtual model of a building constructed digitally. When completed, the computer-generated model contains precise geometry and relevant data needed to support the construction, fabrication, and procurement activities needed to realise the building”. From this definition, crucial adjectives such as “precise” and “accurate” emerged, referring to the 3D geometry of the models. The concept of the process to support the lifecycle of a building was introduced. The BIM benefits were found long ago, with one of the most important paradigms of models being their accuracy. The growing need for a standard definition to be included in national frameworks was due to the interesting perspective offered by BIM methodologies enabling rapid application by different stakeholders in the construction industry to support their specific professional activities [49].

Current studies define BIM as a tool that includes 3D modelling, interoperability concepts among various software applications and open formats such as the Industry Foundation Classes (IFC) (IFC is a standardized digital description of the built asset industry. It is an open international standard (ISO 16739-1:2018) that promotes vendor-neutral (or agnostic) and usable capabilities across a wide range of hardware devices, software platforms and interfaces for many different uses. More about IFC (developed by buildingSMART), its uses and adoption can be found here (accessed on 10 July 2021) [50,51,52,53,54].

The NBS National BIM Report (the annual NBS National BIM Report is now recognised as one of the industry’s most comprehensive reviews on the use of BIM. In the eighth report, achievements and challenges of the industry two years from the BIM mandate are shown. Reports are downloadable from: https://www.thenbs.com/knowledge (accessed on 10 July 2021)) (the industry’s most comprehensive report on BIM) looked at how building design professionals in the UK adapted to the use of BIM. Part of the government’s approach was to establish the UK as a world leader in BIM, having identified it as a means of creating global opportunities for the country’s construction sector [55].

The comprehensive scope was that this new digital construction technology would be taken to the next generation of the AEC sector [56,57]. The industry–government programme was focused on adopting BIM technology by private and public sector organisations involved in the procurement and delivery of buildings and infrastructure. The drivers for the BIM adoption were set out in the BIS BIM strategy. (The BIS BIM Strategy Reports describe the Working Group’s recommended strategies to deliver structured government/sector capability to increase BIM take-up over a five-year horizon as part of a joint plan to improve the performance of the government estate in terms of its costs, value and carbon performance. Reports are downloadable from: https://www.cdbb.cam.ac.uk/Resources/ResoucePublications/BISBIMstrategyReport.pdf (accessed on 10 July 2021)). Another significant statistic is the use of BIM standards and publications. Figure 2 shows the main list of the most famous references (specification of information management for the capital/delivery phase of construction projects using building information modelling. The document is downloadable from: https://www.bimhealth.co.uk/uploads/pdfs/PAS_1192_2_2013.pdf (accessed on 10 July 2021)) in the BIM field, such as the BS 1192:20017+A2:2016, the PAS 1192-2:2013, the RIBA Plan of Work 2013, the PAS 1192-3:2014, Uniclass 2015 and all subsequent amendments and standards published by the specialised control bodies over the years.

Beyond the specific percentage figures, the UK has shown full awareness of each definition and the subsequent level of communication achieved in this particular field compared to most other European countries. However, from a more general point of view, all of these statistics mainly consider the use of BIM for new buildings.

On the other hand, Italy is characterised by built heritage mainly known for its historical uniqueness, representing a foundational element of national and European identity [58]. Each generation finds itself with a considerable amount of capital resources, and the research must be focused on preserving this public asset for future generations.

As is well known, built heritage is made up of various streams that are valuable to society, history and art; specifically, human values, education and religion.

Focusing resources and scientific knowledge on the uniqueness and authenticity of heritage buildings against uncontrolled construction and planning processes represents a step in the right direction. The renewal component, represented by extraordinary/ordinary maintenance, is entirely overpowering the “new”. For this reason, building activity should be aimed, above all, at recovering and regenerating built heritage with an ancient history.

In light of recent EU initiatives, especially during the year of the European Strategic Research Agenda (SRA) for JPI-CH (EYCH2018)(the aim of the European Year of Cultural Heritage was to encourage more people to discover and engage with Europe’s cultural heritage, and to reinforce a sense of belonging to a common European space. An overview is downloadable from: https://europa.eu/cultural-heritage/european-year-cultural-heritage_en (accessed on 10 July 2021)), a shift may now be expected towards addressing cultural heritage-led development. The UN’s 2030 Agenda for Sustainable Development (This agenda is a plan of action for people, planet and prosperity. It also seeks to strengthen universal peace in terms of greater freedom. Poverty in all its forms and dimensions, including extreme poverty, is the greatest global challenge, and overcoming it is an indispensable requirement for sustainable development. An overview is downloadable from: https://europa.eu/cultural-heritage/about_en (accessed on 10 July 2021)) underlines that culture, including world CH and creative industries, can have a strategic role in achieving inclusive and sustainable development and contributing to the adoption of BIM for CH. In this context, the role of digital technology in improving awareness of, access to, management of, and engagement and democratic participation in DCH is important to investigate and demonstrate, addressing the challenges inherent in the evolving concepts of (digital) CH such as preventive conservation, co-creation and management [59]. Cultural and built heritage is the result of an extraordinary stratification of civilisations. Human settlement, over the centuries, has led to an accumulation of unique cultural heritage around the world. However, built heritage suffers from a lack of economic resources allocated to restoration, conservation and maintenance [60].

In the field of built and cultural heritage, the relevance of Italy is universally recognised; it ranks first for the number of sites registered as World Heritage Sites on the UNESCO World Heritage List (An overview of the UNESCO World Heritage List is downloadable from: https://whc.unesco.org/en/list (accessed on 10 July 2021)) (47, accounting for 4.7% of the total). According to the Code of Cultural Heritage and Landscape, (The Code of Cultural Heritage and Landscape is downloadable from: http://www.unesco.org/culture/natlaws/media/pdf/italy/it_cult_landscapeheritge2004_engtof.pdf (accessed on 10 July 2021)), the areas of particular value, subject to protection restrictions, cover almost half of the national territory (46.9%).

On the other hand, the construction market favoured the exponential growth of new buildings, causing areas to become uninhabited and disadvantaging the preservation and rehabilitation of existing buildings. In addition, the economic crisis prevented the advancement of experts who could manage built heritage through new technologies, penalising the re-engineering of the construction sector of the whole continent.

Another impressive figure is illustrated by the famous curve of BIM vs. traditional CAD planning by P. MacLeamy (Figure 3). He compared the traditional method based on 2D CAD drawing (yellow curve) with BIM (blue curve). The transition from 2D CAD drawings to BIM has already passed the experimental phase, leading organisations and businesses to develop guidelines, references and protocols to control the BIM generation. Workloads have increased, while constructions require new technological developments and the adoption of new technologies. Accordingly, the uses of BIM have grown with the companies’ need to remain competitive in the construction field, characterised in the last decade by a deep recession. Even today, this issue represents an obstacle for companies and professional firms that want to introduce this technology in their workflow, specifically oriented to the DCH domain and managing the complexity of heritage buildings.

3.2. BIM Uses and Heritage Building Information Modelling (HBIM) Projects: Orientation of Digital Models for the Digital Cultural Heritage (DCH) Domain and the Growing Need to Define Their Reliability

In the last decade, several methods have been proposed to generate very detailed models based on laser scanning and digital photogrammetry [61]. Interesting studies reviewed the state of knowledge of the HBIM methodology [62,63], reporting significant results that improved the use of BIM for the heritage field.

For current research, it has been advantageous to examine the uptake of HBIM models in reality, with the scan-to-BIM process being used successfully. To reflect this achievement, BIM users, project managers and researchers have published many works on BIM for existing buildings [64], underlining how IT and BIM interoperability will quickly change the “future of making”.

On the other hand, the design and data management categories represent the greatest benefits of this technology. They do not require development or scientific studies to improve the digital tools, which are quite up to the task. Therefore, in the last decade, the real challenge was not BIM with its benefits for new buildings, but built heritage-oriented research fields and HBIM and its interoperability. Accordingly, the scan-to-BIM process has been significantly improved to preserve and conserve built heritage [65,66,67,68]. The most commonly cited articles published in the field of HBIM highlight the importance of developing 3D modelling techniques and creating information models characterised by high LODs to create irregular architectural and structural elements that are not included in default BIM libraries. Reducing HBIM production costs by improving scan-to-BIM modelling requirements and the management of conservation plans for surveyed artefacts has been a focus of this specific research field.

In support of this new discipline, five LODs were established to protect the real level of accuracy and content in BIM models. The progress of building knowledge levels should follow the development of the models, which were created to share, insert and attach information to maximise support for the lifecycle of buildings [69].

In 2012, Bloomberg, Burney and Resnick, through the Department of Design and Construction in New York City, defined the BIM guidelines for the AEC sector [70]. This publication shows how to create new BIM guidelines, proposes new parameters for the AEC industry and demonstrates the need for national reference standards for the entire design process. The technological advancement involved continual updating of applications, procedures and national standards in order to legislate the whole construction sector.

Interesting studies in the UK have verified the emergence of an intermediate level LOD 350, in which BIM practice is mandatory. The fifth level is LOD 500 as-built, which supports asset management. On this basis, it was found that in most cases, a 350-definition level can be an excellent initial compromise to manage new buildings. It is possible to pass to the next levels following the design requirements and updates in the building process.

On the other hand, it is sometimes impossible to define a correct LOD for existing and heritage buildings. As is well known, in this context, laser scanning does not allow the automatic recognition of materials, resulting in the first issue that 3D modelling has to solve for proper BIM generation. The bidirectional relationship between model and information depends on the generation of 3D objects that can accommodate the intangible value of the surveyed artefact. Historical elements that are not present in the BIM libraries must be defined, checked and evaluated from a qualitative point of view.

For that reason, the “reliability” of the model should be based on concepts that evolve and not on the LOD variable, but on increasing the informative value of every element created. Geometry, accuracy and associated information become determining factors for defining LODs for built heritage. Consequently, the need arose to orient the main international guidelines of LODs to heritage buildings, defining scan-to-BIM parameters that accurately describe the quality of the model and its reliability for different types of analysis [71,72].

In this context, the latest national standards are often confused with guidelines for HBIM fields and the digital generation of existing buildings. In most BIM protocols, it is possible to orient the model to the desired LOD during the model’s lifecycle. Because of this, there is often a debate about what should and should not be memorised in the model. Accordingly, recent studies have shown how to convert the LODs for new buildings to the model requirements of heritage buildings [73,74,75].

Since 2012, research centres such as Autodesk Research and Bentley Systems have enabled the management of laser scanned data in their CAD and BIM software, fully grasping this market opportunity and kickstarting the scan-to-BIM era.

Autodesk recognised that the 3D survey market was ripe for a significant number of professionals to be involved in generating 2D CAD drawings and 3D digital models. It could offer a competitive product, providing advanced modelling tools to convert “dumb” scan data to smart HBIM models thanks to the integrated use of Autodesk ReCap, Autocad, Revit and other software solutions. Through new levels of automation and the computing power of the point cloud, software sped up the tedious generative process of registering, stitching and editing captured 3D survey data.

In this context, Murphy defined the term HBIM, describing it as a method for generating as-found BIM models based on point clouds coming from photogrammetry and laser scanning [76]. One of the main goals of this method is to improve the generation of conservation documentation oriented to building restoration, including orthographic projections, 3D documentation, sections, details and schedules (energy, cost decay), adding intelligence to point cloud data. Thanks to the integration of point clouds and 3D modelling, he launched BIM for built heritage.

The point cloud has also become the appropriate basis for the generation of historical infrastructure, leading BIM towards large-scale management (InfraBIM) and reaching proper integration with geographic information systems (GISs) [77]. However, the interoperability related to the generative process of as-found, as-designed and as-built models has been neglected due to the vast exchange of proprietary data formats, applications and 3D model types.

The following paragraphs describe the evolution of BIM uses oriented to DCH, where the paradigms of interactivity, immersion and reliability have been investigated from a scientific point of view to enhance the information sharing of tangible and intangible values of our built heritage.

3.3. State of the Art: Interactivity and Immersion with Virtual and Augmented Reality (VR-AR)

As briefly mentioned, the need to increase the sharing of information through HBIM resulted in a research field that has been widely explored in the last decade. On the other hand, the integration of VR and AR with HBIM has been partially investigated in the DCH domain. In any case, several studies have distinguished themselves in a particular way thanks to the integration of programming languages capable of increasing interactivity and immersion.

Intending to cover the state of the art relating to BIM adoption, the NBS’s 10th National BIM Report (2020) (The 2020 BIM Report marks the 10th year of asking industry professionals for their views about BIM (https://www.thenbs.com/knowledge/national-bim-report-2020; accessed on 10 July 2021)) included some significant data compared to the 2011 report:

• BIM adoption has grown substantially: In 2011, 43% of respondents had not heard of BIM, while today, awareness is almost universal, with 73% using it.
• The same benefits are consistently recognised: There is improved coordination of information, better productivity, reduced risk and increased profitability.
• The BIM standards are becoming embedded: Fewer people see BIM as simply 3D modelling, while the majority approach it by following either the PAS/BS 1192 (The PAS 1192 framework sets out the requirements for the level of model detail (graphical content), model information (non-graphical content, such as specification data), model definition (its meaning) and model information exchange) or BS EN ISO 19650 (The ISO 19650 standard is an international standard for managing information over the whole lifecycle of a built asset using building information modelling (BIM). It contains all the same principles and high-level requirements as the UK BIM framework and is closely aligned with the current UK 1192 standards. These standards are based on the UK’s standards for information management using building information modelling, namely BS 1192:2007 + A2:2016 and PAS 1192-2:2013. The principles remain per these standards with terminology changes being preserved by the UK National Forewords and National Annex. Another document was published alongside the standards: PD 19650-0: UK Transition Guidance, which, along with the UK National Forewords and National Annex, aid in the implementation of the ISO standards in the UK (https://0-www-bsigroup-com.brum.beds.ac.uk/en-GB/iso-19650-BIM/; accessed on 10 July 2021)) series of standards.
• Digital transformation is changing the industry and will continue to do so: Cloud computing, VR-AR and other technologies are being adopted by construction professionals (Figure 4).
• Survey respondents expect these to transform the industry and the way they work.

Consequently, research and continuous updating of commercial offering were key factors in increasing the share of intangible value linked to built heritage through serious games, virtual story-telling and interactive virtual museums [78,79,80]. As is well known, the origins of VR can be traced back to the creation of the stereoscope machine and the first VR viewer. Technology and development allowed us to create increasingly interactive and immersive devices over the next two centuries.

In 1977, Andrew Lippman released the Aspen Movie Map, which anticipated and preceded the famous Google Street View application by 30 years. The first interactive map was designed to provide a virtual tour of Aspen, Colorado; the application even allowed users to choose between three viewing modes (summer, winter or computer graphics).

In the videogame field, Nintendo launched its Virtual Boy game console on the market in 1995. The company managed to sell 770 thousand copies but had poor results. The screen resolution was low and displayed images in red and black, a few minutes of gameplay made people feel nauseous and it was too heavy to wear.

For almost 20 years, companies and the market were silent about VR, until 2010, when Palmer Luckey started a crowdfunding campaign to create the first low-cost viewer. This is how the Oculus Rift was born. Thus began the boom years of virtual reality viewers: from console viewers to PCs, smartphone viewers [81], plastic viewers with felt, and even cardboard viewers like Google Cardboard. In 2018, VR projects were expanding in an increasingly concrete way, so much so that the Chinese company Kat VR presented a compact version of the omnidirectional treadmill (ODT). This is a device that allows the user to walk, run and make movements with the whole body within a virtual world, thus pushing forward the future concept of interaction and total immersion.

As for AR, recently, an in-depth study titled Good Practices in the Use of Augmented Reality for the Dissemination of Architectural Heritage of Rural Areas, conducted by Merchán, Merchán and Pérez, reported on the state of the art [82]. This study analysed scientific advances in AR for the dissemination of cultural heritage, carrying out a bibliometric analysis on the Scopus database. One of the most important pieces of information is that cultural heritage studies in computer science, engineering and mathematics are the most numerous. Italy, the UK and Greece were shown to be the countries where integrating those disciplines into cultural and architectural heritage is more significant than in other European countries. The authors also identified that the factor that connects most of the analysed studies is 3D modelling. In particular, they found that 3D modelling of virtual environments, tracking of user locations and 3D registration of real images with virtual models are the most interesting research and development fields. Furthermore, other studies have underlined how the value of interactivity and immersion related to digital models, and more generally VR and AR, must ultimately also be analysed from a theoretical point of view in order to outline a study aimed at humanists and technicians. Consequently, based on the principles and objectives of this study, it is crucial to break down the concepts of interactivity and immersion concerning the potential and limits of VR and AR.

As for VR, in the 1990s, Steured [83] defined it as a system consisting of a set of devices that allow a new type of human–computer interaction, from which two essential aspects emerge: the set of IT devices and the new type of interaction. The first aspect focuses on technical IT characteristics, and the second refers directly to the type of experience that VR can invoke in the user. As is well known, a virtual environment requires three constituents: content, geometry and dynamics [84,85,86]. The content is defined by the objects and the information associated with them in the environment, and each represented thing can be described as an “object” of the VR world. It can also provide a graphic representation of moving objects, with which the user can interact, moving automatically within the VR world. Finally, the user is undoubtedly part of the content and constitutes the point of view from which the virtual environment is modified. The geometry of a VR world indicates the physical extension that the developer wants to attribute to the environment. The dynamics refers to the rules of interaction between the contents present in the creation of the environment. For a virtual environment to be “reliable”, the behaviour of all the objects within it have to conform to the normal laws of physics in the real world. By interpolating these characteristics, a virtual environment is created, meeting the definition of a “real illusion” in which one or more users can interact with it through the use of specific devices (PC workstation, laptop, VR headset, tablet or mobile phone).

On the other hand, AR is an interactive graphics system that allows users to intervene in a stream of images to modify reality with virtual content and animations in real time. One of the main differences between AR and VR is that AR is bound to an on-screen display, where any surface can support viewing the video stream.

In summary, in VR we find ourselves observing on a screen (including the VR headset) the projection of a world made up entirely of fictitious virtual objects, while in AR what is shown (not necessarily on screen) is a combination of real images running in real time and virtual objects. Therefore, VR is comparable to a virtual world, and AR can be compared to the real world plus virtual objects. Based on these concepts, this study reports decades of experience in the field of HBIM and extended reality (XR), in an attempt to define a scan-to-HBIM-to-XR approach that is as sustainable as possible.

4. HBIM for Digital Cultural Heritage (DCH)

4.1. Measurement Value in the Scan-to-BIM Process: From 3D Laser Scanning and Terrestrial and Drone Photogrammetry to 3D Modelling

Over the past decade, the transmissibility of data and new technologies were crucial in the generative process of heritage models. Surveying and sharing the uniqueness of historical shapes of built heritage handed down through the centuries is the primary goal of 3D data capture. Currently, market applications of 3D data capture technology, such as total station (TS), terrestrial laser scanning (TLS), unmanned aerial vehicle (UAV) and high-resolution cameras for photogrammetry, have been validated as quick and accurate methods for proper surveying of buildings. Thanks to the integrated use of point clouds from TLS, photogrammetry and TS, it has been possible to achieve a proper basis for the digital reconstruction of research case studies, which reduces return site visits and saves a significant amount of time. The LODs of 3D models and the GOA between point clouds and BIM depend on the quality of the 3D survey data (point clouds and geodetic networks). The main instruments used for the 3D survey of the research case studies were a FARO Focus Laser Scanner, a Canon EOS-1D X sensor, a Leica TS30 total station, an AscTec Falcon 8 drone equipped with a Sony NEX-5N RGB camera and a DJI Mavic Mini drone.

The first method of scan collection was laser scanning. Light amplification by stimulated emission of radiation (LASER) scanning is a method that permits the acquisition of different types of surfaces using a laser scanner that captures the real distance of scanned points over a given element, object or building. Points can be handled to generate 3D scans that can be adapted for use in 3D computer-aided design (CAD) modelling or BIM. The FARO Focus Laser Scanner provides a combination of accuracy and speed for non-invasive automated 3D scanning inspection, determining the locations of a vast number of 3D points and related spatial information (Figure 5).

It includes high-speed lasers with an integrated camera mounted on an adjustable tripod. It operates up to a range of 200 metres and at speeds up to 1 million points per second. The average scan resolution of the research case studies was about 40 to 100 million points per scan. Laser scanning is an accurate method for surveying unreachable and distant surfaces and multiform geometries. The clustering of exchange formats (such as PTS, E57, XYZ, RCP and PTG) allows the transmission of data into the modelling application without losing information, which improves the processing, cleaning, editing and measuring phases of scanning. The precision of the final registration of the surveys was 3 mm, and the point clouds were registered automatically by Autodesk ReCap 360 Pro software. The laser scanning survey of the Basilica di Collemaggio was made up of 182 scans (8 billion points), and that of Masegra Castle was 176 scans (7.5 billion points). The complexity of the Azzone Visconti Bridge required 77 scans (2.5 billion points), and the Basilica of Sant’Ambrogio was made up of 30 scans (1.1 billion points).

The second method of 3D data collection is photogrammetry, which is the science of deriving 3D geometry from photographs and determining 3D coordinates and mesh surface models. This technique maximally reduces the lack of 3D laser scanning surveys. Security structures, scaffolding and inaccessible areas (grey areas) found during the survey did not allow for proper positioning of the laser scanner in these areas. The integrated approach provided an alternative solution that drastically decreased the number of grey areas typically obtained from 3D laser scanners. Different strategies for photo capture were used depending on the morphological characteristics of the case studies. The primary goal was to improve the automation of digital reconstruction of complex objects, support the generative process and disseminate true orthophotos and textured models for restoration and conservation projects.

The third method was UAV-based data integration. The AscTec Falcon 8 drone equipped with a Sony NEX-5N RGB camera was useful for capturing pictures at high altitudes. The acquisition of images from the drone was instrumental in the photogrammetric reconstruction of the roofs of the Basilica di Collemaggio and Masegra Castle. In addition, the generation of true orthophotos from a set of UAV images and point clouds was integrated with those previously obtained by laser scanners. This allowed the actual thickness and current condition of the wooden roof layers to be determined, encouraging the development of a powerful tool to complete terrestrial reconstruction [87]. Managing such a considerable amount of points in the modelling environment was solved using automatic cloud decimation techniques. A DJI Mavic Mini was the second drone used for aerial photogrammetry of the Arco della Pace (Figure 6).

Finally, with the integrated use of the total station, it was possible to obtain a rigorous data registry by generating a geodetic network that verified, checked and updated the laser scanning data with the TS’s control points. The TS is an electronic/optical instrument used in modern surveying to read slope distances from the device to particular points of interest. The significant benefits of the total station are as follows:

• It improves the accuracy of measurement and eliminates manual errors.
• It provides fast and accurate calculation of coordinates (x, y, z).
• It makes automatic corrections for temperature and pressure.

The creation of geodetic networks composed of surveyed points and new points of interest allowed the proper referencing of all scans obtained with the laser scanner and photogrammetry, providing a single system with shared coordinates in all modelling software (Figure 7). Combining 3D building TLS with total station surveying led to improved data management for HBIM generation.

4.2. Evolution of HBIM Uses: From Historical and Cultural Backgrounds to the Generation of HBIM Projects for Multiple Uses

Three-dimensional modelling for a heritage building is a complex process that needs new requirements, new knowledge and continually upgraded know-how. This section describes the proposed method for addressing the current HBIM challenges and is intended to encourage a dialogue among operators, software developers and open source communities around automation technology. It was found that the paradigms of the interactivity, immersion and interoperability of HBIM were not always unidirectional. Indeed, the ability to achieve different types of analyses and disciplines allowed the scan-to-BIM process to become more holistic and bi-directional, which could increase the control of built heritage by the experts and disciplines involved, from design to the long phases of restoration, rehabilitation and conservation over time.

The planning of project goals at the inception of the generative process of the HBIM project enabled the development of an interoperable system and a working method aimed at enhancing information exchange at different levels of knowledge and automation, supporting the intrinsic and intangible value of five examples of Italian heritage architecture. The project increasingly required a system capable of holistically representing and sharing heritage buildings with high grades of generation (GOGs) [71]. The GOGs facilitated the exchange of information among the experts involved in the rehabilitation and conservation processes.

The following sections provide an extensive summary of the results obtained from the research over the years. HBIM projects are described to demonstrate how 3D modelling has been a key factor in the digital management of built heritage for buildings such as basilicas, castles and historical infrastructure and the management of novel XR projects (Figure 8).

At the same time, the limits found in the interoperability of the models are also described in an attempt to create a common thread among the various experiences reported and outline a common starting point for the development of XR environments that can go beyond previously achieved levels of interoperability and immersion.

4.2.1. Case Study of Basilica of Collemaggio after the L’Aquila Earthquake (2009)

The HBIM model of the Basilica di Collemaggio in L’Aquila was part of the rehabilitation project of the monument, which was seriously damaged by an earthquake in 2009. In particular, it is important to mention ENIservizi, which financed and supported the restoration of the Basilica and was awarded the European Heritage Award in the conservation category. The entire project was based on a public–private partnership that involved collaboration among three universities. The “Restart to Collemaggio” project funded by ENIservizi involved an interdisciplinary team composed of partners such as the Superintendence Office of L’Aquila, which carried out the restoration project, with scientific support from the Università La Sapienza di Roma and Università degli Studi di L’Aquila under the coordination of the Politecnico di Milano. The rehabilitation project required an innovative HBIM model in which to embed the complexity of a damaged historic building to support the project.

The main goal was to conserve the architectonic value of the Basilica and its structural safety. For that purpose, photogrammetric and laser scanning data were used as the primary data sources of the HBIM model. In particular, 3D modelling proved to be integral in creating complex HBIM objects able to transmit the richness of the monument for different analyses, such as the design, clash detection, simulation of structural behaviour, economic evaluation of the project and final restoration. The method guaranteed the management of the complexity of damaged elements, applying different GOGs and GOAs according to the project’s needs. Consequently, the HBIM model was crucial in highlighting the need for novel modelling procedures to differentiate their generative requirements from those of the leading international guidelines based on the LOD classification of new buildings. The second objective, of equal importance with regard to the first one, was to preserve the Basilica’s authenticity as much as possible and improve its future structural behaviour based on new innovative construction techniques. For this purpose, the generative process needed to take into account a novel level of interoperability able to transmit the complexity of the ancient elements, such as the central nave made of damaged stone pillars, the remains of the transept that completely collapsed after the earthquake and the perimeter walls with their geometric irregularities visible to the naked eye.

GOGs exploited the potential of each environment, even if they were not interoperable with each other, overcoming some critical aspects highlighted by the BIM applications in this border experiment. The program, in particular, envisaged the creation of a model capable of communicating dynamic information of a geometric, material and design nature, the latter alternating with the preliminary design, definition and execution phases of Italian projects. Figure 9 shows the level of dynamism and interactivity achieved for the Basilica of Collemaggio in L’Aquila.

At the same time, the need to orientate the scan-to-BIM model to the requirements of different disciplines was found during the generative process. It was thus decided to create a digital model corresponding to the reality surveyed with a high GOA and then orient the BIM by lowering or increasing the details according to the project’s needs.Consequently, before carrying out the transformation from NURBS to BIM objects, the surfaces were analysed to improve the system, certifying the model’s quality. For this purpose, based on an analysis of the surface curvature, an automatic verification system (AVS) was identified to calculate the grade of accuracy (GOA) and provide an accurate value of the standard deviation between the NURBS surface and the 3D scan (Figure 10).

Point set deviation was identified as the most suitable type of analysis of the proposed AVS to evaluate the GOA of the model between point clouds and NURBS surfaces. A value is provided to quantify the quality of models produced directly from the acquired point cloud. This result provides numerical values representing the standard deviation between point clouds and the whole model without defining the parameters. This solution is an overall indication and can be reused to improve model quality over time. Thanks to its direct computation, it saves a considerable amount of time. The values used are as follows:

• Point count: The total number of points n used in the comparison;
• Mean distance: The average distance between the model and the point cloud;
• Median distance: A robust index used to evaluate the presence of gross errors when compared to other indices (the median separates the higher half of r_i values from the lower half);
• Standard deviation: An index obtained from the squared distances between the model and the point clouds.

Using the case studies, it was possible to apply the proposed method over the years and see how optimizing the modelling procedures allowed for a very high GOA through new generative requirements such as GOG 9, and especially GOG 10. Figure 10 and Figure 11 summarize the process applied to the damaged umbrella vault of the Basilica of Collemaggio.

Furthermore, based on the development and appropriate use of various exchange formats, it was possible to increase the use of BIM for finite element analysis (FEA) [88], CoSIM [89] and decay analysis [90]. Based on the assumption that historical constructions made of specific structural elements such as irregular walls, arches and vaults are appropriately represented by 3D objects in clustered BIM databases, a generative process that could create specific and accurate 3D historical elements such as decayed areas, wall stratigraphic information and structural failure was also investigated (Figure 12). Based on the same logic of the generative process of complex walls, GOG 10 was applied to the north wall and the umbrella vault of the Basilica to create a holistic system that could transmit specific information. The bi-directional relationship between object and information in Autodesk Revit and the use of complex NURBS surfaces led to the creation of thematic mappings in the HBIM with high LODs and LOIs.

A second result was found by transferring the model to software for finite element analysis. With the use of generative tests based on different schemes and exchange formats, it was possible to maintain the complexity achieved by the scan-to-BIM model for structural studies. In particular, different studies have benefited from the proposed method of transferring the complexity of the model through the NURBS interpolation in software such as Midas FEA and Abaqus, which can automatically transform complex NURBS surfaces into mesh models for different software.

Another significant application of the model was in the construction phases and consolidation operations of the Basilica. With the direct use of the HBIM model, it was possible to move directly to the design and dimensioning of the building structures for the reconstruction of the Basilica, such as scaffolding to measure vaults, aisles and pillars and the design of specific arch supports. This approach helped in the study of single activities for safety issues and more complex activities for calculating time and cost. The level of detail of specific elements, such as the pillars, was reduced to add specific operational information. These types of LODs led to an analysis of different types of operations in the CoSIM implementation to estimate the total time to replace a single damaged element.

4.2.2. Levels of Interoperability, Interactivity and Immersion of the Basilica of Collemaggio HBIM: A Critical Review

The HBIM model of the Basilica of Collemaggio was the time a large heritage building was digitized in Italy after a seismic event. As mentioned above, the scan-to-HBIM process, the application of GOGs for complex elements and the defining of AVS to communicate the reliability of the recreated objects made it possible to support heterogeneous analyses and processes, from design to construction site to structural analysis. On the other hand, it is also important to note that the multi-user cloud logics were not fully assimilated 8–9 years ago. Consequently, real-time sharing and updating of the model via a cloud platform was not done by all professionals involved. As is well known, the various exchange formats represent specific model uses.

The HBIM model represents a starting point for professionals involved in the process, in both methodological and scientific terms. The generative process was based on a conversion logic according to project needs. Interoperability mainly resulted in a one-way flow in which the model’s conversion into a specific format represented the final output for the end user who inherited the model. This first experience also highlighted the urgent need to manage the time variable more efficiently in order to acquire, share and view versions of the model (from design to preservation) in real time (automatic synchronization). The main obstacle encountered in this sharing phase was that not all professionals could manage the Revit project or benefit from subsequent sub-versions of the model, such as models focusing on decay and energy analysis. Consequently, the model’s level of interactivity and immersion was limited to a small circle of expert users. For this reason, the research and development activities related to the two subsequent case studies were oriented towards increasing the level of sharing in the models via cloud logic and a 3D web interface that did not require specific BIM skills.

4.2.3. Case Studies of Masegra Castle and Azzone Visconti Bridge

A second opportunity to improve and share interactive information through HBIM models came with the case studies of Masegra Castle (Sondrio) and Azzone Visconti Bridge (Lecco). Thanks to in-depth historical research of the artefacts, it was possible to identify historical phases for the castle and insert levelling and monitoring data for the medieval infrastructure to facilitate the monitoring process through archiving and reading heterogeneous data.

The HBIM of Masegra Castle allowed an improvement of the method applied to the Basilica from both the morphological and typological points of view. Management of the shapes was enhanced by the introduction of volumetric subtraction techniques to remove superfluous wall portions and obtain an appropriate GOA for structural elements such as vaults, walls, bastions and towers, and to further reduce the generative process in terms of time and cost. Unlike the Basilica, for each constituent part of the castle, it was first necessary to find a GOG and GOA corresponding to its complexity and define the modelling approach beforehand. The large volume and number of elements did not allow the author to freely test other forms of modelling. Consequently, based on the results obtained in the first case study, the same generative process was applied to further improve automation. However, the morphological complexity of the surveyed elements of the castle presented a new problem: a large amount of point clouds produced by the 3D survey highlighted complex shapes of each single wall element on the inside and outside.Therefore, it was necessary to improve GOG 9 and 10 to restore wall elements to incorporate irregularities on both sides. The generative gap was overcome thanks to the integrated application of GOG 9 and 10 and the use of subtraction voids (GOG 8). Furthermore, several phases and a digital representation were identified, which involved applying different LODs and GOAs to develop a dynamic graphic information display, thus increasing the interactivity compared to the first case study. This required the identification of objects that no longer corresponded only to complete wall partitions and subdividing the latter in semantic and temporal terms. Figure 13 shows how improving the 3D modelling and the targeted application of the various GOGs also improved the interaction between users and models, obtaining a BIM that can change phase according to the selected historical period. The introduction and application of this new function were carried out in Autodesk Revit to diversify the building status and the newly designed elements.

A further improvement of the proposed method was achieved in the structural analysis and the representation of uneven wall irregularities in Autodesk Revit. As demonstrated, with the use of the NURBS interpolation and the appropriate application of GOG 10, it was possible to define a workflow that could transmit cracks to the structural software, improving the analysis and the simulation based on ultra-defined BIM [91].

In particular, it was possible to intercept masonry anomalies and transfer them from Autodesk Revit and Mc in Rhinoceros to Midas software in order to transform a complex model into a mesh model oriented to structural analysis. A diagnostic analysis was needed to combine material properties and geometric information. For the case of Masegra Castle, non-destructive tests (NDTs) and minor destructive tests (MDTs) were planned in order to characterise the construction materials and techniques, the state of conservation, the mechanical properties of masonry, foundation levels, and the thickness of vaults and retaining walls. In a first step, conversion tests were used to evaluate the required simplifications for numerical computation that could exceed the memory capacity of computers. The results obtained helped calibrate the structural simulation and evaluate its GOA. The HBIM and its geometric information were turned into a solid tetrahedron mesh to preserve the information captured by the integrated use of GOG 9 and 10. The input for Midas was a mesh with a closed surface and a perfect face-to-face correspondence.

The choice of a tetrahedron subdivision was motivated by the automation of the BIM-to-mesh conversion. The main drawbacks of this kind of mesh were the final number of elements and the higher CPU cost during the simulation, which can be much larger than those of other meshing algorithms. The mesh model was split into multiple subparts to link appropriate material characteristics based on inspections. The subdivision included a historical analysis of the castle along with additional considerations about the state of conservation. In particular, irregular edges and small objects could be transformed into distorted elements without applying the GOG modelling requirements. Thus, the automated conversion into a consistent mesh was a highly complex task, especially for the kind of analysis required, which is based not only on geometric constraints but also structural considerations and an interpretation of functional behaviour.

As the BIM reflected the geometric complexity of the historic construction, the finite element model reflected the geometric irregularity without excessive simplification of the structural elements. The uniqueness and authenticity of heritage buildings were preserved for a more detailed simulation with limited extra cost in terms of processing time (Figure 14). Although manual editing was required to obtain a consistent mesh for structural simulation, the performance of the scan-to-BIM-to-FEM methodology was remarkable and merits future work on the automation of the different phases, starting from better algorithms for mesh generation.

Despite the results obtained for the latter two case studies, both models and their related calculations and 2D representations (plans, elevations and sections) could be viewed and analysed through the BIM platform on which they were developed and the BIM-based software used for subsequent analyses. Consequently, the informative value and usefulness of the models were intended for a small circle of experts to interact with and manage complex models using professional software requiring advanced skills. Thus, the growing need to go beyond BIM platforms such as Autodesk Revit and Graphisoft ArchiCAD and facilitate the use and reading of informative models has required the development of methods that can guarantee novel levels of interactivity and immersion of the HBIM. In recent years, one of BIM’s major automatisms has been neglected due to the benefits gained through its multi-tiered management functions. The case studies of the Basilica and the castle highlight these aspects.

At the same time, the case study of the Azzone Visconti Bridge has made it possible to maximise one of the main tools of BIM applications: the automatic extraction of drawings from BIM models. For the first two case studies, 3D drawing and GOGs weres preferred to BIM modelling for the most part. For the bridge model, instead, it was decided that an ultra-detailed model would be created immediately and then, in a second phase, drawings useful for structural analyses would be generated. Thanks to the high GOA, it was possible to use the internal functions of Autodesk Revit to obtain a large number of vertical sections passing through the pillars and arches to show the asymmetries present between the north and south front. Every barrel vault was characterised by irregularities along the entire intrados of the bridge.

Thus, it was decided in this case to make explicit the structural irregularities of the bridge through 22 cross-sections. In this specific case, the drawings were more valuable than the BIM itself from a structural point of view. Using the discontinuities intercepted in the bridge, subsequent structural analyses were carried out to demonstrate the bridge’s stability under load and without load [92].

4.2.4. Levels of Interoperability, Interactivity and Immersion of the Masegra Castle and Azzone Visconti Bridge: A Critical Review

The growing need to share large amounts of information, characterised by different digital formats, required the development of a cloud environment able to connect all of the experts involved in the process. Nowadays, most existing buildings are equipped with sophisticated building control systems (BCSs) that collect significant quantities of data for different types of analysis. These systems support the generative design of building managers to minimise long-term operational costs, ensuring occupants’ comfort. A key challenge was to define a system able to organise, study and share data for any artefacts, extending BIM for use as a tool throughout the lifecycle of the building. In this context, a more interactive approach was created to improve the overall operational requirements and manage complex building information.. Thanks to cloud logic, different types of files can be shared in the cloud, thus avoiding lengthy updating procedures between surveyors, engineers, architects and restorers. This shared logic is supported by the web-based cloud BIM platform, which allows users to share, update and modify any type of file, from simple PDFs to more advanced files such as DWF, RVT and NWC. The main advantage of this system is that it can share complex models even with users who have a low level of knowledge about these new BIM technologies. Visualisation and interaction with the model are flexible and fast at the same time. The generation of HBIM for a medieval infrastructure and a cloud system able to share the levelling results has led to a repository for SHM [93].

Figure 15 shows the level of interactivity and immersion of the digital model.

The results reported here for the three case studies led to the analysis of various methodological aspects that are, for the most part, not connectable to each other. An extracted summary showed that one of the significant “digital” limits is linked to interactivity itself.

The paradigms of interactivity and immersion of digital models in recent years have been investigated hand-in-hand with the possibility that computer development languages have mainly been given to BIM researchers, who can combine different disciplinary fields in a single digital project. As briefly anticipated, today it is possible to develop graphical platforms and interfaces thanks to the study, analysis and re-elaboration of specific APIs shared by the leading architecture software manufacturers. Thus, the common thread that binds these case studies turns out to be precisely the paradigm of interactivity. This new paradigm is broken down in digital form through two case studies, the Basilica of Sant’Ambrogio and the Arco della Pace in Milan (Italy), which did not require efforts and developments aimed at supporting the lifecycle of the buildings, but rather a virtual environment that could align with the new needs of virtual museums.

The next section shows how the scan-to-BIM process and HBIM models can be oriented towards advanced forms of XR reality that can virtualize complex museum itineraries and achieve new forms of interactivity and immersion concerning the case studies described so far.

5. Novel Levels of Interactivity and Immersion of HBIM Models: From the Scan-to-BIM Process to Interactive Virtual Objects (IVOs) and AR Libraries

The new era of technology is based on development and computer programming able to support market needs. The open-mindedness between development and market needs can encourage researchers to create novel, innovative solutions to enhance products, activities and workflows. In recent years, one of the main applications proposed in VR and AR is the integration of new devices and tools such as Samsung Gear, Microsoft Hololens, Oculus Rift and Guest, which can take users to new levels of immersion in digital models. Thanks to these innovative technological developments, it has been possible to start a phase of experimentation in the field of XR in order to increase the LOIs through more increased interactions between models and users. In order to increase the sharing of information, generative tests and new immersive environments were developed for the case study of the Basilica di Sant’Ambrogio in Milan, laying the foundation for multi-level information environments.

XR development platforms have been used in the construction field to increase the digital sharing of architectural projects. At the same time, platforms such as Unity, Blender, Unreal Engine and Twinmotion have allowed, in a completely open logic, the development of digital worlds capable of interacting with users through different types of devices and new commands and programming languages. Thus, the case study of the Sant’Ambrogio Basilica was oriented towards the virtual visual storytelling (VVS) of its historical, cultural and artistic background.

The most commonly implemented forms of VR are:

• First- and third-person experience: Users can immerse themselves in the scene and explore the space freely through the construction of an avatar;
• Interactive virtual environment (IVE): The VR experience is based on interactive virtual objects (IVOs);
• Visual programming language based on Blueprints;
• Three-dimensional mapping based on high-resolution orthophotos.

Thanks to different devices (hand controls, motion sensors, VR headsets), 3D modelling, novel exchange formats and computer programming, it was possible to expand the 3D scene by developing new “moving” virtual 3D objects. Consequently, it was necessary to use scripts to add interactivity to VR projects. The scripts reside in the project like the other assets (textures, models), and a file represents each script. XR development platforms allow for the use of informatic languages such as C#, Javascript and Boo. C# is the most commonly used language in Unity and Unreal Engine.

Unity scripts are classes inherited from MonoBehaviour. A class identifies a category of objects that all have the same properties. Each class is contained within a text file, which contains all the properties of the objects of that class. A good part of the work consists of creating the desired effect in object modelling and Unity scripting. However, this often requires long generative times, which restricts the development possibilities. Three-dimensional objects for VR implementation (mesh models) are characterised by a series of vertices, edges and triangles/polygons. In particular, the vertices are points in 3D space, the edges (or segments) are the lines connecting them and the polygons are the surfaces formed when the edges join three or more vertices. This applies to all engines and is always kept in mind when specifying a model’s “polygonal resolution” for serious games and virtual museums.

Many formats are imported and they are generally equivalent, so the modelling techniques and the software can be instrumental in generating hyper-realistic objects, thus favouring proper interaction between users and objects.

An innovative approach is to generate customised VR objects that can exponentially simplify the modelling phase of both the scene and the objects. Thanks to new levels of interoperability between modelling applications and XR development platforms, it is possible to develop more detailed experiences in less time. In particular, novel IT developments such as real-time rendering, blueprints and FBX format have made it possible to create models and XR projects (Figure 16) simultaneously.

This specific digital workflow allowed the author to do the following:

• Carry out modelling and programming at the same time;
• Verify changes in real time and avoid modelling and geometric changes in the XR development software;
• Define a 3D mapping process based on textures from digital photogrammetry, avoiding textures and generic images from the default libraries;
• Extract 3D urban contexts without having to model them thanks to a recent tool developed in Twinmotion;
• Georeference the models correctly;
• Make use of temporal (phasing), daylight and season and lighting simulations;
• Take advantage of different types of exports and outputs, such as rendering, 3D animations, panoramas and VR environments;
• Move from static mesh to IVOs.

In this context, 3D modelling and programming become indispensable factors for the generation of VR experiences (VR-AR) to achieve a strong sense of being present in a virtual environment enriched by IVOs with a high level of content.

Consequently, it was found to be necessary to fully understand the added value of the visual programming language (VPL). This was the key element that made it possible to achieve the objectives in Section 2, combining disciplinary sectors and development phases related to geomatics, representation and information technology. Thanks to open development logic, it was possible to give life to static objects by developing gameplay mechanics or interaction logic with a visual, graph-based system instead of writing traditional lines of code. It was found that non-programmers such as architects, engineers and AEC professionals can use a workflow that uses visual, node-based graphs to design behaviours rather than writing lines of C# script (Figure 17). In particular, Blueprints (Unreal Engine visual scripting) and State Graphs allowed the author to create self-contained behaviours that tell objects what actions to execute when they are in a particular state. They were developed for high-level logic such as AI behaviour, scene or level structure, or any aspect of a scene that requires behaviours that transition between states. Unreal Engine and Twinmotion turned out to be the best performers in terms of development, synchronization and interoperability. With the use of advanced real-time 3D creation tools, it was possible to migrate the HBIM models into the XR platform without saving the model and exposing it in turn. With an add-in inside MC Neel Rhinoceros and Autodesk Revit, it was possible to synchronize the model and the scene with the Twinmotion and Unreal Engine platforms in real time. In a completely open logic, software applications, 3D modellers, visual scripting and interfaces oriented to VR and AR devices have been merged into a single digital process.

Furthermore, the production-ready support for the OpenXR framework, the multi-company standard for XR development, enabled the targeting of multiple XR devices with the same API, offering support for additional features. In this process, the cloud became an increasingly important component of the technology ecosystem, including real-time content development. Pixel Streaming enabled Unreal Engine and applications built on it to run on a high-powered virtual cloud machine and to deliver a full-quality experience to end users on a normal web browser on any device anywhere.

Thanks to this new feature, it was possible to orient the VVS of the Basilica of Sant’Ambrogio and the Arco della Pace for mobile phones, VR headsets and PC workstations.

Following an in-depth study of historical records, a different implementation phase was conducted with the final goal of achieving an alternative form of human–model immersion: AR.

In particular, AR was considered by the author to be a suitable solution for different purposes, such as [88]:

• Enrichment of human sensory perception through the superimposition of digital content united with palpable reality;
• Content enrichment by adding multiple layers of information, such as text, images, videos and holograms;
• Provision of strategic information on the surrounding environment;
• Creation of virtual tours of structures and places to visit, historical and structural information of monuments and interaction with works of art exhibited in virtual museums;
• Making the travel experience more engaging, viewing an object from the comfort of one’s own home before proceeding with the visit;
• Creating interactions that could not exist between the user and reality;
• Adding complementary information in real time without the user having to search for it.

Figure 18 shows the web-based AR library developed for the Arco della Pace in Milan. XR objects are easily accessible in AR mode. The associated information was selected to reach the user in a targeted manner (on-site or remotely) with precise, concise descriptive text, thus providing a cognitive approach to the historical background of the Basilica and the decorative schema of the monument.

6. Discussion and Critical Analysis of the Proposed Workflow

The case studies of the Basilica of Collemaggio, Masegra Castle and Azzone Visconti Bridge demonstrate how HBIM models can be useful for different purposes. However, not all of the solutions reported present high levels of interactivity and immersion. These two factors are now decisive for many other sectors, such as entertainment, gaming, museums and tourism. Inevitably, professionals who want to improve information-sharing through digital models must understand and assimilate novel IT development techniques. For that reason, the case studies of the Basilica of Sant’Ambrogio and the Arch of Peace were found to be areas of development and research useful for fully understanding (minimum and recommended hardware specifications and necessary software for developing with Unreal Engine, with specific hardware and software requirements for running the editor (https://docs.unrealengine.com/4.27/en-US/Basics/InstallingUnrealEngine/RecommendedSpecifications/; accessed on 10 September 2021)) the advantage of visual programming language (VPL). The proposed process is based on the production, analysis, creation and transformation of a large amount of data. Accordingly, the knowledge of many latest-generation applications is necessary to develop HBIM models and VR-AR experiences. In addition, the continuous advancement of technology requires professionals to continuously renew their daily practices. In this context, the interoperability paradigm of digital models becomes a fundamental factor to be kept under control. Exchange formats (proprietary and open), export schemes and software applications have become the main digital tools for AEC professionals.

Consequently, hardware and software represent essential factors for correct implementation. The FBX and OBJ formats are the best performers for migrating geometric and material information in the early stages of the scan-to-BIM process. On the other hand, the proposed workflow still presents a large number of transformations. However, the method uses real-time synchronization between the main modellers (Mc Neel Rhinoceros and Autodesk Revit) and the XR development platforms (Unreal Engine and Twinmotion). Thanks to this development, it is possible to minimize exports and transformations of the various models.

As shown in Figure 19, the devices used for the Basilica of Collemaggio and Masegra Castle were limited to PC workstations and laptops. The main types of storage were servers, hard disks and PCs. The generation of the multi-user model had not yet been considered. Consequently, the generative process was one-way: once the information mapping phase was completed, the models were shared for subsequent analysis. However, with the case study of Azzone Visconti Bridge, it was possible to develop a cloud platform that could share models and information coming from the 3D survey and levelling. The main devices for sharing information and viewing 3D models were mobile devices (mobile phones and tablets). As a result, the immersion levels of the models were limited. The models could be navigated through keyboards and touch screens and viewed through traditional monitors. On the other hand, thanks to the integration of the VPL and the development of new immersive tools such as the Oculus Rift, it was possible to investigate more immersive forms of interactivity between models and users, passing from static objects to IVOs.

In Figure 19, in addition to descriptions of the primary development environments and devices, a comparison between VR and AR highlights critical issues and possible future investigation and development phases.

Finally, an implementation difficulty related to VR and AR implementation was encountered due the systems and software used. The minimum recommended hardware specifications by Epic Games for Unreal Engine13 to develop a high-level experience are: Windows 10 64-bit, 64 GB RAM, 256 GB SSD (OS drive), 2 TB SSD (data drive), NVIDIA GeForce GTX 970, Xoreax Incredibuild (devtools package) and six-core Xeon E5-2643 @ 3.4 GHz.

For a mid-level experience, the recommended parameters are: Windows 10 64-bit operating system, Quad-core Intel or AMD processor, 2.5 GHz or faster, 8 GB RAM memory, DirectX 11 or 12 compatible graphics card.

Both solutions are still addressed to a limited and highly specialized audience and to the fields of information technology and programming. Thus, future developments will be aimed at reducing the technological gap between users (professionals and non-professionals) with the latest-generation VR-AR tools, increasing the usefulness of HBIM models for the art, tourism, culture and entertainment sectors.

7. Conclusions

The digital acceleration due to the pandemic has influenced the way we communicate and share information. In the time of COVID-19, over about four months in 2020 with closed doors, the Italian state museums, monuments and archaeological areas presented the results: visitors were down by 75.67% and net revenue was down by 78.98% for a total loss of 41,991,929.03 EUR for the 268 institutions monitored by the Sistan (National Statistical System of the Ministry of Culture). Today, innovative studies offer digital solutions that can support the art, tourism and cultural sectors. These systems are joining and integrating more and more with social platforms, education and entertainment. In this context, the most investigated solutions are VR and AR, which are going beyond the simple 3D representation of digital space. Meanwhile, virtual and augmented reality (VR-AR) appear to be research fields not yet fully investigated in the construction sector and the digital cultural heritage (DCH) domain.

Based on these circumstances, this study proposes a methodological approach based on the latest generation of tools in the field of VR and AR to improve the interoperability and use of HBIM models to create new forms of extended reality (XR).

Based on the research results reported in this study, it is possible to verify that virtual space offers a much stronger sense of presence than traditional forms of data sharing. With VR and AR, users feel closer and are in more direct contact with buildings, information and experiences that, in most cases, they can not reach or experience in person. Through several case studies, the author shows the pros and cons of HBIM projects carried out in the last decade and tries to define a process that can support professionals and not BIM users in creating new digital experiences such as virtual museums and serious games.