Mapping the Soundscape in Communicative Forms for Cultural Heritage: Between Realism and Symbolism

Abstract

The dimension of sound plays a central role as a form of cultural representation. Sound is a means of knowledge and experiential involvement, as it is inextricably linked to place and space, mind and body, cultural context and emotion. This contribution aims to explore how sound design follows different paradigms and methods in the various media. Virtual reality, videogame, cinema and documentary have differently codified rules to provide acoustic verisimilitude to the simulated space, to orient or stimulate the user, to suggest contents or evoke events and to emotionally involve the public. These rules follow artistic principles closer to psychoacoustics than to scientific reproduction of sound in the simulated space. Under what conditions, however, is the scientific simulation of an acoustic space preferable to the more common paradigms of psychoacoustics? How could this be created? Immersive and non-immersive virtual reality for cultural heritage is currently the field of experimentation most open to future developments. Some virtual reality and mixed reality applications will be presented, dedicated to archaeological or historical-artistic contexts, where a fundamental relationship between sound and multisensory interaction has been created.

1. Introduction

1.1. Ambition

This contribution stems from the academic and artistic background of the author, as a cultural heritage conservator, art historian and musician. Since the end of the 1990s, the author has carried out research activities at the Institute of Cognitive Sciences and Technologies (ISTC) and the Institute of Heritage Sciences (ISPC) of the National Research Council (CNR), in the field of multimedia technologies for the communication of cultural heritage, museum studies and virtual museums. In this field, she has specialized in experimenting with new forms and languages that combine narration and interaction. This research has led her towards a progressive convergence of different media: virtual reality, videogame, mixed reality, documentary, theater and cinema, in the conviction that an engaging cultural experience for the public originates from the indissoluble union of science, art and technology [1].

The research presented here aims to underline the importance of sonic heritage, and sound design in particular, applied to audio-visual media for the valorization and narration of cultural heritage.

Nowadays, cultural heritage experts seem to be quite familiar with the importance of the auditory domain. Important studies on historical acoustics have been developed, and many researchers have highlighted the importance of a multi-sensory approach in the transmission of tangible and intangible heritage as well as in the design of “holistic” cultural experiences. Similarly, in the last years, many museum exhibits have been conceptualized as multi-sensory projects [1].

However, this awareness has not yet led to adequate consideration of all the sensory dimensions necessary for the perception and understanding of cultural evidence in museums. The perception of sound contents is very much disregarded in museum spaces, and not-sufficient care has been addressed to architectural setup, technologies and daily management. In the communication to the public inside cultural places, sound and music have so far been considered as minorities, compared to image and visualization aspects that are predominant. Less attention, less investment and very few infrastructures, less economic and human professional resources and less time are devoted to sound simulation, sound composition and sound design accompanying the visual narration in the field of cultural heritage and education.

However, sound is the first media able to stimulate our perception, anticipating the image itself in evoking an emotional state that preludes the development of a story or of an experience. Sound is an expression of identity, of forms of individual and social practices and aptitudes [2].

Therefore, the ambition of this contribution is:

• To discuss the fundamental importance of sound in cultural transmission and storytelling.
• To describe the kind of sound experiences commonly proposed to audiences of different audio-visual media, in various contexts of cultural fruition, with a particular reference to museums.
• To provide a basic understanding of sound devices and solutions in museums, their quality and differences, and to discuss how to best employ them, in which contexts and conditions, and how to make the acoustic experience fulfilling in cultural venues.

Therefore, this article is addressed to designers of audio-visual narratives for cultural heritage, museum directors and curators of digital installations in places designated for public use and education.

Traditionally, soundscapes are created according to different scientific, aesthetic and cognitive approaches, grammar and techniques in the various media.

Therefore, this article will examine the state-of-the-art and will make an excursus of the different approaches employed in sound design, opening perspectives for reflection on next developments, especially in the field of immersive virtual reality.

This contribution is composed by 6 sections. Section 1 consists of the introduction to the subject. The author’s scientific background and research domain are presented, the ambition and target of this contribution are declared, as well as the main problems the article tries to face, in relation to the state-of-the-art and the literature review. The structure of the article is also presented. The research domain of acoustics and archaeoacoustics is discussed, and conceptualization and definitions of soundscape are presented. Finally, the state-of-the-art of professional and consumer audio technologies that can be proposed in museums are introduced, and their characteristics are explained.

Section 2 deals with data and methodologies, and the focus is on sound experiences inside museums, their potentialities related to cognitive aspects and problems related to their management. Some possible solutions to improve the quality of the sound experience in museums are presented.

Best practices referring to accessibility of museums’ contents are then introduced, in particular with reference to the principles of Universal Design (UD), which also concern sound contents.

Section 3 discusses the different approaches to sound design in the various audio-visual media: documentary, cinema and virtual reality. In each of them, sound is generated and post-processed according to specific paradigms and codified rules, in order to involve the public/users in the story, passively or actively. In Section 4, some results related to virtual reality applications realized by the author and her team are presented, in relation to the methodological approach followed in sound design and in the creation of the whole user experience.

Section 5 is a discussion on further perspectives, investigating whether and under which conditions it is better to create evocative soundscapes, according to the principles of psychoacoustics, or scientific modeling of the acoustic space, based on its physical properties, especially in the context of immersive virtual reality. It discusses the techniques, advantages and limitations of the two choices and what specific objectives they can pursue.

Section 6 is dedicated to the conclusions, with particular reference to the future perspectives in virtual reality.

1.2. Domain, Literature Review and State-of-the-Art

Acoustics is a branch of physics that deals with the study of production, transmission and effects of sound. The discipline was theorized approximately 200 years ago, even though the interests towards sonic and acoustical phenomena accompanied the human communities since the beginning of time. The first scientific studies date back to the 6th century BC, thanks to the Greek philosophers, Pythagoras, who studied the special combination of musical sounds to produce pleasant effects, and Aristotle, who understood that the nature of sound consists in compressions and rarefactions phases. Additionally, Roman architects and engineers studied acoustical phenomena, for instance in 20 BC, Vitruvius wrote a treatise about the acoustic properties of theaters, discussing sound propagation, interference, echoes and reverberation effects, which he also reported in book V of his De Architectura [3].

Jumping ahead to recent time, several important studies have been carried out in the field of architectural acoustics. Archaeoacoustics has investigated the acoustic behavior of pre-historical and historical sites, and the acoustic simulation and auralization of past cultural landscapes, approaching the subjects from several perspectives.

In a conceptual essay, published in 2021, Hasan Baran Firat considers sensory/acoustic heritage within the bipolar classification of the tangible and intangible cultural heritage [4].

In this regard, there have been many ideological changes over the years about what is considered tangible/material and intangible/immaterial cultural heritage. The UNESCO’s World’s Heritage Convention in 1972 was the basic text referring to tangible heritage. The Burra Charter [5] was the new starting point for the definition of the immaterial culture that was deepened in the UNESCO Convention for the Safeguarding of the Intangible Cultural Heritage in 2003 [6]. The UNESCO Intangible Cultural Heritage List, established in 2008, includes a variety of traditions, significant sound, music, individual sounds of instruments or vocal (chanting) techniques and soundscapes, while their original practicing places are included in the World Heritage List as tangible.

Hasan Baran Firat approaches the topic from an interdisciplinary perspective, combing studies in cultural heritage, sensory semiotics and acoustics. He proposes that culturally significant and authentic sensory objects, and sounds in particular, should be classified as tangible, because they have existentially nothing different from the visual and touchable cultural objects. Ephemerality is one of the main attributes of auditory, olfactory or gustatory products and cultural objects, which cause them to be traditionally considered as abstract phenomena and associated with the intangible heritage. Their experience is considered a single and not repeatable event [7]. However, Firat affirms that auditory objects can be recorded, preserved and transmitted to future generations or even reconstructed, with their extrinsic features, as tangible assets.

Firat also analyzes the concept of sound heritage, sound sources and soundscapes. In addition to individual sound sources, the combination of different sounds generates the sound environment, i.e., the soundscape. He underlines that while a single sound source may not have any cultural uniqueness, the combination of several sound sources may create culturally distinct soundscapes, that are representative of very specific communities.

A series of studies have analyzed and demonstrated how soundscape is relevant for cultural and social identity of local communities [8]. It contributes to individual and collective behavior, helping to shape the relationships we form with others. For these reasons, UNESCO Resolution 39C/59 [9] addresses sound-related issues, namely those involving the sound environment, health, control of noise in nature, in built environment and in workplaces, sound recording, reproduction and conservation technology and the relation between image and sound.

When dealing with the sounds of the past, several disciplines such as archaeology, art history, anthropology, musicology, ethnomusicology, acoustics, architectural acoustics and archaeoacoustics have to be combined, as demonstrated by the studies reported in the Special Issue of the Acoustics journal “Historical Acoustics: Relationships between People and Sound over Time”, edited by Francesco Aletta and Jian Kang [10]. Here, a common methodological approach is the use of both acoustic measurements and acoustic simulations in the investigated case studies.

In his contribution, Rupert Till [11] points out that architectural acoustics has traditionally focused on issues related to industrial practice, such as concert hall design, traffic noise and health and safety issues. The result of such an approach is a number of assumptions and standard practices in acoustics methods that act to remove uncertainty and measurement of variation, aiming, in many cases, at an averaged result. Concert halls, for example, are designed to create a sonic diffused field, which means that a similar acoustic result is perceptible in all the points of the space, independently from the position of the sound emitter or receiver.

However, he points out that the application of standard procedures can be problematic when using acoustics to study archaeological sites with complex or irregular shapes. In archaeological sites, a large number of measurements are needed in order to comprehensively understand the acoustics of the space. For instance, Till has investigated how the acoustic behavior of archaeological sites used for ritual ceremonies changes during different stages of human culture: from Palaeolithic painted caves in Northern Spain, that are over 40,000 years old, through the monumental stone circle of Stonehenge in England, that is 4000 years old, to the public theater of Roman culture, in Paphos, Cyprus, which is 2000 years old. In such places, the experience of acoustics was not conceived to be uniform in the space, but, on the contrary, to vary considerably, depending on listening (receiver) position and source position and in relation to the symbolic meaning of ritual activities. The parameters that have been measured are: early decay time (EDT), clarity, speech transmission, consonant intelligibility and bass response. These data are compared to the same ones in modern concert halls. This study is very interesting because it demonstrates the potentiality of the discipline, and it explores how and why the acoustic behavior of such kind of spaces has been conceived, and how the acoustic metrics and effects change, depending on human social and cultural transformations over time.

In the same Special Issue, Sü Gül discusses about the acoustics environments of the Hagia Sophia and the Süleymaniye mosques, by carrying out measurements of room acoustics parameters and comparing results to those reported in the previous literature [12]. Alonso analyzes the acoustic evolution of the choirs of several Spanish cathedrals, by means of both on-site measurements and simulation models, to investigate the suitable intelligibility of sung text [13]. Jordan investigates variations in past and present soundscape at the Berlin Wall Memorial site, in Germany, applying both qualitative and quantitative methods, with the support of binaural recordings, psychoacoustic analysis and soundscape surveys based on standardized soundscape protocols [14].

Besides the academic interest in soundscape of the past, scientifically measured or simulated, in recent years, techniques and approaches for sound design have been borrowed from videogames and cinema, in order to evoke the virtual reconstructions of past sonic worlds.

For instance, Firat et al. have reconstructed the soundscape of the early 20th century in streets of Istanbul and Naples based on the psychoacoustic approach [15]. The written historical sources on vendor cries have been selected to virtually reconstruct the historical soundscape, which rely on physically based modeling, following audio-game design procedures (procedural audio) or sound design for films.

However, despite the richness of research and the novel perspectives opened up by such studies, the contributions dedicated to psychoacoustics, auditory narration, evocative soundscape accompanying audio-visual media and related technologies to be proposed in museums, are still few and marginal.

Studies on virtual museums are mostly focused on visualization aspects, in terms of contents, methodologies and technologies.

The present contribution relates more to the virtual museums domain, focusing on sound design of digital installations, and providing technical solutions aimed at preserving the quality of sound contents created by the composers, or resulting from acoustics measurements and simulations, and presented in museums. Therefore, it aims to be a resource useful to people and institutions working on cultural heritage valorization and narration, following an educational purpose.

Many digital creations dedicated to cultural heritage cannot be considered merely as a vehicle for information. They are new formalizations, sometimes with their own artistic value: they are new ‘works’, characterized by a combination of science, technology and art. Unfortunately, they are often penalized because the sound contents are not adequately perceptible.

1.3. Soundscape

The soundscape is the field of study of acoustic design. ISO 12913-1:2014 aims to enable a broad international consensus on the definition of ‘soundscape’. The standard distinguishes the perceptual construct (soundscape) from the physical phenomenon (acoustic environment), and clarifies that soundscape exists through human perception of the acoustic environment. In fact, while the acoustic environment has to be interpreted as “the sound at the receiver from all sound sources as modified by the environment”, the soundscape is defined as “the acoustic environment as perceived or experienced and/or understood by a person or people, in a specific context” (ISO 12913-1:2014, accessed on 26 November 2021).

The term “soundscape” was coined in the 1970s by the Canadian composer, writer and environmentalist Raymond Murray Schafer (Sarnia, 18 July 1933–Peterborough, 14 August 2021), who defined it as “any field of acoustic study […], a musical composition, a radio program or an environment” [16].

His theorization was developed as part of the World Soundscape Project, conducted in the 1970s by Schafer himself at the Simon Fraser University in Vancouver [17]. The aim was to promote a new ecology of sound, also in relation to the growing problems of noise pollution in urban environments. Schafer refers to the natural acoustic environment, consisting of the sounds of nature and animals, including humans [18]. Various elements converge in the soundscape according to Schafer: keynote sounds, sound signals and sound marks.

The keynote sound is the first fundamental note of the scale in tonal musical harmony. It indicates stasis. The World Soundscape Project defines it as a sound that may not always be consciously heard, but which “highlights the character of the ecosystem and the people who live there”. Keynote sounds are created by nature, for example: wind, water, forests, birds, insects and animals. In many urban areas, traffic has become a keynote sound.

Signals, instead, are foreground sounds that are consciously heard, for example: bells, alarm devices, whistles, sirens, horns, etc.

A sound mark is a sound characteristic of a specific area. According to Shafer [16], “once a sound mark has been identified, it deserves to be protected, because sound marks make the acoustic life of a community unique”.

The sound environment, conceptualized by Schafer, is therefore understood as an interaction of sound, space and time; from this concept, the notion of acoustic design derives.

The notion of acoustemology, formulated by Steven Feld in the early 2000s, stems from the same theoretical approach and deepens the idea of sound as a cultural ecosystem: “Soundscapes, just like topographical landscapes, are not only physical exteriors that surround human activity or are distant from it. Soundscapes are perceived and interpreted by human actors who, through them, create their own place in the world [….]. Acoustemology is a way of simultaneously investigating the place of sound and the sound of place. The idea is that sound is not only projected, or emitted, or heard in space. On the contrary: sound is a localized mode of existence. Sound is an instrument of knowledge.” [19].

Soundscapes are often used in musical performances [20,21].

A soundscape composition is an electroacoustic musical composition that creates a sound portrait of an acoustic environment. Already in the 1950s, John Cage introduced the concept of indeterminism into Western music, i.e., the negation of the notion of choice, based on Eastern philosophies. Music is nature, not an imitation of nature, and therefore indeterminate. The artist does not organize, does not dominate nature, but he/she listens to it. Man has a subordinate role, he is neither a performer nor creator of music, he is a liberator of sound [22].

Composers who have historically worked with the concept of soundscapes include Barry Truax, who was one of the members of the World Soundscape Project, and Luc Ferrari, who used environmental recordings as a distinctive feature of his musical language. David Monacchi is an eco-acoustic composer, best known for his multidisciplinary project “Fragments of Extinction” [23]. David Monacchi captured and preserves the unique sound marks of the last remaining areas of undisturbed primary equatorial rainforest, which are rapidly vanishing. Through his musical compositions and immersive sound installations, he tries to raise public awareness on the biodiversity crisis.

Walter Branchi goes even further in the ecological conception of music. His music does not have a classical structure, a self-completed ‘narrative’, but rather acts as a ‘frame’ for the complex environmental sound world [24,25]. It welcomes unexpected and random sounds from outside, with which it establishes a balanced relationship, also in terms of acoustic intensity. The music and the environment are indissolubly linked, becoming a global sound atmosphere, where there is no indication of a specific content to listen to. This idea of “atmosphere” takes its cue from real life and leads us to become one with the environment in which we live, to perceive it also with the body, and not only with the mind.

The American artist Bill Viola captures in his art all the vital manifestations that arise from the interaction between man and natural forces: from the cyclical nature of events, from corporeality and from metaphysical impulse. It is a large-scale vision of the human being and life, in relation to the natural environment and the forces of the cosmos. His installations are alterations of space and time, a sacred and eternal time in which the ephemeral human time is included. This idea is expressed especially through extreme slow motion in the representation of human actions and of phenomena that occur. Soundscape supports this duality. A low note (keynote sound) underlies the whole representation: it is the noise of cosmos, the sound of existence, which we do not consciously perceive, and which accompanies, like a drone, the specific signals of the scene which we consciously perceive. Video art and sound environments, electronic music performances and technology are linked together in a new form of living art [26].

1.4. Audio Technologies in Museums

Audio format and technologies that can be proposed in museums are: Stereo, Dolby Digital Surround, Mono/Directional sound and Binaural [27,28].

The stereophonic technique involves two sound information streams, recorded separately with two microphones and reproduced by two loudspeakers placed in two different positions in the listening room, generally equidistant and symmetrical with respect to the optimal listening position. It is based on the ability of human hearing to distinguish the spatial origin of sound sources (through the ears), propagation delays, decays and a set of perceptions that depend on the phase of the wave, the time of arrival, the sound level and the different harmonic components.

Stereophony is opposed to monophony, which involves a single sound information stream.

The advantage of stereophony, in a museum context, is its portability and scalability, and the possibility of using headphones or loudspeakers, these latter for a collective experience. However, it requires the listener to assume an ideal position at the apex of the triangle [29].

Dolby Digital Surround is a proprietary multi-channel audio standard, developed by the American company Dolby Laboratories in the 1990s, mainly for home theaters, used in many DVDs and Blu-Rays [30]. In this case, the audio streams are more than two, they can be 5 + 1 or 7 + 1. They are the result of post-processing of the sound that can originally be captured with various types of microphones and techniques. Each audio stream is transmitted by a dedicated loudspeaker placed at a specific point in the room. The user, placed in the center of the listening space, is surrounded by sounds, being able to fully appreciate their three-dimensionality [31]. 5.1 surround sound uses five full-bandwidth channels and one low-frequency effects channel (subwoofer) (Figure 1). 7.1 surround sound, introduced in 2010, is an extension of 5.1 that uses four surround zones: generally, two on the sides and two at the back.

Due to its potentiality to enable a three-dimensional perception of sound and a sense of acoustic fullness of the space, the Dolby Digital Surround system is now widely used in the videogame industry, cinema and home theater for an immersive and qualitatively satisfying listening experience. Within museums, Dolby Digital Surround is almost nonexistent. The main reasons are the poor attitude of the museum curators to use an invasive sound, especially along the exhibition routes, and the frequent difficulty to find a dedicated space, large enough to accommodate a group of people for multimedia experiences. Dolby Surround could certainly be effectively used as a room audio, to narrate the various exhibits, by placing speakers along the perimetral walls. It is not applicable to the use of headphones, for which binaural audio is used to achieve a similar effect.

Monophony, unlike the multi-channel systems mentioned above, was the first audio recording and reproduction technique implemented. It involves a single sound stream, reproduced by a single loudspeaker positioned in front of the listener. The listener is therefore unable to spatially locate where the different sounds come from [28]. For this reason, mono-audio is generally considered less realistic and non-immersive, more “flat”, functional to simple explanations rather than to aesthetic experiences. Therefore, very often, mono-audio is transmitted with poor quality systems, especially in museums. Its success is linked to the simplicity of the required equipment, the low listening volumes, the cheapness of the hardware and the simplicity of its maintenance.

Despite this, there are now very advanced monophonic systems. Of course, mono-audio does not take advantage of the spatiality provided by headphones, as each ear would have the same content. However, in the post-processing phase, a good mono-signal can be positioned at any point in a Dolby Surround system.

A stereo signal, on the other hand, can hardly be converted to mono with good results, unless special software is used. In fact, the phase opposition of the two original channels can easily result in the deterioration or loss of certain frequencies in the monophonic translation [29]. Some high-quality, state-of-the-art mono-loudspeakers incorporate software that can perfectly convert a stereo signal into mono-audio.

Directional audio is a monophonic audio using a very directional sound source. This source allows the sound to be directed precisely into a narrow, well-defined space, resulting in a sound area beyond which the audio is no longer audible. This condition allows the sound to reach a specific group of people, without disturbing everyone outside the listening area. Independent and different sound contexts can thus be created in the same environment. Technologies can be (1) high directionality sound panel, and (2) sound shower. They are generally equipped with systems able to attenuate environmental noise arriving from the space outside the listening area. These systems are quite often used in museums, especially the sound showers, of which very cheap models are available, for modest audio experiences, at low volume and narrow frequency band. They are more functional to transmit descriptive and prosaic audio contents, rather than to arouse emotions. High-quality technologies of this kind would be available, but with higher investments that museums usually do not consider as a priority.

Binaural audio (i.e., two-ear) is based, unlike all others, on a three-dimensional sound-recording technique, aiming to optimize headphone listening. The sound transmitted through headphones reproduces as faithfully as possible the acoustic perception of a listener located in the original environment in which the sound event was recorded, maintaining its directional characteristics at 360° [32]. The most sophisticated recording technique involves the use of a dummy head, built with dimensions and materials that faithfully reproduce the sound absorption of a real human head, and its function as a natural separator between the two auditory channels (right and left). The head reproduces the shape of the pinnae and the ear canals, inside which two high-fidelity microphones are placed (Figure 2). In this way, the microphones pick up the sound as close as possible to how a real listener would have perceived it. This technique is not quite appropriately translated into a stereo system.

Ambisonic microphones retain all spatial information about sound in its three dimensions. Four microphone capsules are used for recording.

Ambisonic microphones allow a different decoding of the signal depending on the number of loudspeakers used for playback, so the audio can be rendered for stereo systems, Dolby Surround, binaural, etc. [33]. Of course, the most faithful output is the binaural one, reproduced in headphones. This kind of technology is not used in museums. However, it could be of great impact in specific installations and in spaces created for the audio-visual immersive enjoyment of specific natural or urban environments.

2. Data and Methodology

2.1. Sound Experiences in Museums

One of the challenges for the cultural transmission today is to strengthen the interconnection between contents (real and digital), storytelling and audiences, in order to create new scenarios of the cultural experience.

In a museum, the main objective is not pure description, but the creation of a dynamic space of relations. What matters is the processes of interaction we undertake with the contents, the experience that links past and present. The feedback we get from the real and digital contexts modifies our behavior, and stimulates understanding and the process of attribution of meaning [34]. Virtual reconstructions facilitate the mental process of imagination [35], giving form to an otherwise abstract concept (a vanished ancient context cannot be perceived and experienced), making the object recognizable and contextualized. Sonic contents play a fundamental role in this process. The association of sound to images can better clarify their value by conveying a specific meaning. On the contrary, a sound can be associated to different situations that images can define. For example, the sound of a gunshot can be associated to the war or to a ritual event. Thus, sound is essential in storytelling and in meaning making. A multi-sensory approach can augment the role and the interaction of human beings in real and virtual environments through increased affordances.

Moreover, several surveys on user experience in museums that CNR has conducted over the last 10 years [36,37] reveal, unequivocally, that narration, considered as a coherent harmony of text, acting, music and soundscapes, colors, visual mood, camera movements and rhythm, is what, more than any other element, is able to arouse emotions in the visitors, unlike cold descriptions. Storytelling creates connections and relationships among objects, contexts, people and meanings, and in storytelling, the contribution of sound is as important as image [38]. Emotion generates motivation and thus translates into potential learning [39,40].

Sound communication is a powerful vehicle for knowledge transmission, and therefore hearing and listening, acoustic dimensions, play a central role in social and cultural representation, even in museums.

Sound is essential to transmit the cultural identity, to increase the cognitive and emotional involvement of visitors in virtual reconstructions of cultural contexts or historical events, or to imagine the life beyond an object.

Moreover, sound in museums can strengthen the collective dimension of the experience, because it conveys a sense of convergence of visitors towards contents, synchronizing their minds and feelings, creating a common vibration, a sense of cohesion and sharing of the meaning.

Of course, this condition also arises from the adequate quality of audio production and reproduction.

2.2. Problems in Audio Management in Museums

Still today, museums are conceived as places to see objects, not to listen to their stories. The traditional approach in the organization and presentation of collections is taxonomical and typological, aiming at the comparison of forms and their variations rather than at narration, which, on the contrary, ties together different types of contents [41]. Although the adoption of multimedia technologies is gaining ground in museums, viewing is enormously prevailing over listening.

Sound, in fact, is generally considered as a problem, something disturbing, especially if used along the visit path, in the presence of collections, whose enjoyment usually occurs silently. Often, images and virtual reconstructions are shown without any audio comment. If there is any, its volume is set very low by the museum’s staff, at the limit of being perceptible. Therefore, sensitivity for a good audio experience is very scant, and also investments in audio technologies are almost absent.

Traditional audio-guides, used through earphones, are the most accepted audio devices in museums. However, even in this case, audio has exclusively a descriptive function, with no attention for sound design able to provoke an emotional reaction.

Another recurring problem is the resonance of the museum space, which fosters certain frequencies and cuts off others, or causes rumble phenomena, producing unexpected bad effects: storytelling cannot be fully enjoyed, and the perception is compromised. Typically, this condition is due to the lack of accurate planning in the sound design of multimedia in relation to the real space. In fact, often, the location of multimedia in the museums’ rooms is not decided since the beginning, with the museums’ curators, but it is approximate or modified with respect to what was initially foreseen.

In many cases, museum curators and traditional designers autonomously decide how to set up an exhibition including real artifacts and digital contents. Multimedia devices, and audio systems in particular, are chosen without taking into account the instructions of the creators of the original installations.

As a result, the enjoyment of multimedia contents, and especially of sound contents, is compromised (Figure 3).

From the public’s side, the awareness and expectations regarding picture quality have become increasingly polished, in terms of aesthetic of rendering, resolution, range, color, camera movements and animation, thanks to the great and rapid evolution of digital technologies and communicative media (advertisement, cinema, television, games). On the contrary, people have progressively become accustomed to poor-quality audio reproduction: they listen to music through the headphones of their mobile phones or iPods, web streaming has forced the diffusion of compressed formats (although compression algorithms have improved over time) and television is often used with scant quality integrated loudspeakers. Online social interaction and communication using auditory live chat are very different from real-life scenarios: cues, tones of voice and audio frequencies are lost and, similarly, spatiality and directionality, as all the participants’ voices come from the same direction. The improvement of these systems is the goal of a recently financed European project, “SONICOM-Transforming auditory-based social interaction and communication in AR/VR”, that is expected to be developed within 2026 (FETPROACT-EIC-07-2020-FET Proactive: emerging paradigms and communities [42]).

In conclusion, a recurring tendency is undervaluation of the importance of soundscape in multimedia installations in museums, where only the comprehensibility of spoken contents is saved.

2.3. Possible Audio Solutions in Museums

The sonic experience in a museum has to be conceived carefully and considering many factors: the harmonic integration between real and virtual contents, the configuration of the spaces, the visitors’ way, the lighting, the noise of the environment, the possibility for the public to sit or stand during the interaction with contents, the balance between free interaction and guided experience, the duration of narrative units and their distribution along the visit path, and the communicative rhythm and the harmonic coherence of words, images and sounds [1].

As a public place of education and culture, the museum favors collective communication, sharing and social exchange. Thus, communicative formats, languages and technologies must be chosen to involve mainly groups of people. This is the case, for example, of a multimedia installation, interactive or not, projected on a large screen, accessible to several users at the same time.

In these cases, the preferred multi-user audio technologies are:

-

Stereophony with good-quality speakers.

-

5.1 or 7.1 Dolby Digital Surround system.

-

Monophony (not the preferred choice, but possible, using a good-quality speaker with a software encoder capable of converting the signal from stereo to mono).

-

Some monophonic sound showers of good quality, capable of preserving the entire frequency spectrum of the sounds, placed close to each other to ensure sound diffusion over a sufficiently wide area.

-

Sound panels with high directionality; in this case, one, or very few people, could have a good perception of the sound contents.

-

Good-quality Wi-Fi headphones, to be distributed to visitors entering the museum, to enjoy synchronized audio and visual contents. In this case, a binaural listening can be proposed, allowing very good perception. This solution, however, is certainly more expensive and requires assiduous control and maintenance of the devices. Moreover, it does not encourage dialogue and social exchange.

If the multimedia installation broadcasts audio without directional systems and is located along or near the visitor routes, it is desirable that, when no one is using the system, it enters a standby state, with attractive graphics but no audio content, to avoid bothering the visitors—a possible message can invite them to activate the content. The application should hopefully integrate a sensor and a special software function, which would warn the system that someone has entered the multimedia area. The volume of the application should then automatically set to the appropriate level, for good listening, before the content starts. This would avoid the manual intervention of the museum staff on the volume of the installation. Some technological solutions based on computer vision algorithms (e.g., infrared camera or normal webcam) would make it possible to detect the number of visitors who are accessing the installation at a given time, and to send input to the software to raise or lower the volume proportionally (the more visitors, the louder the volume should be). The effectiveness of such technologies also depends on the space available in the room and the environmental conditions (light–shadow contrast).

It is necessary to use an equalizer in the amplification system in order to obtain the best relationship between the sound content and the acoustics of the museum environment (determined by its size, shape and materials), limiting rumble effects or frequency cuts.

Alongside the social dimension, it is important that the museum also manages to convey moments of intimate and personal reflection, of profound contact with the artwork. Spaces must support the transition to this dimension of meditation. In this case, technologies that encourage a solipsistic, multi-sensory and embodied experience can be very effective. It is not necessary that they must be interactive, but immersion is certainly a necessary dimension.

The most appropriate single user audio technologies are:

-

Head-mounted display with binaural listening (associated to immersive VR).

-

Stereo speakers with the user positioned perfectly in the center of the triangle (associated to non-immersive vision, on screen).

-

All systems based on loudspeakers can also be used for the individual experience, but the space should be as isolated and closed as possible.

Multimedia and sound contents can therefore be used to reconfigure the museum map in terms of attractiveness of objects, rooms and routes; as creators of experiences, they influence the length of stays, the degree of collectivity of the experience and the level of interaction and social exchange [1].

The numerous user experience evaluations carried out by CNR on the impact of multimedia, holographic and virtual reality installations on museum visitors reveal a constant fact: the public explicitly states to appreciate audio contents, even if they are diffused in the environment for collective enjoyment. If the narration and the soundscape are interesting and engaging, the high-volume level is considered positively [43].

2.4. Solutions for Sound Accessibility

According to ISO 9241-11:2018 (i.18) [44], the term “accessibility” refers to the extent to which products, systems, services, environments and facilities can be used by people in a population with the widest range of user needs, characteristics and capabilities, to achieve an identified goal in an identified context of use.

In December 2019, a new version of the European guideline “Accessibility requirements for ICT products and services” (EN 301549) [45] was published to specify functional accessibility requirements applicable to ICT products and services, to be used in public procurement in Europe. This standard has been drawn up under the competence of the authority federated to UNI, UNINFO—“Information Technologies and their applications”. Most of the principles here established are addressed to improve the accessibility and usability of multimedia applications by blind and deaf people, or by visitors with limited abilities in vision and hearing, visitors in wheelchairs and persons with cognitive disabilities. However, they can improve the experience of all users, generating ease of use and well-being, in accordance with the Universal Design (UD) principles [46]. The proposed solutions are based on multisensory and multichannel approaches: the same content has to be delivered using more than one sensory solicitation: by touching, listening, watching, reading (subtitles) or using the International Sign Language (IS).

With regard to audio content, some recommendations are [47]:

-

Access an audio-description, useful for visually impaired persons, i.e., an audio describing the significant aspects of the visual content and which can be activated on a specific channel of the audio device, as an alternative to the main spoken storytelling. This audio-description must not be disturbed by other audio interferences and must be well-designed in terms of choice of significant contents.

-

Receive audio feedback to actions, to facilitate the experience: waiting, confirmation, contact, obstacle, etc.

-

Benefit from a magnetic induction amplifier in multimedia applications, or a transmitter with bluetooth connectivity: these can improve listening by users with cochlear implants or sound amplifiers. The magnetic induction amplifier eliminates ambient noise (https://www.centroacusticofirenze.com/induzione-magnetica, last access on 16 August 2021).

-

Sign language videos, to enable deaf visitors to understand the contents.

Each one of these recommendations is a complex matter, and it is not the main topic of this contribution. The purpose here is to strengthen the importance of an issue that should be more considered by designers and developers of audio contents, for extended shared enjoyment.

3. Sound Design in the Different Audio-Visual Media

As introduced, the creation of sonic environments and soundscapes follows different paradigms and methods in the different audio-visual media.

This section will discuss, at a basic level, how documentary, cinema, virtual reality and videogames have refined different codified rules, usually following artistic principles closer to psychoacoustics than to acoustic simulation based on scientific measures of the space.

3.1. Documentaries

The main goal of a documentary is to represent a reality as it is, without fiction (even if hybrid forms of docufilm exist). The story is real, and it is transmitted through the evidence of its true protagonists and historical sources. Differently from cinema, voices and sounds of the environment are recorded at the same time, as direct input from the reality. The place of sound and the sound of place are simultaneously represented.

Besides, in documentaries, sounds are not used to evoke something that cannot be seen: sound sources must be visible in the scene to communicate a message coherent with the reality, and external sound sources are therefore considered disturbing.

The level of recorded sounds can be adjusted in the postproduction phase of the documentary, to make the speech intelligible, if needed.

A new revolutionary perspective is represented by 360° cameras, associated to ambisonic microphones to record the real acoustic space. In this case, users can explore interactively a specific reality, living an immersive experience in it, rotating both the point of view and the listening point, thus evading the traditional editing approach.

3.2. Cinema

In cinema, sounds are created to compliment images. Above all, it is through the image that space is represented and understood. Editing rules of movies are mainly based on this axiom.

The relationships between sounds and space, directionality and intensity based on distance are not realistic. The ratio of voices to background noise is disproportionate.

As the fiction aims to evoke, suggest and incite imagination beyond what is explicitly seen, many sounds come from sources located outside the scene, and they contribute to create the emotion the director wants to arouse in the spectators. Soundscapes are thus created following psychoacoustics principles rather than realistic reproduction [48].

Therefore, soundscapes, artificial or natural, are created in postproduction. Natural sounds are recorded on the set, without the actors speaking. They are mixed after, according to the voice levels.

The Dolby Surround system strengthens an immersive audio perception but it is arbitrary and unrealistic, as the editor decides where to send a sound to solicit the user.

In dubbed television films (especially in Italian), the ratio of voices to background sounds is even more disproportionate. Voices are in the very foreground. This sound idea is not the director’s original one: soundscape processing is artificial for both artistic and technical reasons of internationalization.

3.3. Virtual Reality and 3D Videogames

Virtual reality (VR) is a 3D artificial (mainly digital) environment that should give a human being the impression of interacting in a real environment, perceiving it through his five senses. The user, included in the space, freely explores it from several points of view, analyzes it and interprets it, moment by moment, in real time. However, we are still far from multisensory. Great technological progress has been limited to visualization, partly because sight is our most developed sense. We can digitally simulate sounds, but the market has neglected the scientific simulation of a sonic environment, favoring approaches closer to psychoacoustics.

Figure 4 shows a screenshot from the Unity 3D game engine [49], in relation to audio parameters and controls (Figure 4).

An audio clip can be played by an audio source in the scene, as 2D or 3D or as a mixture (Spatial Blend). The priority of one audio source can be determined over the other ones that coexist in the scene. Sound can be controlled over distance through falloff curves, and the speed of the attenuation can be set: it can be influenced by reverb zones, individual filters can be applied, and the Doppler effect can be applied to the audio source for an even richer audio experience.

However, even if in VR and 3D game engines more attention is dedicated to spatiality of sound, cinematic paradigms still prevail. Even when using acoustic modeling software with ambisonic technology (such as Odeon Room Acoustics, OpenAL, FMOD), the rendering of the sound and its reflections in the space is calculated on the position of the emitter and the listener with a high degree of approximation.

In 3D interactive environments, we can use sounds as follows:

-

To confer realism to the space, mapping it with realistic sounds.

-

To strengthen the symbolic meaning of the space, mapping it with evoking sounds.

-

To attract the user towards specific objectives in the virtual space, even if the sound source is not visible in the frame (by virtue of interactive exploration).

-

To mark the space with meaningful sounds to suggest an itinerary.

-

To give feedback to the user’s actions, and facilitate experience.

-

For storytelling and dialogues with characters.

-

For audio descriptions addressed to visually impaired people, avoiding noise confusion and sound overlapping.

4. Results

The following results come from VR applications and 3D videogames the author realized in the last years, in collaboration with the Virtual Heritage Lab at CNR ISPC and with creative industries. They are mostly representative of the psychoacoustics approach.

For each of the applications mentioned below, user experience evaluations were carried out in the various museums where they were proposed, in order to verify their attractiveness, usability, comprehensibility of contents and educational impact. The sound component, as part of the storytelling, is usually appreciated as an essential element in the evocation of past landscapes and of the whole experience. Most visitors prefer spoken to written language, and often complain when the perception of sound is inadequate, due to poor use of devices. The articles mentioned as references for each project offer details in this regard (Figure 5).

4.1. The Rule Confirmation: Virtual Experience among the Characters of Giotto’s Work, 2010, in Assisi, Italy

Within the exhibition “Giotto’s colors”, opened to the public in 2010 in Assisi, the National Research Council realized an innovative project [50], creating a virtual environment from the scene “The Rule Confirmation”, painted by Giotto in the Upper Basilica of St. Francis of Assisi at the end of the XIII century. The scene has been represented in 3D, starting from an accurate study of Giotto’s space, and the models have been textured using the original artist’s painting. On this basis, a VR installation has been realized, allowing the users to virtually enter Giotto’s representation. The scene was brought to life in a real-scale projection: characters were animated and represented while performing the actions painted by the artist (Figure 6). Storytelling consisted in the vocal dialogue between Francis and Pope Innocenzo III, taken from the literary source of Legenda Major written by S. Bonaventura from Bagnoregio [51]. Sounds also have a key role to evoke an atmosphere: environmental noises and murmurings are contextualized in the three dimensions. Medieval Gregorian chant, polyphonic fragments, are not performed according to a philological approach, but they have been elaborated electronically, through granular synthesis algorithms.

Granular synthesis is a method of sound synthesis that works with elementary acoustic elements, called micro-sound or grains. They can be derived, for instance, from a melody or from any composition. The grains are between 1 and 50 milliseconds in duration and can be combined and played together at varying speed, phase and amplitude. The result is not a single tone, but a cloud of sounds that is manipulated differently from any other natural sound. The theory of granular synthesis stems from the research of Dennis Gabor (1900–1979), the Hungarian inventor of optical holography, while Curtis Roads (1951, Ohio) and Iannis Xenakis are credited as the first to implement a digital granular synthesis technique [52,53]. Finally, the Canadian composer Barry Truax was one of the first to implement a real-time system version of this technique [54].

Therefore, in this project, the soundscape consisted in a combination between ancient and contemporary styles and languages, aiming at a new interpretation of the meaning and creation of a suggestive virtual journey. Sounds sources and music were not distributed in the 3D space, but used as a simple soundtrack. The installation was enjoyed in a dedicated and secluded space, in stereo, with a properly high volume.

Visitors could interact within the virtual space and mix with the characters only by using body movements, without any device, in a simple and natural way. This was possible using last-generation computer vision techniques for motion capture, through an infrared camera. In this way, the space illustrated by Giotto becomes a place of experience open to multi-sensorial narration and participation.

Video demo: https://vimeo.com/223943953, (accessed on 26 November 2021).

4.2. Etruscanning: Virtual Exploration of the Regolini Galassi Tomb, 2013, at the Vatican Museums, Vatican City

In 2013, the famous Etruscan Regolini-Galassi tomb, located in the Sorbo necropolis in Cerveteri, has been virtually reconstructed by CNR ISP (ex ITABC) in the context of the Etruscanning European project. Such a reconstruction has been presented as a VR permanent installation in the Gregorian Etruscan section of the Vatican Museums. The grave goods exhibited in the Museum have been contextualized in the virtual model and narrated through the direct voices of the two buried persons, in a dramatized style (Figure 7). Additionally, in this case, visitors could explore, one at a time, the virtual environment, using only body gestures, through a motion capture sensor (Microsoft Kinect), which strengthened embodiment [55].

While one active user is interacting with the system, the other visitors can passively attend the experience, but easily alternate in the active role. Again, music and sounds were really fundamental to produce an emotional involvement. In this case, the “soundscape” evoked past and, at the same time, contemporary cultures. A SuperCollider was used as a generator of synthetic sounds (cymbals, little bells, bass drum) using granular synthesis, and “live” melodic instruments were also recorded: bass flute and alto flute, as a similar timbre is documented in the Etruscan world, especially associated with religious ceremonies. Sounds are combined with noises to evoke the meaning of each object: water, chariot wheels, a galloping horse, crying, sighs, fire, sword hitting a shield, etc. In this way, each object, when selected, reveals its story, through the voice of its owner, accompanied by the universe of sounds enhancing its symbolic value. All the fragments are related and connected by a similar style and a common drone. While freely exploring the space, when no story is told, soft background music can be heard. The installation was located in a dedicated room of the museum, and the audio was in stereo, using good-quality speakers; however, the acoustic of the environment was not completely satisfying because of reverberation that cut off some frequencies. Unfortunately, no equalization system was used. Video demo: https://vimeo.com/61736198 (accessed on 26 November 2021).

4.3. Virtual Museum of the Tiber Valley, 2014, in Rome, Italy

The project “Virtual Museum of the Tiber Valley”, designed and realized by ITABC-CNR, aims at providing an integrated platform for the enhancement and promotion of the medium Tiber Valley, north of Rome, an area of 60 km long and 40 km wide. The project is quite original for its multidisciplinary approach to the study of the landscape considered in its several cultural components: geological, natural, historical, archaeological, literary, evocative and symbolic [56].

One of the results of the project is a VR application characterized, again, by gesture-based interaction and by an artistic and evocative style, presented as a permanent installation in the Etruscan National Museum of Villa Giulia in Rome, in the same room where the collection is also on display. This installation consists of 4 scenarios, visualized on 3 aligned 65 inch screens, that allow to explore the Tiber through the eyes of: (1) a fish swimming in the river, among floating images and voices of historical memories, (2) a bird that flies over the landscape represented in evocative style, (3) the ancient characters living in the Roman city of Lucus Feroniae and (4) a freed slave that has been introduced in the famous ancient Roman Volusii’s villa.

The more game-oriented scenarios are the underwater and the aerial ones. In the underwater world, the user can swim in the current of the Tiber, like a fish, experiencing the memory of the river: he meets fluctuating images, iconographies, sounds, musical grains and spoken literary fragments from ancient and contemporary authors. He/she can swim freely, following these dynamic memories, or he/she can awake audio contents by touching fragments of painted walls collapsed in the river. Literary quotations are spoken by a multitude of voices. Sometimes they overlap, but intelligibility of voices is not always an issue, because the idea of an uninterrupted flow is a deeper concept.

In the aerial exploration, the user can fly, like a bird (using arms), over an evocative 3D representation of the middle Tiber valley landscape. The territorial representation is based on an accurate Digital Elevation Model (DEM), but the rendering is not photorealistic; on the contrary, it aims at stimulating the imagination of the visitor following a symbolic approach. 3D graphics resemble a videogame, and sounds have been composed starting from ethno-musicological studies, redeploying traditional popular sounds and folk songs documented in the Roman countryside in this area [57], cowbells, whistles of shepherds and peasant calls. On this basis, some voices chase each other in space, telling fragments of poetries and stories quoting ancient authors, regarding the civilizations of these lands (Figure 8). Audio is stereophonic and uses professional loudspeakers, diffused in the museum’s room, next to the collection.

Video demo: https://vimeo.com/129867454 (accessed on 26 November 2021).

In the context of the same project, the authors were asked to create a multimedia installation for a little museum located in the Archeological site of Lucus Feroniae, North of Rome. The goal was to show the virtual reconstruction of the Roman city as it could appear in Augustan and Trajanic ages, while visiting the archaeological remains. In the original virtual reconstruction presented in Villa Giulia, a storytelling was developed in the scenario, using actors, voices and music. On the contrary, the director of Lucus Feroniae museum did not want to include any sound because it was considered disturbing in the room. Therefore, the story was adapted in the style of a silent film from the 1920s: silent scenes with actors performing actions are alternated with text signs showing written dialogues. This was an opportunity to experiment with a new format, albeit caused by a serious limitation imposed by the museum (Figure 9).

Video demo: https://vimeo.com/173026603 (accessed on 26 November 2021).

4.4. Defend the Walls! 2015, Paestum, Italy

“Defend the walls!” is a videogame developed by EVOCA srl (Rome) in collaboration with CNR and Paestum Foundation, about weapons and tactics of siege and defense in the IV century BC in Magna Graecia [58].

It is composed by an introduction movie in 3D computer graphics, introducing the historical background, and a VR environment using gesture-based interaction, to be played one person at a time, but also in team, in the context of a collective experience.

The installation was destined to the ground floor of tower n. 27, located along the city walls of Paestum. The projection, in stereoscopy, takes place indoors, in a dark space, on a large screen directly facing the battlefield. Thus, it is configured as a large virtual window that brings the visitors directly in connection with the real environment of the site, allowing them to better understand the ancient space. Visitors are brought inside the action and live the scene of the siege, using the war machines (gastraphetes, oxybeles, tortoise, mobile tower, catapults and so on) and choosing the best defense strategies.

The visitor is asked to enter the game and play the role of a soldier fighting on the wall, defending the city of Paestum (Figure 10).

Professional voices of famous cinema actors have been used to obtain the best users’ emotional impact and engagement.

The “soundscape” is rendered in Dolby Digital Surround 5.1, including the noise of weapons, voices, clamors and rumblings.

The player moves exactly on the virtual line between the city (behind him), which he/she must defend, and the external battlefield from which the attack comes (in the front); thus, the soundscape has been designed to create frontal and rear sounds, producing an immersive impact.

Video demo: https://vimeo.com/128983100.

4.5. The Box of Stories: Holographic Showcase for Museums, 2017–2019, Museums in Five European Capitals

The reconstruction of stories, sensory dimensions, daily practices, actions and symbolisms, that are “beyond” the physical objects, can be a powerful vehicle to drive museum visitors into the middle of a lively and emotional experience. Past history becomes alive—part of our present. This was the goal of the holographic showcase conceived and realized by CNR ISPC (ex ITABC) in 2017–2019. It was presented in the context of an itinerant exhibition organized within the European project, Connecting Early Medieval European Collections (CEMEC), that involved museums in Budapest, Amsterdam, Athens, Bonn and Bruxelles [59]. It is a mixed reality installation: the real artifact exhibited inside is “enriched” by virtual events happening on and around it, to suggest virtual restoration, improved perception and contextualization. The style of the story is not descriptive, but dramatized, consistent with the magical effect of the hologram, as an illusion of reality [60].

The holographic showcase was used to include the Kunágota sword, a weapon belonging to an Avar warrior chief in the 7th century A.D. and currently preserved in the Hungarian National Museum in Budapest. The evocation and dramatization are brought inside the showcase, as a holographic projection, transmitted through a mix of several voices and whisperings, episodes and personages belonging to the object’s past life. It is a first-person drama. What is represented here is a Byzantine casket in its original context, its purloining during the battle, its dismantling and the adaptation of the golden pieces to the sword, once its new owner, the Avar chief, died. The viewer cannot see personages, but can look at their hands and listen to their voices, during the various actions (Figure 11). The showcase fills with virtual dust, raised by the hooves of horses in battle, or is sprayed with blood splashed by swords.

Soundscape reinforces the rhetoric of the different moments of the drama. It has been realized in stereo. Unfortunately, it was played with different audio devices in the venues hosting the travelling exhibition, with some criticalities.

Video demo: https://vimeo.com/236305120, https://vimeo.com/221057009 (accessed on 26 November 2021).

4.6. Sonic Experience While Walking across an Archaeological Context: The Case of S. Anna Site in Teramo, 2010, Teramo, Italy

In the urban archaeological site of S. Anna, in the city of Teramo, located in the Abruzzo region in middle Italy, the ancient Cathedral of Santa Maria Aprutiensis was built, perhaps in the VII century, on the remains of an important Roman domus of the I century BC. Today, few remains of the two structures are visible because the site was destroyed in 1156 by the Normans. In 2009–2010, CNR ITABC realized a valorization project consisting in virtual reconstructions, videogames, multimedia and sound mapping of the archaeological space [61]. On the archaeological site, two large displays with visual contents (without audio) were installed, showing virtual reconstructions of the past contexts.

In order to suggest the ancient atmospheres and the functions of the different structures on the excavations, the real space of the site has been enriched by sound. Moving across the site, from one side to another, people can perceive different emerging audios: Gregorian chants in the apse of the ancient Basilica, noise of steps and bells near the entrance of the Basilica and sounds of birds, wind and water in the peristilium of the Roman domus (Figure 12).

Some of these symbolic fragments of sounds have been elaborated using the granular synthesis in SuperCollider. Thus, they are still intelligible but transformed in the light of the contemporary digital era, bridging past and present.

Sound diffusion was guaranteed by 8 DVD, each one connected to a mono-speaker, distributed through the archaeological space.

The following table summarizes the results of each project. See attachment.

5. Discussion: Evocation or Scientific Modeling of Sound?

Architecture is not only limited to shape the space, but also to sound modeling. Consider a church, a Greek theater or a garden: space and sound are indissolubly linked in the conveyance of a message, in the solicitation of an emotional condition in the participants/attendants [11]. Space and sound are parts of the same rituality, as discussed in Section 1.2.

This interconnection is bidirectional: the form and the function of the space contribute to create sonorities, or encourage the composer to create sonorities, rhythms or rhetorical formulas. On the contrary, a sonic or musical form conditions the architectural design of an environment, its volumes and the materials, in order to optimize listening and thus the aesthetic or spiritual participation of the audience.

Consequently, every architectural virtual reconstruction without a soundscape is incomplete. Next-generation virtual museums should consider such an important aspect with more awareness, as well as potentialities and results coming from acoustics studies.

However, under which conditions is the scientific modeling of an acoustic space preferable to the more common paradigm of psychoacoustics?

How can philology and creativity relate?

Which methods can be followed in the scientific modeling of an acoustic space, in a virtual reality environment?

Research Perspectives for the Scientific Modeling of an Acoustic Space in VR Applications

The objective of scientific acoustic simulation within a virtual reconstruction is to offer the users a credible and meaningful cultural experience: a journey through the simulated acoustic space of a historical place or event (archaeoacoustics) [62].

The behavior of sound derives from:

• The rendering of the emitting sound and its reverberations in relation to the volume and materials of the surrounding space.
• The location of the sound emitter.
• The location of the receiver (listening point), as it moves through space.

Therefore, the resulting behavior is dynamic. This is a very onerous calculation for today’s real-time sound engines.

How can this problem be solved technologically?

A solution could be to create a grid of pre-rendered sound points (obtained by software such as Odeon Room Acoustics, Ease, OpenAL, FMOD) [63] (Figure 13), that is superimposed on the spatial grid. At each point, the sound behavior is calculated and returned in binaural or ambisonic format. The sound space between two points could be calculated by interpolation. Between these points, the movement of the user can be continuous and free (this is the most difficult option), or set on predefined trajectories, or discontinuous (jumping by click and go).

Advanced experimentation must still be attempted in this regard, and more approaches could be tested.

Such an interesting area of research covers technical issues (generation, reproduction and perception of sound during users’ movement in real time), and perceptual, motivational and cognitive aspects. At the methodological level, the research may involve the following tasks:

(1)

Analysis of the issues to be addressed and definition of a detailed methodology.

(2)

3D modeling of the environment to be implemented in RV: measuring its acoustic behavior in terms of decay time, reverberation, clarity, frequency response and so on.

(3)

Recording sounds in an echo-free room, which minimizes the reflection of signals on the walls.

(4)

Positioning of sound sources in the reconstructed virtual space.

(5)

Rendering of sounds in relation to the acoustic behavior of the environment, on sound grid points.

(6)

In case of existing sonic environments, spatial acquisition and sound modeling will be performed differently: using a 360° camera and ambisonic microphones to record 360° spatialized sounds.

(7)

Data and software implementation in the VR application.

(8)

Creation and implementation of other non-spatialized sound contents, such as audio-descriptions, prompts and feedback actions.

(9)

Design of the user experience and interaction.

(10)

Final implementation of the VR prototype.

(11)

Quantitative and qualitative evaluations of the user experience, according to a codified method. This is useful to understand the perceptive and emotional reaction, the usability of the interfaces, the effectiveness of the contents and the optimal duration of the experience.

However, who could really derive benefit from such a development, aiming at the scientific modeling of an acoustic space, in VR applications?

Surely users of immersive VR, using head-mounted displays (HMD) and professional earphones, would greatly benefit from such a sound design.

In this case, they could live a reliable experience inside a virtually reconstructed environment, having the possibility to better understand how a specific space worked in relation to its acoustic properties and its functions.

Similarly, people who want to live an immersive sensory and perceptual experience in existing but remote ecosystems could enormously enjoy such a kind of implementation: for example, the sonic habitat of a forest (as in the film “Dusk Chorus” https://www.fragmentsofextinction.org/dusk-chorus-film/ accessed on 26 November 2021) [23], or painted rock-cut churches, resounding with religious chants.

Most probably, also visually impaired people could benefit from realistic acoustic space modeling, to improve their sensory orientation in VR environments. This would be an interesting and original study because until now, experimentation has focused mainly on tangible interfaces to improve accessibility [64].

Technologies needed to create these kinds of experiences pertain to different tasks: sound recording, editing, rendering, implementing and reproducing. Professional earphones are required for single user access to content; on the contrary, secluded audio-visual theaters for deep listening are specifically designed for multiuser access. An interesting infrastructure is the “sonosfera”, an eco-acoustic theater that was installed in January 2020 at the Civic Museum of Pesaro town (Italy).

The audience is seated in an amphitheater-like double cavea, built to be acoustically perfect, isolated from the outside and completely sound-absorbent on the inside. The sound comes from 45 loudspeakers positioned in the entire hemispherical surface around the audience and under the cavea, which combine to create a spherical sound field in the center that is very close to reality. Sound spectrograms and audio-visual content are displayed in a 360° high-definition multi-projection. The resulting experience is a true immersion in a sound and visual landscape (Figure 14) [65].

6. Conclusions

Sound is an integral part of the human experience, of the life of a community and of narration. Therefore, it is essential in psycho-physiological aspects of the individual, as well as in collective cultural identity representation, transmission and understanding.

However, since the beginning of theoretical studies, concrete developments and experimentations in the field of digital museology and virtual museums, minor attention and poor investments have been addressed in sound design, soundscape and music composition, in comparison to image.

In general, museums still have a problematic relationship with sound, whereby they consider it disturbing and difficult to manage in the visiting experience. They are configured as silent environments where people can look at objects and read captions rather than listen to their stories.

Reading and listening are very different processes, with a different impact on education. Reading is connected to the symbolic-reconstructive process of learning (mentally difficult and less immediate—contents are less easy to memorize in the long term). On the contrary, listening is connected to the sensory-motor process of learning (contents are easy and immediate to understand, and easier to memorize) [41,66]. Through hearing, we perceive, interact with and understand the world around us, both real and virtual.

To contextualize an object means to represent the meaningful moments of its construction, life, use and attribution of meaning, which also requires sound contents. The creation of a soundscape that lives beyond the physical dimension of a museum object establishes a connection between the meaning of the object and the perceptive and cognitive dimension of the user, between the past world and the present time, as it occurred for instance with the holographic showcase realized in the context of the CEMEC project, described in Section 4.5.

The psychoacoustic approach in sound design is generally used in combination with narration, to direct the users’ emotional and cognitive reaction according to the director’s intentions. In fact, as seen in Section 3, traditional narrative media such as documentaries, cinema and videogames make use of artificial sounds, or they use sound recorded from reality but in different steps, edited and filtered in the studio, selecting or excluding specific sound sources.

Today, most parts of virtual reality applications dedicated to cultural heritage, among which are those described in Section 4 developed by the author’s team at CNR, adopt the psychoacoustic approach, being strongly focused on storytelling.

When the kind of experience is narrative, or fictional, scientific 3D modeling of an acoustic space could be disturbing for the selection and comprehension of useful contents. Our brain, in fact, isolates the sounds that are of interest for us, selecting them among the myriad of sensory stimuli reaching our ears, that are excluded because they are not useful and confusing.

What emerges from many projects and studies is that soundscape, associated to images, is revealed to be essential in the user’s sense of embodiment in the virtual space, as in narration. Several data have been collected through observation, quantitative and qualitative analyses, related to the reaction and the feedback of the public in the presence of virtual and multimedia applications in museums (reported, for each project of Section 4, in the bibliography).

They clearly show that a good sound design makes the experience of a virtual context much more attractive and exciting—it catches the users’ attention for a longer time, as it immediately involves and suggests a rhetoric, a rhythm, an emotional condition and a shared feeling, that silent images alone cannot establish.

Scientific modeling of an acoustic space, instead of psychoacoustic approach, would be an interesting challenge in museums in order to make simulations and involve the public in realistic, multisensory and immersive experiences in ancient places where acoustics was a fundamental component to express their functions, as demonstrated by the archaeoacoustic projects mentioned in Section 1.2.

However, it would require the use of specific technological devices, such as head-mounted displays equipped with good-quality headphones for ambisonic listening (for single user experiences), or sonic infrastructures, such as the sonosphere, to share the sonic experience with a group of people.

Virtual reality seems to be the most promising media today for such experimentations and new experiences. Virtual and augmented reality present many challenges in immersive audio simulation and tailored audio solutions in the perspective to come.

Unfortunately, until now, visualization has attracted the majority of economic resources and research objectives: very rarely do laboratories include composers and sound designers, or research experts in audio technologies.

There is no cultural strategy or business model in this field. Even technological infrastructures dedicated to sound recording and listening, that could be shared by research laboratories, creative industries and museums, are almost nonexistent, in comparison with the several dedicated to images. This is also the case of the European Research Infrastructure for Heritage Science (E-RHIS) [67].

The videogame industry is probably the area in which sound composition and design of the acoustic space receives the greatest investment. Gamers pay much more attention to equipping themselves with good listening technology, because the videogame industry has traditionally attributed great importance to audio contents in conveying game strategies, strengthening users’ engagement

In conclusion, simulating rich sonic environments is of vital importance to deliver an immersive virtual experience in cultural heritage transmission and communication. As discussed in Section 2, various strategies can be adopted both in sound design and in technological implementation inside museums, to make sound a great added value for the understanding of cultural contexts, both in single- and in multi-user experiences. Surely, a stronger cooperation among museum curators, virtual heritage experts, creative industries, musicians and sound engineers could bring a great benefit to the cultural heritage sector.