Time Travel to the Past of Bosnia and Herzegovina through Virtual and Augmented Reality

Abstract

Bosnia and Herzegovina (BH) has a very picturesque past. Founded in 11th century, it has always been a crossroads of faiths and civilizations. Extended Reality (XR) technologies can finally take us to time travel into this history, enable us to experience past events and meet historical characters. In this paper, we overview the latest applications we developed that use Virtual Reality (VR) video, Virtual and Augmented Reality (AR) for interactive digital storytelling about BH history. “Nine dissidents” is the first BH VR documentary, tackling a still tricky subject of dissidents in the Socialist Yugoslavia, artists and writers falsely accused, persecuted and still forbidden. “Virtual Museum of Old Crafts” aims to present and preserve crafts intangible heritage through Virtual Reality. “Battle on Neretva VR” is recreating a famous WWII battle offering the users to experience it and meet comrade Tito, the commander of the Yugoslav Liberation Army. “Sarajevo 5D” shows the cultural monuments from Sarajevo that do not exist anymore in physical form using Augmented Reality. Through user experience studies, we measure the user immersion and edutainment of these applications and show the potential of XR for the presentation and preservation of cultural heritage.

1. Introduction

The first mention of Bosnia dates from around 950 A.D., when Byzantine Emperor Constantine VII Flavius Porphyrogenitus mentioned a country called “Bosona” in chapter 32 of his article “De administrando imperio” [1]. The Bosnian medieval kingdom was established in the 11th century and dissolved when the Ottomans conquered the Balkans in the 15th century. In the 19th century, Bosnia and Herzegovina (BH) was occupied by the Austrian Hungarian Monarchy, and after WWI it became a part of the Kingdom of Serbs, Croats and Slovenians. The first Yugoslavia was a monarchy established in 1929 and after WWII it became the Socialist Federative Republic of Yugoslavia, where Bosnia and Herzegovina was one of the six Republics. After the referendum in 1992, Bosnia and Herzegovina became an independent state again and was recognized by the United Nations.

During its turbulent history, traces of different cultures, nations and religions have remained in Bosnia and Herzegovina. BH cultural monuments date from Neolithic times until the recent war in the 1990s. Despite all the conflicts and dominions, life in this country has always been based on mutual respect, so that cultural diversity has survived until today.

Digital technologies have brought many possibilities to preserve and present this treasury of cultural heritage. Using Virtual (VR) and Augmented Reality (AR), cultural monuments are being reconstructed in the form of 3D models and these 3D models are now inhabited by historical characters who tell stories about events and life in the past. Our research within the Sarajevo Graphics Group has been taking internet users and cultural monument visitors to time travel into Bosnian history over the last 15 years. In this paper we present an overview of our recent projects which exploit the potential of VR and AR technologies in enabling the users to experience the past. The high immersivity of VR headsets easily transfers the users into a different place and time, but they need to be offered content that can use the potential of this technology and recreate life in virtual environments. VR video overcomes classical video as a medium and breaks the rules of film language and grammar. The viewer is now placed in the middle of the scene and can freely choose his/her insight, so shot composition rules do not apply. The new directing methods need to enable following the plot while being free to turn around and look in different directions. That means that the user’s attention should be captured in new ways.

Our research group has been working on applications of digital technologies to cultural heritage preservation and presentation for 15 years. The research problem we defined is to familiarize museum visitors and Internet users with Bosnian cultural heritage through Virtual and Augmented Reality, combined with interactive digital storytelling. There are many high quality virtual reconstructions of cultural monuments that lack the feeling of life in the past [2]. The aim of our methodology is to improve the edutainment value of digital content for museum exhibitions and digital cultural heritage applications, introducing actors playing historical characters as narrators and recreating life in virtually reconstructed cultural monuments.

Our experience with the use of these technologies in cultural heritage presentation will be shown through descriptions of four projects. Firstly, we will present the VR documentary “Nine dissidents”, which deals with the problem of intellectuals persecuted in the socialist Yugoslavia for their dissent with the regime. As this is the first BH VR film, we will describe what we learned during the production and what elements of the new film language grammar we defined. Then we will present the Old Crafts Virtual Museum in order to show how the intangible heritage of crafts in Old town Sarajevo can be preserved and presented using Virtual Reality. The Battle on Neretva VR combines historical information about one of the most important battles in WWII with a gameplay in order to enable the museum visitor to complete the visit experiencing the battle as a participant. Finally, we will describe the production of the first Augmented Reality guide through Sarajevo—called Sarajevo 5D—and show how AR can bring back the cultural monuments to places where today something else exists.

The main contribution of this paper is to show how VR and AR technologies can be combined with storytelling. As described in detail in [3], the use of real actors conveys emotion to the audience and enhances their immersion in the past. The hyper storytelling methodology [4] brings historical information closer to the users while obtaining a high level of edutainment. We believe that this combination could be used also in other areas of life, such as education and journalism. The workflows described in this paper can be reused in various kinds of applications.

2. Related Work

2.1. Virtual Reality and 360 Video

Virtual Reality, even though it is not a new technology, was not available to a broad audience until recent years. Additionally, Virtual Reality was usually a computer generated space in which the user would be placed. Technological advancements made filming of 360 video possible, and opened a whole new area of video storytelling. However, the main tool for any video storytelling was shot framing which is now completely gone since everything is now in frame, but the basic rules of cinematography should still apply in the same way.

These were the first things we had to tackle technically, through the same approach we can find in [5]. Even though basic rules of cinematography do apply, we had to consider the fact that no lights, light stands, microphones and similar things can be placed anywhere on the location of shooting. All of that has to be a part of the actual scene. Another issue we had to consider was the distance of the action from the camera. In conventional video storytelling, framing and cuts in editing allow us to change our distance from the action happening without losing the attention of the viewer. However, in 360 video things change, especially as the video is viewed with a VR headset. Objects that are very close to the viewer can cause uneasiness and if they get too far away, the viewer loses interest. Alia Sheikh, in [6] did a study with 26 participants, and the results show that viewers preferred the action at 3 m distance from the camera. Our own tests showed that action beyond 3 m is not so appealing to the viewer, but we also observed that action happening close to the camera can add to the emotions the viewer is experiencing. It can contribute to experiencing a deep connection to the video. Therefore, we decided to film in a smaller space for our first VR experimental film “Nine dissidents”. Camera placement was very close to main actors, who were sitting in the improvised court room.

Other 360 video guidelines considered the BBC, Jaunt and Oculus guidelines [7], and these also proved optimal in our test, such as preferred camera height, shot length, and so forth. Both directors we worked with pointed out that directing a VR video has more similarities with theatre directing than film directing, but also some uniqueness of its own. Other researchers also explore these similarities between theatre and VR noting that theatre directors use space in a complex way for narrative purposes, which can also be used in VR directing [8].

2.2. Augmented Reality

Augmented reality has been widely used in cultural heritage presentation in past years. This topic has recently received much research attention, with various forms and topics presented. There is a rich body of work on augmented reality in general [9], and usage in cultural heritage in particular. This usage varies from history teaching and tourism [10], cultural heritage sites enrichment [9,11] and an increasing sense of place for heritage places [12], to remote cultural heritage site presentation [13]. In [14], the authors describe the possibility of using augmented reality as a medium for teaching history and tourism.

One of the first works in this field proposed a personalized electronic guide and tour assistant for cultural site visitors [15] which provides augmented reality reconstructions of ancient ruins. A similar approach is used in our project Sarajevo 5D, where we created 3D models of historical objects that do not exist anymore, or at least not in the same condition as before. Besides this, our work aims to make use of Augmented Reality content to attract users to come to the site, without limiting them to using the application on site only. A similar approach has already been used and examined [16,17,18] and there are multiple confirmations of the value of Augmented Reality applications in heritage presentation [19,20,21].

As stated in [22], results from previous research are highly encouraging for future studies and projects regarding Augmented Reality in cultural heritage presentation. They also show that there is a lack of empirical research performed within the area, which serves as a motivational factor for further research. In the light of this, our work aims to present one step forward to wider use and engagement of Augmented Reality technology in this field, combining our interactive digital storytelling methods with AR content.

2.3. Web Technologies

Web technologies are an important tool for presenting the story to a wider audience. Using web technologies for presenting cultural heritage brings with it a problem of incorporating multimedia content for multiple web browsers and different devices. Presenting 3D content can be problematic for some browsers or devices. Some have no support for 3D libraries, and some support different technologies. The popularization of WebGL was an important step for making these types of applications possible.

There are a number of projects using web technologies for presenting cultural heritage. In [23] the authors describe the application of web technologies in the presentation of the Iberian Art Museum. They made a web based solution where 3D models are stored in a MySQL database. Models in the scene are dynamically loaded using Ajax. For displaying the models, X3D is used. It is a standardised solution but does not offer a large number of reusable components like Unity.

The authors of [24] present the 3D content management system DynaMus. The system uses Google and Europeana web services to generate 3D objects. The visitor can select some objects and click on them opening a 3D view with additional text. A similar functionality is described in [25] for artefacts found on an archaeological site. The authors of [24] discuss the possibility of using a game engine. They developed their system so it is easy for non-technical people to publish 3D content.

Another project that uses WebGL for presenting cultural heritage is described in [26]. In this paper, reconstructions of a shipwreck and items found on the site are presented. The authors combine different multimedia content to guide users through the shipwreck. The system is made as a WebGL-based 3D virtual roaming application.

The following papers introduce projects of presenting cultural heritage in Bosnia. The use of WebGL is described in [25]. This paper presents the virtual reconstruction of the White bastion in different periods. Virtual reconstructions are made in Unity and are exported as an optimized WebGL instance that is embedded in the web page. Before viewing the reconstruction, the user can view video stories that are related to the period he/she wants to observe. In the 3D virtual scene, the user can see reconstructions of artefacts from a specific time period, and for each object view a 3D model of the scanned artefact and its reconstruction.

In [27] a presentation of hyperlinked video stories is described. This application was made in Unity with Unity Web Player export. This type of web presentation is no longer supported by Unity or major web browsers. Even at the time when the application was made, support for Unity Web Player was limited. In all future projects, WebGL was used and its support is getting better every year.

Multiple 3D objects and high-resolution textures can be demanding for any internet connection. The authors of [28] discuss possibilities and approaches to minimizing the size of 3D objects on the web. They use compression and decompression algorithms to allow 3D data streaming in web browsers. This approach is good in terms of network usage but it can be straining on the CPU. The authors of web-based cultural heritage applications that use 3D objects and scenes must always balance the quality of objects and textures and their size. They need to factor in devices that most of their users have, so that the web solution is available for a wider audience but also to conserve the quality so the content is informative and interesting. Using compression algorithms is one way of having both at the same time.

3. Case Studies

3.1. Nine Dissidents

3.1.1. Motivation

There are still a lot of conflicting opinions on the character of the socialist regime in Yugoslavia. A number of people nostalgically claim that they were better times, with no ethnical tensions, times of “brotherhood and unity”, while others expose the facts about oppression and dissent, still carefully hidden from the general public. Within the COST Action CA16213 New Exploratory Phase in Research on East European Cultures of Dissent—NEP4DISSENT, we explored the topic of banned intellectuals. After the establishment of the socialist regime all writers, poets and other intellectuals who did not join the Communist Party were accused of collaboration with fascists and were prosecuted in trials mounted by the regime. Found guilty, they served prison sentences and were forever erased from bibliographies and histories. The movie “Nine dissidents” was an attempt at the rehabilitation of those people and drawing public attention to this issue [29].

3.1.2. Scenario

The movie plot is happening in an improvised courtroom where two solicitors, one representing the dissidents and other the regime, are fighting in front of the judge for rehabilitation. The courtroom is full of communist symbols and the judge and the audience are very communist. During this satirical story, the solicitor for the regime and the audience are singing communist songs and dancing, as well as shouting the slogans from socialist times, while the solicitor for the dissidents tries to present his case. At the end the regime wins, showing that there is no hope for the dissidents. As the movie is recorded in VR video, and the budget was very limited, the scenario is focused only on one space, the courtroom.

3.1.3. Workflow

In every digital cultural heritage application, we start with the preproduction phase. This phase is very important as it includes the planning of all the elements needed for production and postproduction. Whatever is not considered in this phase might later cause a loss of time and budget. At the beginning, the producer and director agree on the concept of the application. In this case it is a VR video presenting the selected topic. According to the available budget they plan the resources, such as the number of actors, number of filming days and filming locations. The expert historians provide information on the topic for the scenario writer. The writer creates the scenario based on the prepared concept.

When the filming location is selected and the actors are familiar with the script, we begin the production. For this movie we had a very limited budget, so all filming had to be done in one day. We used only one professional actor and local amateurs. Filming in VR is particular as no artificial lights could be visible, so we used the daylight coming through the windows. The scenes were recorded in an abandoned building room, without cuts, while the crew was outside in the hallway, observing the preview from VR camera on a tablet. Sound recording had to be performed with no visible microphones. In the production phase the graphics design of the movie opening was created. This was also a challenge as the motion graphics had to be in 360 degrees and the viewer must not notice the connection between two ends of wrapped image.

We used a Garmin VIRB 360 5.7 K camera. The improvised courtroom had a huge window on one wall, and we used it as our main and only source of light. Diffusion filters were placed on it from outside to avoid sharp shadows and get even light throughout the room. Tackling the sound, we decided to go with the lavalier microphones and used four Sennheiser G3 models and Tascam DR70-d as the sound recorder. Since we did not have more microphones at our disposal, we placed two of them on main actors, and the other two in the middle of groups in the scene, so we could use it for different characters in post-production. The results were very satisfying.

All post-production works were done in Adobe Premiere and Adobe After Effects since they now support 360 videos natively. Additionally, our sound engineer processed sound to an ambisonics format which gave each sound a position in the room. When the users turn, the sounds come from different directions.

The postproduction included sound postproduction, color correction and editing. The sound was recorded in mono and had to be distributed in 3D space, so the user has the right perception of each sound direction. It was mixed with the original music composed in the production phase. Figure 1 shows the elements of the movie creation workflow.

Another challenge was to organize a premiere of the movie for the audience while having only one VR headset. We solved that problem by recording a gameplay video consisting of screen recordings of the movie and HD video of a person watching the movie using headset and having a big screen with headset preview behind (Figure 2). We played that movie to the audience and offered them to try individually on the headset afterwards.

3.2. Old Crafts Virtual Museum

3.2.1. Motivation

Intangible heritage seems even more difficult to preserve than the tangible. Customs, legends and crafts are slowly disappearing from public memory. After the successful Mostar Bridge Diving VR project [30] where we digitized the 300 year old tradition of jumping from the Old Bridge in Mostar, Bosnia and Herzegovina, we decided to use the VR technology for the presentation and preservation of crafts present in Old Town Sarajevo—Baščaršija since the 15th century. Throughout history their number has been growing and in 1898 57 different crafts and 31 guilds were recorded in the census of Bosnian Sanjak [31]. Unfortunately, today many of them are not performed any more. There is only a small number of families who still preserve the craftsmanship in their family tradition. The Old Crafts Virtual Museum has the goal of drawing the attention of city authorities and public to this issue and at least in Virtual Reality preserve the selected crafts.

3.2.2. Application Structure

According to the recommendation from the Museum of Sarajevo curators, we selected three crafts where nobody will inherit the skill after the last craftsmen. These are četkar (the brush maker), kazaz (the tailor decorator) and bozadžija (the maker of a drink called boza). The Virtual Reality application has three parts:

• Stories about crafts told by the actor dressed as each of three craftsmen;
• Stories about crafts told by original craftsmen in their workshops, while showing their work processes;
• Virtual Reality videos of Baščaršija with superimposed names of crafts recorded in three censuses through history and explanations of their meanings.

The application consists of four virtual environments (Figure 3). The first virtual environment is a drawing of Baščaršija created as a 360 image. The actor recorded against the green screen is wearing the traditional costumes of the craftsmen and telling a story about each craft. Clicking on the entrance behind the actor, the user enters into the VR video of the craftsman’s workshop, where a story told by the last craftsman of that craft is superimposed. The real craftsman presents the process of his craft making.

On clicking on the arrow positioned on the street pavement, the user moves to the second virtual environment, a VR video of Baščaršija square, where the story about crafts recorded in the oldest census from the 15th century is told. Around the user are names of crafts which existed at that time and explanations of their meaning. From that virtual environment the user can move to the third one where the story about crafts from the 16th century is told and they can see the names of those crafts around them. The last virtual environment offers the story about crafts from the 17th and 18th centuries, together with their names and explanations.

3.2.3. Workflow

The work on the Virtual Museum started with the design of the application concept. Based on information provided by historians from the Museum of Sarajevo, we created the application structure in such manner that the users can learn about crafts from Baščaršija in an attractive and amusing way, as well as get a feeling of life in the place during the Ottoman period. According to that concept, the writer created three stories for the actor, with literary descriptions of craftsmen’s life in past times. We selected the actor who was suitable for this performance and the costume designer prepared three different costumes. A famous Bosnian illustrator created the drawing that would be placed in the background, enjoying the opportunity to “enter into his own illustration”.

In the production phase we filmed the interviews with real craftsmen, as well as their work processes. In VR, we recorded their workshops and three locations in Baščaršija. The actor was recorded on a green screen. The music was composed by a Bosnian rock star, who also created his original song from that music.

We recorded the actor on a green screen with a DSLR camera (Canon 5D Mark IV) and inserted him into a spherical background image. When seen with a VR device, the viewer is placed in the middle of the picture, getting the feeling that the picture is spread 360 degrees around him, and the actor is placed at different positions inside the picture. Both actor and camera are static again, since we did not experiment with camera movement. Further tests are needed before we decide to use it in a final product.

In post-production we edited the interviews with the craftsmen, inserting some historical photos that they provided. We created a composite of the background image drawing and the actor performances and programmed all those assets into a Virtual Reality application. Considering that not many users possess the VR headset, we also created a desktop version of the application and posted it online on a landing page of the project [32]. The workflow of the project’s creation is shown in Figure 4.

Motivated by the quality of the song composed for the project, we also created a music video for that song. This is the first VR music video in Bosnia and Herzegovina. The video is uploaded on YouTube and can be watched in VR on all devices [33].

The Virtual Museum should be installed in the Info points offices of Sarajevo Tourist Organisation to introduce visitors to the intangible crafts heritage. We also created posters with QR codes linked to craftsmen videos. As they cannot afford expensive displays, they will put those posters in their shop windows so that visitors can access their videos and the Virtual Museum application (Figure 5).

3.3. Battle on Neretva VR

3.3.1. Motivation

The Battle on Neretva was one of the most important battles in Yugoslavia during WWII. After many difficulties, Tito’s partisans succeeded in defeating German-Italian forces and their allies—Serb chetniks, in the region along the river Neretva and evacuated their wounded within the Central military hospital. Therefore, this battle is also known as “the battle for the wounded”. Based on the story of this event the most expensive Yugoslav partisan movie was created, casting famous actors such as Yul Brynner and Franco Nero.

The Memorial Complex of the Battle on the Neretva in Jablanica, located at the place where the battle occurred, keeps the memories of this important historical event. Within the reconstruction of the Museum, we were recruited to create digital content for attracting visitors and complementing the physical exhibition. The only part that cannot be provided within the physical exhibition is to experience the battle. Now it can be done using Virtual Reality. The Battle on Neretva VR application places the user in the middle of the battle, giving him a mission to destroy the bridge and save the wounded comrades.

3.3.2. Application Structure

The Battle on Neretva VR application consists of storytelling and gameplay parts. The aim of such application design is to offer the users to learn about the Battle in an amusing and immersive way. The storytelling starts with VR video of the Museum, where the curator character is telling the story about basic facts of the situation before the battle. After that, the user is placed in the VR video of a partisan headquarters, where he/she meets the commander and several soldiers. The commander recounts the situation and informs the soldiers about the orders for the battle. The user is asked a question about the strategy and, if answered correctly, the gameplay loads. The user is on the bridge over Neretva river with a task to destroy it using explosives. After successfully completing the first mission, the user is placed back at the headquarters, this time destroyed by the enemy bombs. The commander informs on Tito’s orders for saving the wounded and again asks the user a question. If answered correctly, the user is tasked to destroy the enemy airplanes. After each destroyed airplane, a part of the improvised bridge is being built. The last task for the user is to transport the wounded comrade across the improvised bridge.

Upon successful completion of the mission the user is back in the Museum, where the curator informs them of the outcome of the battle. As a reward for bravery, the Supreme Commander, comrade Tito, shows up and congratulates the user. The application structure is presented in Figure 6.

3.3.3. Workflow

The application is created in Unity 3D from the following assets: VR videos of two museum and two partisan headquarters scenes, virtual environments of bridge surroundings with 3D models of airplanes, detonator, anti-aircraft machine gun, the improvised bridge and the wounded soldier in the trolley. The virtual environment of the bridge was modelled based on the drone photogrammetric model of the real location. The VR videos were directed according to the new VR film language grammar, using the possibilities of user’s freedom to look around. Comrade Tito was recorded on green screen and superimposed in the VR video of the museum scene. Using make up we tried to make the actor as similar as possible to the historical character. He studied the way of talking and mannerisms of Josip Broz Tito recreating them in his speech. In sound postproduction we coloured his voice to achieve the maximum resemblance. The folley sounds, environment sound and the audio of actors were mixed into ambisonic format together with the original music. The workflow of application creation is showed in Figure 7.

Since we were satisfied with end results of “Nine dissidents” movie and viewers were pleased with what they saw, we decided to apply most of these experiences in “Battle of Neretva VR”. We again used smaller space adding to the tense atmosphere, and also the space was converted into partisan headquarters. All the lamps in the room were placed in a way that they could both be parts of the set, and help with lightning it. One important difference we implemented was that the camera is now actually the viewer and other actors play towards it, where in “Nine dissidents” it was just “unseen” observer. This added another layer of immersion.

The Museum will install the application in the space called VR showroom, where it will be available for visitors on Oculus Quest headsets. We believe that it will motivate the visitors to come and complete their visit of the exhibition with experiencing the battle and becoming its participants.

3.4. Sarajevo 5D

3.4.1. Motivation

The history of Sarajevo starts in medieval times. Until today the appearance of the city has been changing. Some cultural monuments disappeared and in their places different buildings have been built. Those objects are facing a danger of being completely forgotten. Augmented Reality offers a possibility to recreate them in their original locations and display them on mobile devices. This way the users of AR application can learn about their historical significance and their original appearance. The application can also guide them to the places where the objects have once existed.

Due to budget constraints, we selected six cultural monuments or objects important for memory of Sarajevo’s past. In the very downtown there was a mosque of Havadza Kemaludin. Shortly before the World War II, during the reign of the Kingdom of Yugoslavia, the authorities destroyed the remains of the mosque and began the brave construction venture that will permanently mark the panorama of the city. They built a large building—the first skyscraper in Sarajevo, one of five in the region at the time.

A memorial to Archduke Franz Ferdinand and his wife Sophie Chotek was erected in 1916, on the left-hand corner of the Latin Bridge. It was removed by the authorities of the Kingdom of Serbs, Croats and Slovenes in 1919.

The fortification known as “White Bastion” is one of the most impressive and important historical sites in Sarajevo. The archaeologists found remains from medieval, Ottoman and Austrian Hungarian period on the site. We present the assumed appearance of the fortress at the beginning of the Ottoman period. A department store popularly called “Sarajka” (Sarajevo girl) or “Plavuša” (Blond girl) as the citizens of Sarajevo called it because its facade was lined with blue panels, was built, equipped and put into operation on 5 April 1975, as a gift from the UNIMA company for the 30th anniversary of the liberation of the city of Sarajevo. During the siege of Sarajevo—Sarajka was completely destroyed. The new building that replaced Sarajka, known today as the BBI Center, was opened in 2009.

The building of the Great Sephardic Temple was completed in 1930. The size of the temple was 100 × 30 m and could accommodate up to 1000 worshipers, while the dome that adorned the top of this building rested on 28 concrete columns, also decorated in Maori style. During 1941, the Nazi government persecuted the Jewish population, looting property and destroying a culture that had coexisted harmoniously with other cultures in Bosnia and Herzegovina for centuries. In 1965, architect Ivan Strauss was tasked with rebuilding the old temple and adapting it to the city’s current needs, because after the war, the Jewish community donated it to the city of Sarajevo. Today the Temple dome is not visible from the street any more.

In place where is today hotel Europe once existed the largest inn in Sarajevo, called Tašlihan. Today, only one wall is preserved from this cultural monument. Figure 8 shows how the user can see the virtual reconstruction of Tašlihan in its original location on the mobile device screen.

3.4.2. Application Structure

The main functionality of the AR application is showing 3D models when tracker images are being scanned from the map (Figure 9) with mobile camera. Each tracker image activates a model of different building, as well as a pin for showing additional informational content in order to deliver most important facts about selected object to the user. If the user is not at the location where the object was originally located, Google maps guidance to the site can be started directly from the application, in order to easily navigate him/her to the exact location. When the user comes to the location and loads the object model, additional content is shown. This feature was added in order to provide more interesting information and thus motivate users to come to the site and explore it. Application is bilingual, so both local and foreign English speaking users can use it. Previous can be seen in the Figure 10.

3.4.3. Workflow

The application design process began with consulting historians to select the most interesting cultural monuments which do not exist on their original locations, or their appearance has changed. After selection of 6 monuments we created 3D virtual reconstructions, upon advices of experts for cultural heritage. The graphics designer created the visual appearance of the application and the corresponding map with tracker images (Figure 9). The application can be used in any location, but if the user is not on the location of a monument, the navigation to the place is offered, as well as additional digital content as motivation for physical visit. After tracking by mobile device camera the tracker images on the map, 3D models of selected monuments appear on the screen. The info button opens the panel with basic historical information about the monuments. The application creation workflow is shown in Figure 11.

4. User Experience

Approaches in usability evaluations are primary determined by the goal of the study. Broad classification of evaluation goals according [34] is:

• Exploration to understand novel problems;
• Explanation of a context of technology use;
• Description of a context of technology use;
• Demonstration of novel tools.

A common feature of these different evaluation goals is informative feedback to designers and developers, especially when it comes to new technology and usability is additionally challenged with users’ attitude towards the adoption of the VR and AR technology.

A systematic review of papers presented at the International Symposium on Mixed and Augmented Reality (ISMAR) in the period 2008–2017 showed that not only has the number of evaluation papers increased, but the number of evaluations reported in all papers increased [35]. The same review established the following classification of AR evaluation research: (1) surveys providing response to new technology or an overview of important areas of AR research, (2) user evaluations which report on an interesting user interface, and (3) fundamental perceptual studies. The authors conclude that nowadays it is expected that most AR papers should include some form of evaluation when a new input or interaction method is introduced.

It is interesting to note that research related to VR/AR evaluation is mainly focused on novel interfacing solutions and technologies, and quite rarely to genuine and overall user experience. The later objective is foundation of our approach.

Within the framework of our research topic of time travel to the past of Bosnia and Herzegovina through VR and AR we will describe two different UX evaluation methods that we usually employ: (1) qualitative studies for exploring new modalities of expression and pilot solutions, and (2) quantitative studies to obtain reliable measure of success of VR/AR applications.

The main purpose of the qualitative studies is to explore user attitudes, and compare their experience with respect to different levels of preconditioning, and identify strengths and limitations of the approach. The quantitative approach in measurement is important when assessing the level of development of the application or comparing different solutions, and we primary focus on measuring success in achieving features as edutainment and immersion.

We mainly use interpretation of UX evaluation results to identify flaws in user interface and in overall presentation of narrative, as it is already noted that for developers the goal is to improve solution and there is no claim that the results should be generalised [34]. However we design our quantitative studies following the established framework to enable comparison between the VR/AR applications and to identify patterns in users’ preferences.

Qualitative studies are essential in understanding user behaviours and evaluating the situational use of technology, especially to address novel designs and understanding users’ response to new modalities for presenting narratives. We have opted for the qualitative approach for the initial evaluation of the “Nine dissidents”.

In our study we combined semi-structured interviews and contextual observations for data gathering, and basic content analysis of the answers, as a typical approach when the research question is to understand the phenomenon. We have interviewed 12 participants with different professional backgrounds: arts, humanities, and computer science, balanced age distribution, with half participants from Bosnia and Herzegovina and other half from different partner countries from the COST Action CA16213 “New Exploratory Phase in Research on East European Cultures of Dissent” (NEP4DISSENT).

In order to document emotional experience of the participants; we have included the open questions asking for the feelings after viewing the film and their feedback. The answers were analysed to identify key themes linked to the user experience. The most frequent themes have been identified as: Story, Side, Accept, Communists, Felt, Impression, and Narrative. The list of the themes identified from the open questions are shown in the word cloud diagram depicting their frequency, see Figure 12.

In addition to open questions we asked participants to provide their opinion on how the film “Nine dissidents” contributed to consolidations regarding the objectives of the COST Action CA16213 NEP4DISSENT. The users rated statements using a 5-point Likert scale with the following order of grades: 1—the lowest grade, and 5—the highest grade. A very sensitive and complex topic explored in the VR movie were reflected in divided opinions and with all scales converging to average—neutral grade. The statements reflecting the COST Action CA16123 objectives and respective statistics (mean and SD) are presented in

Table 1. Summary of statistics (mean and SD) for users grade of the fulfilment of the COST Action CA16123 NEP4DISSENT objectives.

CA16123 NEP4DISSENT Objective	Mean	SD
O1. Consolidation of the inheritance of Cold War-era conceptual distinctions.	3.42	1.55
O2. Consolidation of the confinement of research within national silos.	3.75	1.36
O3. Consolidation of the neglecting the problem of access to original archival sources.	3.50	1.44
O4. Dissemination through engaging in productive dialogue.	3.08	1.61
O5. Providing a platform for incubating transnational and multidisciplinary research.	3.67	1.55
O6. Providing a platform for technology-conscious research.	3.58	1.50
O7. Best practices of cooperation between multidisciplinary experts.	3.75	1.42

. The objectives related to multidisciplinary research potential received the highest grades—objectives O7 and O5, but also statement reflecting consolidation of the confinement of research within national silos. The statement “Dissemination through engaging in productive dialogue” received the lowest grade—3.18.

It is important to note that the users recognized the potential of this type of medium for presenting such challenging topics, as reflected in the following feedback: “Creating a VR film that places us in the courtroom, a setting where multiple narratives could play themselves out, confront one another”.

Example of a more quantitative oriented evaluation can be found in our evaluations for: Old crafts Virtual Museum, Battle on Neretva VR and Sarajevo 5D.

The quantitative questionnaires were organized in different sub-scales, where each sub-scale was represented by balanced number of items. We have tried to follow similar approach in designing questionnaires in order to provide a standardized measurement tool for user experience.

Our studies focused on examining the basic constructs: Performance Expectancy (PE) and Effort Expectancy (EE), adopted from the UTAUT model [36]. Evaluating Performance Expectancy of a digital storytelling applications is linked to measuring good experience and its important feature: immersion. In addition, evaluation of multimedia historical and cultural heritage applications involves measurement of another important factor combining education and entertainment: edutainment. The Effort Expectancy is linked to perceived ease of use, referring to the degree to which the user expects effortless use of a new technology.

Developing standardized system for evaluation requires development of a framework questionnaire which is adapted to a specific content of a specific VR/AR application.

The framework questionnaire contains four sections: (1) introductory part with demographic data, (2) questions assessing “completeness of interaction”, (3) main part for evaluation made up of 3 sub-scales addressing immersion and edutainment linked to Performance Expectancy (PE), and usability satisfaction linked to Effort Expectancy (EE); and (4) open questions where users can express their opinion about the presentation and the most favourable and the most problematic parts. Evaluation of sub-scales is performed using a 5-point Likert scale.

We present the UX evaluation study of the VR application Battle on Neretva and the results obtained. The experiment was conducted at the Sarajevo Graphics Group Lab at the University of Sarajevo. The questionnaire structure and overview of statistical measures, the mean and standard deviation are presented in

Table 2. Questions are split by evaluation construct, with EDx directed towards Edutainment and IMx towards Immersion, both linked to Performance Expectancy, and EEx addressing Effort Expectancy. Responses were delivered on a 5-point Likert scale, with 1-strongly disagree and 5-strongly agree. Respective summary statistics are shown: Mean and Standard Deviation (SD). Note: Questions EE2* and EE6* are presenting negative attitudes, hence the responses were complemented prior to calculations.

Item	Mean	SD
ED1 I learned a lot from the stories	4.58	0.51
ED2 I become interested in the history of that period	4.50	0.52
ED3 I would like to learn more about the historical events in BH during the World War II	4.50	0.90
ED4 I was amazed by the idea of demolishing and building the bridge	4.75	0.45
ED5 I was surprised how important the demolition of the bridge was	4.58	0.67
ED6 I would love to see more stories about historical events	5.00	0.00
IM1 I liked the implementation of the VR environment	4.92	0.29
IM2 The voice of the actor was authoritative and aroused interest in the battle	4.67	0.49
IM3 My time passed quickly	4.67	0.65
IM4 The games were interesting to me	4.50	0.80
IM5 I almost felt like I was on the field	4.08	0.67
IM6 I could imagine partisans in the snow and on the bridge	4.00	0.85
IM7 I liked the VR reconstruction of the temporary bridge	4.67	0.65
IM8 The environment of the bridge is realistic	4.33	0.78
IM9 My time passed quickly as I moved	4.58	0.90
IM10 The possibility of crashing the plane and building a bridge is interesting	4.75	0.62
EE1 I recognized commands for moving and interacting with objects	4.00	0.95
EE2* I felt dizzy from moving	3.75	1.22
EE3 Navigating was easy	4.50	0.52
EE4 Interacting with objects like detonators was simple	3.58	0.90
EE5 It was clear to me how to navigate through the VR app	3.83	1.03
EE6* I had a hard time controlling movement through the model	3.67	1.15

. Mean values for Performance Expectancy items are in the range of 4.0 and 4.92 for the immersion and in the range of 4.50 and 5.00 for the edutainment. Mean values for Effort Expectancy items are lower and in the range of 3.58 and 4.50.

Notable higher mean values indicate a non-symmetric distribution of answers so better insight in response frequencies is provided in Figure 13 and Figure 14.

Our findings are that the user experience is influenced by the quality of their gesture interaction and user skills, but mostly by the unique experience of the VR/AR simulation and also the quality of narrative presented. The perceived user satisfaction with navigation is not correlated to the level of assistance participants needed, what we have proved in our previous research [37].

It is important to mention that IEEE Computing Edge magazine discusses usability in AR and VR presenting editorial opinion that in AR and VR, “clunky headsets and unintuitive interfaces that require exaggerated gestures can create negative user experiences” [38]. The evaluation results we obtained indicate that the quality of narrative has a positive effect on immersion and presence, even in presence of identified problems, when it comes to ease of use. We have previously concluded that for non-business applications the standard Technology Adoption Model (TAM) changes so that the Performance Expectancy influence the Effort Expectancy [39], instead of opposite, being influenced by.

In simple words: users when motivated to experience VR/AR narrative are prepared to put more efforts in usage of the application. The UX evaluations for our VR/AR applications for time travel to the past of Bosnia and Herzegovina supported this conclusion.

5. Conclusions and Lessons Learned

The presented research shows how powerful eXtended Reality technologies are in presentation of tangible and intangible cultural heritage. Four projects we described brought our users back to the history of Bosnia and Herzegovina and recreated objects, events and characters from its past.

VR video and Virtual Reality applications we created are appreciated by the users as very immersive and educational. They motivate them to make physical visits to presented museums and cultural monuments and enhance cultural tourism development. We confirmed the working hypothesis that interactive digital storytelling through actors playing historical characters adds to edutainment value and overall user appreciation of digital cultural heritage applications. They breathe life into virtual reconstructions of cultural monuments and create feeling of immersion in the past.

From the described projects we learned the following lessons: in VR video the camera can take over the role of the user, adding immersion to the experience. Virtual Reality applications are still not accessible to all internet users, so the alternative desktop VR version should be produced. For desktop VR, the best solution we experienced was WebGL. Augmented Reality offers great potential for presenting reconstructions of cultural monuments on-site, as most of the users possess mobile devices. However, the virtual models have to be optimized for mobile platforms and that could reduce the realism. The use of actors as storytellers is highly appreciated by the users. They feel empathy and emotion and perceive the life in the virtual environments of cultural monuments as it used to be in the past. Museum visits can become more attractive by adding a VR experience of the exhibition topic, such as the Battle on Neretva. The users feel as participants of historical events and are able not only to see, but to experience cultural heritage. We believe this will become a new trend in museology. Adding gameplay elements makes the digital heritage applications even more attractive, motivating the users to achieve goals and solve challenges. However, the level of gamification has to be appropriate and should not decrease the seriousness of particular historical topics.

Digital content should complete the museum exhibitions presenting the heritage aspects that can not be seen in the physical space, such as recreating historical characters and life in the past. Exhibits preserved in fragments can be virtually reconstructed enabling the visitor to understand how they used to look like originally. Another added value is the possibility of interaction with digital exhibits in a way impossible for the physical exhibits.

Limitations of presented approach are mainly in the affordability of VR headset setups. Most of the museums in Bosnia and Herzegovina are in difficult financial situation and purchasing this equipment presents a problem. Also, some visitors not familiar with computer games struggle with interaction in VR applications. We believe that in the future these limitations will be overcome by decrease in prices of technology and raised familiarity with VR through more frequent use in various areas of everyday life.

In the future we plan to explore the camera movement in VR video, new ways of User eXperience design and interaction for Virtual and Augmented Reality applications and possibilities of interacting with virtual storytellers. Cultural heritage of Bosnia and Herzegovina offers endless inspiration in its variety. Our next topic will be medieval Bosnian Kingdom.