ARTutor—An Augmented Reality Platform for Interactive Distance Learning

Abstract

Augmented Reality (AR) has been used in various contexts in recent years in order to enhance user experiences in mobile and wearable devices. Various studies have shown the utility of AR, especially in the field of education, where it has been observed that learning results are improved. However, such applications require specialized teams of software developers to create and maintain them. In an attempt to solve this problem and enable educators to easily create AR content for existing textbooks, the ARTutor platform was developed. It consists of a web-based application that acts as an AR authoring tool, and an accompanying mobile application that is used to access and interact with the educational AR content. In addition, the ARTutor application allows students to ask questions verbally and receive answers based on the contents of the book. This means that the system is suitable for distance learning and promotes self-study and independent learning.

1. Introduction

Modern students have changed, sometimes being referred to as “digital natives” [1], since they were born in the era of digital technology. Most of them have everyday access to the Internet, use smartphones, and can study educational content remotely using various learning tools and platforms. Consequently, their educational needs and ways of perceiving information have changed over the last years. On the other hand, technology advances and the progress observed in the field of mobile technologies and telecommunications have caused enormous change in learning environments, leading to the conceptualization of “mobile learning” and “ubiquitous learning” [2]. Information and communication technologies (ICT) provide new means for content dissemination and user interaction, both with the content and the learning platforms. Therefore, more dynamic learning environments are needed. AR is a technology that enriches human senses and mixes real and virtual environments, leading to a new, more informative and stimulating environment where the user can interact in real time [3]. AR applications can be used on many different platforms such as desktops, notebooks and mobile devices; however, AR applications are usually available through mobile devices, such as smartphones and tablets, and employ built-in cameras, GPS sensors, and Internet access to embed real-world environments with dynamic, context-aware, and interactive digital content [4]. These capabilities have created an interest in using AR applications for educational purposes in the field of mobile learning, and the new possibilities for teaching and learning have increasingly been recognized by educational researchers [5].

AR is currently being applied across disciplines in primary, secondary and higher education, and has been found to increase the academic success levels and motivations of students [6]. Fotaris et al. [7], in a recent systematic review of 17 studies between 2012 and 2017, conclude that AR in education can potentially influence the students’ attendance, knowledge transfer, skill acquisition, hands-on digital experience in education of a variety of domains. Other studies, such as [8], conclude that AR is useful especially for the instruction of subjects that students are not fully able experience and understand because of real-world constraints.

Several areas of AR have been indicated in an educational context, including AR books, AR gaming, discovery-based learning, skills training and object modelling [9]. One of the simplest applications of AR in education is AR books. These are normal books, except that their pages contain extra digital content that can be seen through an electronic device such as mobile phones, tablets, or even personal computers enhanced with cameras. Through the use of AR in printed book pages, books can become dynamic sources of information. In this way, people with no computer background can still have a rich interactive experience with educational benefits [10]. Over the last years, many educators and researchers study the educational value of augmented reality books. Karamanoli and Tsinakos [11] created an Android application that uses Aurasma to provide augmentations in the Greek history book of the 4th grade of primary school. Hung et al. [12] developed an animated AR graphics book displaying 3D images of bacteria. It utilized a virtual reality head-mounted display with a small video camera attached at the same level as the students’ eyes and the results showed that AR improved students’ learning performance and raised their motivation. Tobar-Muñoz et al. [13] presented an AR game inspired by traditional “pop up” books, which is played in a naturalistic environment. Each page of the book comes to life using book illustrations as AR markers.

According to a recent review of 46 AR books [14], the main feature of these AR books is the augmentation of book images and/or markers by providing auxiliary information through 3D models, videos and images. The user can interact with the AR items either through markers or virtual buttons. Some other applications use natural language recognition in order to allow users to interact with the application [15].

Apart from mobile applications that use AR for educational purposes, there have been various authoring tools developed to allow teachers to create AR educational scenarios. One such example is ARLearn, an authoring tool that supports the creation of training scenarios and serious games based on decision making [16]. Another example is mLearn, a platform developed by the authors to help teachers create and manage educational field trips and enable students to access the educational material from their mobile phones [17].

This paper presents ARTutor, a platform that consists of an AR authoring tool that allows educators to augment existing books (either in printed or in electronic format) in an easy and simple way without the need of programming knowledge, and a mobile application that is used to access and interact with the educational AR content overlaying the textbooks. In addition, the ARTutor application has a unique feature that allows students to interact with the augmentations using voice commands, and also to ask questions verbally and receive answers based on the contents of the book.

2. Materials and Methods

The ARTutor platform consists of two parts: (a) the authoring tool, which is a web-based application used to upload the learning material and the assorted learning objects; and (b) the mobile application, which downloads and displays the learning objects and also allows interaction between the learner and the learning material.

The authoring tool has been developed using client-side technologies such as Javascript and jQuery in order to create the user interface and handle user-triggered events, and server-side PHP scripts to implement the web service and to handle communication with the database. To enter the authoring tool, a user must login with a Google account. Authentication has been implemented using the standard Google Sign-In process to manage the OAuth 2.0 authentication flow.

The entire ARTutor web application has been installed on an Apache web server running a MySQL database. The database consists of two tables to store book information (table ‘books’) and asset information (table ‘assets’) respectively.

Table 1. Structure of MySQL tables.

‘Books’	‘Assets’
id	id
title	book
description	title
filename	type
author	filename
author_name	commands
last_modified	trigger_image
category	last_modified
language	location

summarizes the fields that are contained in the MySQL tables.

Books created on the authoring tools are typical PDF files, which are uploaded to the server during book creation. Rendering and display of the contents of these files after upload has been achieved with the open source PDF.js library, which is a general-purpose, web standards-based platform for parsing and rendering PDFs, developed by Mozilla. In addition, the PDF files are processed after upload, and the text they contain is extracted using the PdfToText library, which is also available for free.

A web service has also been developed for the authoring tool. The web service is a collection of scripts that allow the ARTutor mobile application to access the database, retrieve data such as book and asset information, and download target images and augmentation files.

As far as the ARTutor mobile application is concerned, it has been developed for both the iOS and the Android platforms using native developing tools, i.e., XCode and Android studio respectively. It has been designed to operate in the Greek and English language, depending on the device’s language setting (the default language is English).

The main function of the application is to recognize target images and superimpose augmentations on the trigger images. For this part, the Kudan AR Software Development Kit (SDK) has been utilized. Before deciding to use the Kudan augmented reality engine, several other AR engines that provide AR functionality were considered for use in the ARTutor mobile application, namely Apple’s ARKit, Google’s ARCore, Vuforia, Wikitude and ARToolkit. Kudan was determined to be the most suitable solution for the following reasons:

(a) A cross-platform solution was needed, so that the mobile application would be available for both the iOS and the Android operating systems, ensuring use by the majority of available devices. Therefore, proprietary augmented reality libraries such as Apple’s ARKit and Google’s ARCore were dismissed.

(b) In addition to compatibility across operating systems, there was the requirement of compatibility across a variety of devices. The aforementioned AR libraries are suitable to a limited number of devices. In the case of Apple’s ARKit, its use is limited to newer devices running iOS 11 and above, and in the case of Google’s ARCore it is currently compatible with only two device models only, due to specific hardware requirements.

(c) In terms of functionality, the basic requirement of the ARTutor was the superimposition of augmentations on trigger images (i.e., image recognition) and not the placement of digital objects on planes (i.e., plane recognition), which is the objective of both the ARKit and ARCore libraries.

(d) Focusing therefore on cross-platform AR libraries that have image recognition functionality, several alternatives have been considered, i.e., the ARToolkit, Vuforia, Wikitude and the Kudan libraries. For the first three libraries, the trigger images have to be converted to a proprietary format or trained in advance, and be embedded into the application before they can be used in image detection. This presented a significant problem in the development of ARTutor for two reasons. Firstly, in the case of ARTutor the trigger images are not predefined and embedded in the application, but instead they are created by the user in the authoring tool and have to be downloaded by the application at runtime. Secondly, the conversion tools for these AR libraries in some cases are not suitable for execution as a service on a server, and in other cases they are not available for free. In contrast, the Kudan SDK has no such limitations; the user-created trigger images can be used directly as common image files without further conversions. This also implies that the authoring tool can be independent of the augmented reality engine used in the application.

(e) In addition, the use of Kudan SDK is free for non-profit and educational institutes, which is in line with the authors’ vision of ARTutor being free to use for academic purposes.

For these reasons, the Kudan SDK has been selected as more suitable for the purposes of developing the ARTutor mobile application. As far as the 3D models that can be uploaded in the authoring tool and displayed in the mobile application are concerned, the formats FBX (*.fbx), OBJ (*.obj) and COLLADA (*.dae) are supported. These files must be converted to the Kudan *.armodel format using the application Kudan AR Toolkit, which is available for free on the Kudan website (www.kudan.eu). After conversion, the 3D models need to be archived in a zip file together with any texture files, before being uploaded to the ARTutor authoring tool at book creation time.

The speech recognition and text-to-speech functions have been implemented using the iOS and Android official libraries for these functions.

3. Results

Using the tools and methods described in the previous section, the ARTutor authoring tool and the mobile application were developed. This section discusses the resulting functionality of both aspects of the ARTutor platform, i.e., the authoring tool and the mobile application.

3.1. The ARTutor Authoring Tool

The web-based authoring tool is used by the teacher to upload the educational material, namely a book and its corresponding augmentations. After signing in with a Google account, the teacher can view his previously created augmented books or create a new book. When creating a new book, the teacher must upload a document in PDF format and assign a title, a short description and a category. The document can be an electronic book, lecture notes, presentations and other educational material. Also, the teacher selects the language in which the electronic book is written. Figure 1 illustrates the login screen and the form displayed when creating a new book.

Immediately after the upload of the PDF file is completed, the server automatically executes a script in the background, which extracts the text contained in the book and stores it in a separate plain text file. When the process is complete, the teacher can save the book, which results in its information being stored in the database, and the relevant files (the PDF file and the plain text file) being stored on the server. In addition, a new file which contains the book information in JSON format is generated. This JSON file will be used by the application to retrieve the book and corresponding files information, as it will be discussed later. The file also contains a timestamp which indicates the last time the book was edited.

When the book is created, the teacher can then start to add augmentations and associate them with the various pages of the book. Various types of augmentations are supported, namely images, videos, 3D models and sounds. To create a new augmentation, the teacher needs to complete the dialog that appears after the corresponding button is pressed. The information entered in the dialog are used to create an asset i.e., a learning object which contains information about the trigger image (the image that when recognized by the application will initiate the augmentation), the augmentation itself, a descriptive title, and some interaction commands that can be associated with the asset. The required fields of the asset creation dialog are shown in Figure 2.

Initially, the teacher will have to select the area of the PDF file that will serve as the trigger image of the augmentation. This is achieved by displaying the book in a separate window so that the user can browse through its pages. When the desired page has been reached and rendered for display, the teacher can use the mouse to drag-select an area. The selected area is converted to a picture file and the file is stored in the server. Figure 3 illustrates the rendering of the electronic book and the selection of an area in the selected page.

This process enables the teacher to easily create the trigger images from inside the authoring tool, and not rely on other image capturing software. The intuitive process of trigger image creation also implies that there are no special skills required in the use of other software.

Next, the asset is given a title and the actual augmentation that corresponds to the selected trigger image is uploaded. This is the digital object that will be superimposed on the trigger image every time the trigger image is recognized by the ARTutor mobile application. The process is simple; the teacher drags and drops a file in the designated area of the dialog, and uploading commences. The file can be an image, a video in mp4 format, an mp3 file, is or an archive file (zip) which contains a 3D model and its associated texture images, if available. The authoring tool automatically detects the file type and so the type of the augmentation is determined internally without user intervention.

Each augmentation can also be accompanied by voice commands that are used in the mobile application to interact with the augmentation. Such commands are: Start, Stop (applicable to video augmentations), Enlarge, Shrink (applicable to image and 3D model augmentations) and Rotate (applicable to 3D model augmentations). The teacher can add these commands to the asset, depending on the type of augmentation that was uploaded.

The creation of the asset is completed by pressing the appropriate button, which saves the trigger image and the augmentation file on the server, and inserts the relevant information into the database. It also appends the asset’s information on the previously mentioned book’s JSON file, so that the mobile application will be able to download the corresponding assets. The process of adding assets to the digital book is repeated in the same manner, for as many augmentations the teacher needs to add.

An additional educational benefit of such a process is that any augmentation that is created on a book is instantly available to all of the end users of that book (i.e., students or other teachers). In that way ARTutor also provides a sense of collaborative augmented reality content enhancement in a very transparent and easy form.

3.2. The ARTutor Mobile Application

The mobile application serves as the tool that students use to access the augmented books created in the authoring tool described in the previous section. Upon starting the application, the book categories appear on the screen. When a category is selected, a connection with the web service of the authoring tool is initiated, and all the augmented books in the selected category are displayed. Selection of a particular augmented book results in the downloading of the JSON file for the book on the device. As mentioned earlier, the JSON file contains information regarding the assets with which the book is associated. The information contained in the JSON file is compared by date to any existing JSON file on the device. This comparison prevents the same book assets being downloaded on the device if the book has already been viewed previously, and therefore ensures that the latest version of the book’s assets is stored on the device at any time. With the latest version of the selected book’s JSON file on the device, the actual files (trigger images and augmentations) that are related to the assets are downloaded. When the download is complete, the Kudan augmented reality engine is initialized, and the camera preview appears on screen. It is in this state that the device is ready to start scanning the printed version of the book and recognize the various trigger images. The process is summarized in the flowchart of Figure 4.

Figure 5 illustrates two examples of augmentations, a video and a 3D model augmentation, that appear when the respective trigger image is recognized by the application.

When an augmentation is active and visible, it can be manipulated according to the commands that have been assigned to each asset in the authoring tool. The student can use finger gestures to start or stop a video (tap on the video display area), enlarge or shrink a video or an image (pinch-to-zoom gesture) or rotate a 3D model (horizontal slide gesture). Only commands defined in the authoring tool are allowed for each augmentation.

Another way in which the student can interact with the augmentations is using voice commands. This feature is important for both improving the interaction of students with the augmentations as well as for providing accessibility for students with physical disabilities. As seen in the screenshots in Figure 5, when the trigger image is recognized, apart from the augmentation, the ARTutor icon appears on the bottom right corner of the screen. This is a button that, when pressed, starts recording voice. If the recognized word/command matches one of the voice commands assigned to the active asset, then the corresponding transformation is applied to the active augmentation. If not, no action is performed, and a visual and audio message indicating an unknown command is produced. Voice commands must be uttered based on the application’s active language setting, i.e., in English or Greek.

The ARTutor voice recognition button also serves another important purpose. It allows the student to ask questions and receive answers based on the contents of the electronic book. The objective of this function is to simulate a scenario in which a student asks questions to an actual tutor while studying the material. This feature is not only expected to make information retrieval easier, but aims at promoting independent study and reinforcing distance learning. To indicate a question instead of a command, the student must first utter the word “Question” at the beginning of his question. When the recording of the question is complete, the word “Question” is removed and the rest of the recognized sentenced is sent to the server as a search string via the web service. On the server side, a script looks for the search string in the plain text file, which was created when the electronic book was initially uploaded using the authoring tool. At this stage of development, the search script is a simple word search algorithm that searches for words of the search string in the text and returns the sentence in which the words are contained. A more sophisticated search approach is currently being implemented, which involves more complex information retrieval algorithms. The answer formulated in the server is returned on the device and it is displayed on screen with a popup window and it is also pronounced using the text-to-speech function. Figure 6 shows the popup window containing a retrieved answer after a question has been submitted by the student.

In contrast with the voice commands, questions have to be formulated in the language of the book, which may be different from the language setting of the device, and therefore the language of the application. This allows the use of the feature for books of any language, without the need to change the device’s language settings.

3.3. Evaluation Method

ARTutor is composed of a number of features that need to be tested further. Each feature, as well as the whole system, needs to be evaluated with specific measurements according to the goals of the evaluation. Hence, the evaluation goals have to be determined in advance. The Technology Acceptance Model (TAM) [18] attempts to anticipate technology acceptance by individual users. According to TAM, the user-perceived ease of use and usefulness of a technology are two determinants of its adoption. Therefore, the ease of use and usefulness of ARTutor should be evaluated, in addition to any possible effect on the learning process. Based on this model, it was decided to complete the formative evaluation of ARTutor in two phases: (a) the qualitative assessment of the tool by a small number of evaluators, and (b) testing in an educational setting by a larger number of students.

At this time, the first phase of formative evaluation has been completed. The main goal of this stage was primarily to ensure correct operation of the software, and receive initial feedback regarding the usefulness of ARTutor. The evaluators were two members of academic staff of the Computing and Informatics Department at the Eastern Macedonia and Thrace Institute of Technology, as well as three members of the Advanced Educational Technologies and Mobile Applications Lab, chosen because of their experience in the field of educational technologies. Apart from helping with improving the functionality and the user interface, the experts’ comments in terms of the platform’s academic value were encouraging. The ease of use of both the authoring tool and the mobile application were recognized, as well as the novelty of the voice-controlled virtual tutor. Experts also provided valuable insights on the possible uses of the platform as well as considerations for the next phase of the evaluation process.

In the second phase, which is to follow, both formative and summative evaluation are planned to take place during the next semester, since the authors’ intention is to use the ARTutor platform as an integral part of the Educational Technologies unit that is taught at the Department of Computing and Informatics Engineering of the Eastern Macedonia and Thrace Institute of Technology, engaging more than a hundred and fifty users. The goal of this evaluation phase is to involve undergraduate students in testing all aspects of the platform, in terms of both usability and educational value. An experimental procedure is currently being designed to meet both of these objectives. According to this experimental procedure, the students will be divided into two groups. The first group will be tasked with developing augmented books, and the second group will assess the educational value of using these augmented books for studying various subjects. The results of the experimental procedure will be quantified in a structured manner using specially designed questionnaires, which will be compiled according to generally accepted methodologies for measuring user experience and usability [19].

The authors will also run, in parallel, a summative evaluation using the SECTIONS evaluation framework, which was initially proposed by Bates and Poole [20], in order to determine the most effective use of the technology in various subjects.

4. Discussion

In this paper, the ARTutor platform was presented. This is a platform that aims at enabling teachers to create augmented books, so that students’ study of traditional books is improved. On one hand, it has been shown that the authoring tool is designed in such a way that teachers, including those with very limited IT skills, are given the opportunity to enhance their traditional educational materials, with minimal training in the use of the platform. There is no prerequisite software or design skills that are required for creating augmented books, since (a) the original books are common files in PDF form, and (b) the augmentations that can be added in the books’ pages are in common image, video and audio formats that can be retrieved from various sources or created using non-specialized software. An advantage of the ARTutor authoring tool is that, in contrast to other augmented reality authoring tools, the trigger images are created on the ARTutor authoring tool itself, and do not need to be generated and uploaded separately for each page of the same book. Apart from enabling the book creator to easily generate these images, it also ensures that each book is treated as a single entity, and not as a collection of target images. The ARTutor authoring tool is free to use, and the resulting books are to be visible to everyone using the application.

On the other hand, the ARTutor mobile application enables students to enhance their understanding of the teacher’s material by displaying explanatory interactive digital content on top of the traditional book. The students are therefore able to better visualize the concepts presented in the various pages of the book and interact with the various types of augmentations. The augmentations can be activated by both pointing to the electronic as well as the printed version of the book. Contrary to other augmented reality applications, the content accessible to students is dynamic as more books are added to the authoring tools, and there are no limitations as to which books are visible and accessible to the user. Interaction with the augmentations can be both haptic as well as verbal. Voice commands not only allow hands-free operation, but they are also an accessibility feature that will enable students with certain kinds of disabilities to manipulate the augmentations. Another very important feature of the mobile application is the capability to perform searches on the content of the educational material. This feature is a novel approach designed to reinforce the self-study practice and independent learning process by providing a virtual tutor who, based on the learning material, will provide relevant information according to the students’ questions.

The ARTutor platform can be improved and expanded in various areas. Currently, our research efforts focus on improving the search algorithm used to retrieved answers from the electronic book based on the student’s oral questions. The present search algorithm will ultimately be replaced by a combination of natural processing and information retrieval algorithms. The objective is to create a system that will be able to handle student questions, expressed in natural language form, and assemble parts of textual information contained in the electronic book in order to compile a natural language answer and return it to the student. At this stage of development, a solution combining the NLTK natural language processing library and the Whoosh information retrieval library is being explored. Both libraries were selected mainly because they are both open source, and they can also work with various languages. In addition, it is being considered that location augmentations should be added to the framework in order to increase its potential. In a similar manner to the other augmentation types, a location would be added as an asset on the authoring tool. When the corresponding trigger image is recognized by the application, a location marker, along with other relevant information, would appear on the screen when the mobile device is pointed to the direction of the specified location. In educational terms, this feature would be useful in geography lessons or during educational field trips. This feature would also reinforce the potential of the ARTutor platform to be a more general-purpose tool for creating augmented documents, for example in augmenting existing tourist guides. Location augmentations have already been implemented, and are currently in the testing stage. Further development and testing is required to ensure full multilingual support across the iOS and Android operating systems.

So far, the software has been tested by a limited number of evaluators, mainly for the purposes of ensuring correct functionality and eliminating any software errors. Having ensured correct operation of the software will allow further investigation in the actual usability and educational value of ARTutor. The experimental procedure that will be followed during the Educational Technologies unit is designed to, firstly, further evaluate the platform across a number of devices and operating systems and, secondly, to evaluate the usefulness of the platform in an educational context. To achieve the second goal, an experimental procedure has been designed. Future research will focus on studying the platform’s usage at various levels of education.

5. Conclusions

In summary, the ARTutor platform is a novel educational tool developed specifically to enable teachers and students to take advantage of the educational value of augmented reality. Its main contributions in the field of education illustrate its potential and can be summarized as follows:

• Easy enhancement of existing learning material by teachers;
• Easy access to and interaction with the enhanced digital content by students;
• Voice-based interaction with a virtual tutor for information retrieval;
• Consideration of students with disabilities by the introduction of accessibility features;
• Instant access to Augmented content;
• Collaborative development of AR content.

Expert review results of the ARTutor authoring tool and the mobile application during the first phase of the evaluation were promising. Over the coming months, the second phase of the evaluation will be completed, and more results will be available regarding the effect that the use of ARTutor has on the improvement of the educational process.