Live Sharing of Learning Activities on E-Books for Enhanced Learning in Online Classes

Abstract

While positive effects of imitating other learners have been reported, the recent increases in the number of online classes have seriously limited opportunities to learn how others are learning. Providing information about others’ learning activities through dashboards could be a solution, but few studies have targeted learning activities on e-textbook systems; it remains unclear what information representations would be useful and how they would affect learning. We developed a dashboard system that enables live sharing of students’ learning activities on e-textbooks. An experiment was conducted applying the dashboard in an online class to evaluate its impact. The results of questionnaires and quizzes were analyzed along with learning activities on the e-textbook system. From the questionnaire results, the most useful feedback types were identified. Regarding the impact on learning, the study found that a higher percentage of students who used the dashboard followed the progress of the class than those who did not. The study also found that students who used the dashboard were more likely to achieve higher quiz scores than those who did not. This study is the first to reveal what specific feedback is useful and to successfully investigate the impact of its use on learning.

1. Introduction

Learning analytics (LA) [1] is based on the analysis of large numbers of learning logs and aims to understand and improve learning and learning environments [2]. As teachers and learners make such improvements, they need to be presented with the findings of the analysis as feedback. The time required for feedback to become available will vary greatly depending on the underlying data and analysis. For example, the final evaluation of lectures and learning materials requires analysis of student’s exam results and final grades, which constitutes feedback on a semester basis. In addition, studies [3,4] have been conducted to analyze attendance, assignment submissions, quizzes to predict student’s final grades or students at risk of dropout, which are examples of feedback on a weekly basis.

A more immediate form of feedback would be to provide information in real-time. The term “real-time” is used here to mean that feedback is provided not only immediately but continuously. The advantage of real-time feedback is that it enables activity optimization in response to minute changes in learning activities. For example, using feedback on students’ learning progress affords teachers flexibility in adjusting the time allocation during lectures to provide support for students who are not following or by adding supplemental explanations to assist their understanding.

On the other hand, students may also benefit from feedback on other students’ learning progress. To begin with, students communicate with each other regularly and exchange information about their study. By referring to others, students can imitate the learning of others. Zhou and Guo reported that undergraduates imitate others in their learning and demonstrated positive effects of doing so [5]. In the past few years, however, online classes using web conferencing systems have become very common due to the 2019 coronavirus outbreak. In such learning environments, face-to-face communication and opportunities to monitor the activities of the teacher and other learners are significantly more limited than in regular classes, which, in turn, leads to negative effects on the sustainability of learning. With regard to the teacher, methods such as video streaming of the blackboard and sharing the teacher’s screen are used to ensure opportunities for imitation. However, except for forums for discussion, means of learning what other learners are doing in a typical lecture-style class are very limited, making imitation difficult. Therefore, assistive technologies that enable robust and sustainable learning under such constraints are becoming increasingly important, and real-time feedback to learners has the potential to be one such technology.

Today, many kinds of media are available for teaching [6], such as text, image, audio, video, animation, and 3D. Nevertheless, one of the typical forms of teaching is lecture-style classes using textbooks or presentational slides, a form that is widely used whether offline or online. For example, at the authors’ university, electronic textbook systems have long been provided as one of the core educational systems [7,8] and have been increasingly used in classes during COVID-19. Besides note-taking, other typical learner activities in classes using e-textbooks include browsing the textbook and writing in the textbook. Providing real-time feedback based on these activities and sharing the situation among learners may not only restore but even enhance some of the interaction between learners that was lost when classes went online. For example, sharing what other learners are focusing on may lead to new insights.

The key to real-time feedback is to present information about the learning activities of others in a form that can be easily and quickly understood during class. In other words, a user interface that allows teachers and students to intuitively understand large amounts of analytical data is essential. Visual feedback is a typical format of LA feedback, among other types, such as audio feedback [9]. The dashboard is often used as a user interface system to provide visual feedback in one place. Dashboards support quick and intuitive understanding by consolidating the information needed by teachers and learners for various purposes in an easily understood format, such as graphs and tables for each type of information, on a single screen. These learning support dashboards are commonly referred to as Learning Analytics Dashboards (LAD). While there has been much research on the design and effectiveness of LADs for students [10,11,12], few have focused on the learning activities on e-textbooks, and it is unclear what the effect of sharing how other learners are using textbooks might be.

This study focuses on LADs that provide real-time feedback based on learning activity logs in e-textbooks and seeks to answer the following research questions:

RQ1

What is the effective representation of activities from the e-textbook in the real-time LAD?

RQ2

How does live sharing of learning activities affect students’ learning in online classes?

In the remainder of this paper, we first describe the related research and then describe a dashboard system that provides real-time feedback. We describe how the learning logs of e-textbooks are analyzed and show the user interface of the dashboard, which provides information based on the analysis results. Additionally, the results of the usefulness evaluation and the analysis of the utilization patterns and impacts of the proposed system will be presented. Analyzing users’ analysis of its usefulness and the relationship between LAD utilization patterns and learning performance are essential to establishing more effective LADs for learning [13]. We analyze the use of the proposed system in a real university’s non-face-to-face online lectures in order to evaluate its usefulness to students and the changes in their learning activities and learning outcomes.

2. Related Work

Many Learning Analytics Dashboards (LADs) have been proposed in the last decade [10,11,12]. The most representative of them focus on activities on Learning Management Systems (LMSs). For example, Arnold et al. [14] proposed a LAD that provides students with success prediction signals based on LMS logs. Kim et al. [15] proposed a LAD that presents the number of logins to the LMS, frequency, and cumulative time, and allows comparison with the overall average. Aljohani et al. [16] proposed a LAD that enables the comparison of student activity on learning content such as quizzes and discussion forums on the LMS with that of a group of superior students.

Focusing on learning activities besides those on LMSs, Minovic et al. [17] proposed a LAD that allows instructors to track the progress of students working on educational games in real-time. As for those focusing on activities in programming learning, Diana et al. [18] proposed a LAD that can validate the effectiveness of a student score prediction model, and Fu et al. [19] proposed a LAD that enables the visualization of error classifications and activity time series.

Although such a variety of LADs has been proposed, few have focused on activities on e-textbooks. Shimada et al. [20,21] developed a LAD that presents students’ preparation status and lecture progress and their changes based on the learning logs of the e-textbook system and proposed a method that allows teachers to grasp information in real-time while conducting classroom lessons. However, only the impact on teachers’ use of LADs has been investigated; the impact on learners has not been studied. Chen et al. [22] proposed a LAD based on clickstream data from an e-textbook system, which provides feedback that enables learners to objectively perceive their learning activities such as browsing time, page browsing paths, and knowledge maps and to compare them with other students. This LAD is quite similar to that proposed in this study in that it is a dashboard that allows an understanding of other learners’ activities, but its main purpose is to support self-regulated learning, and it does not answer the Research Questions of this study. Another difference is that their dashboard is specialized in representing learners’ activities cumulatively, whereas our dashboard provides a time-series representation that allows the observation of changes over time and a live representation that enhances the sense of realism. In summary, while LADs that focus on activities on e-textbooks have been studied, it remains unclear what representations are effective in ensuring that learners share other learners’ learning activities in real-time and on an ongoing basis and what impact sharing has on learners’ learning.

An early version of our dashboard was developed and evaluated in an earlier study [23]. Based on these previous findings, we improved our dashboard system to provide more information and afford better usability in the present study. Furthermore, we more comprehensively evaluate the dashboard’s impact on learning by analyzing the results of multiple questionnaires and the changes in both learning activities and learning outcomes.

3. Proposed Dashboard

Figure 1 shows an overall picture of our learning environment. As shown in the figure, the proposed LAD works with two other systems: a Moodle learning management system [24] and a BookRoll e-textbook system [8,25].

In our learning environment, Moodle serves as an entry point to the environment; the other systems are only available after logging into Moodle. Leveraging the learning tools interoperability (LTI) standard [26], both BookRoll and LAD are implemented as LTI tool providers; these are placed in a lecture course among several learning support tools. When a user uses these tools, Moodle notifies these LTI tool providers of the course and user information via a LTI protocol, and thus they can offer course-specific information to users.

In our university, the following is a typical usage of BookRoll in classes. Teachers register their own course materials to BookRoll prior to the class. Most of the materials are in the style of presentation slides; only a few teachers use book-style materials. During class, both teachers and students browse the same lecture materials. In the case of online classes, a teacher opens an e-textbook on BookRoll and shares the screen with students via a web conference system or delivers only audio to students. Students use their own laptop computer; they browse the e-textbook on BookRoll independently of the teacher while participating in the web conference.

Figure 1. Overview of the learning environment as a whole. The environment consists of three separate systems.

Figure 1. Overview of the learning environment as a whole. The environment consists of three separate systems.

Our dashboard analyzes the usage records of several BookRoll features and provides feedback based on the results. BookRoll’s user interface for browsing an e-textbook is depicted in Figure 2 with such features annotated. As shown in the figure, when browsing, students are able to use functions that support learning activities, such as bookmarking, highlighting, noting, keyword-searching, and accessing recommended websites related to the page content. There is also a function for learners to state whether or not they have understood the content of the page with “I understand” and “I don’t understand” buttons. The uses of these functions are recorded immediately by BookRoll and stored in its database as learning activity logs.

Figure 3 shows the overall view of the proposed dashboard. The dashboard was developed originally for Japanese students; in the figure, we added English descriptions and translated Japanese text of the system user interface and the Word Ranking feedback into English. At the top of the dashboard, there are teal-colored toggle buttons to toggle the display of each feedback type available on the dashboard. The dashboard consists of the following eight feedback types: Browsing Statistics, Browsing Heatmap, Following Status, Comprehension Status, Highlight Heatmap, Word Ranking, Page-Wise Comprehension, and Past Learning Activities. The remainder of this section describes the individual feedback types and the necessary analysis of the activity logs.

3.1. Feedback Types

3.1.1. Browsing Statistics

In classroom lectures, the viewing status of course materials is one of the most important pieces of information for understanding the learning status of students. The proposed dashboard provides the viewing status of learning materials in three ways. First, Browsing Statistics is the most simple one. It displays such information as the number of views, the students’ focus page, the user’s view page, and the teacher’s view page.

3.1.2. Browsing Heatmap

Second, Browsing Heatmap provides information in a more visual form, which displays the distribution of the number of users viewing particular pages. The horizontal axis shows the time, the vertical axis shows the page number of the chosen e-textbook, and the color of each cell shows the number of users per minute; to visualize how many students are viewing each page at a given time. The cells framed with a blue line are the pages that the teacher was viewing at that time. Among such pages, the current page is always shown at the center of the heatmap. From this heat map, users can intuitively see the current and past browsing status of the entire class.

3.1.3. Following Status

Lastly, this feedback type provides a summarized version of viewing status, especially from the perspective of the students who are following the page the teacher is reading. The number of students reading a page ahead of the teacher’s page (red), a page behind the teacher’s page (blue), and the same page as the teacher (green), are shown. The chart next to the numbers represents the ratio of each of the following statuses. Although the heatmap described earlier provides a detailed view of the browsing situation, it takes users a relatively long time to comprehend, and they do not always need that much detail. The summary visualization is complementary to the heatmap because it allows for a quick understanding while limiting the information provided.

3.1.4. Comprehension Status

This type of feedback displays the students’ stated understanding of the page being read by the teacher. The numbers of votes cast by the “I understand” or “I don’t understand” buttons are shown, and their percentage of the total number of votes is shown as a chart. This feedback type is useful for students to quickly find pages that other students also find difficult.

3.1.5. Highlight Heatmap

Highlights are a typical learning method of students, wherein they mark words and passages for future reference. Data about when, where, and how many marks were drawn throughout the class provide useful learning opportunities regarding students’ perceptions of specific areas of interest and their changes.

The Highlight Heatmap provides this information by aggregating the highlights drawn by students as a heatmap. BookRoll provides three colors of markers, each with a different purpose: important, understood, and not understood. Users can choose which types of highlighting to consider when creating the heatmap. In addition, the time of creation of the highlights to be considered can also be limited to those created within the last minute, 5 min, 10 min, or 90 min. With this feedback, users can visually grasp the points that have attracted learners’ attention on each page, helping them grasp the main points on the page while incorporating the perspectives of other learners. They also have the opportunity to view their own learning objectively through the differences between themselves and other students.

3.1.6. Word Ranking

The dashboard also provides a semantic summary of highlights and notes. The word ranking table presented in the dashboard shows the ranking of frequent words in the highlighted areas and in the notes left by students. The ranking of frequently occurring words gives information about what topics learners are focusing on, and which students can utilize, as in the case of the heat map.

3.1.7. Page-Wise Comprehension

This feedback type shows the overall comprehension statuses of pages for a selected material. The bar graph is based on the responses from “I understand” or “I don’t understand” buttons. The horizontal axis corresponds to the page number in the learning material. The blue bars in the upper half and red bars in the lower half show the total number of students who posted “I understand” or “I don’t understand”, respectively. The bars for pages containing the user’s own evaluations are highlighted with a yellow box. The tooltip on the graph shows the number of students, including the percentage of students who rated the page in question. Students can learn about what other students understand, but they do not, which can help them plan which pages to prioritize for review.

3.1.8. Past Learning Activities

While the focus of this study is on understanding the learning progress of other learners in real-time, there is some information that is difficult to provide immediately. For example, it is not possible to know which pages a learner focuses more on until a certain amount of time has elapsed. The proposed dashboard provides such information based on past records of the same class, as it may be beneficial to share the status of past classes if the content of the classes is similar.

Past Learning Activities shows a summary of the learning activities in previous lectures where the same e-textbook that was used can be viewed for each page of the material. The following five pieces of information are presented here:

• the amount of time spent viewing the page,
• the percentage of learners who made marks on the page,
• the percentage of learners who bookmarked the page,
• the percentage of learners who left notes on the page, and
• the percentage of learners who stated that they understood the page.

Students do not have unlimited access to this feature, but only for three pages: the page the teacher is viewing and the pages immediately before and after it. This is to encourage students to follow along in class. The procedure for calculating the above information is described in Section 3.2.2. Students can use these statistics to estimate the importance and difficulty of the page content and reflect on their own learning behavior.

3.2. Analysis for Feedback

In this section, we describe how the dashboard analyzes the learning logs and materials. The overview of the data analysis process is shown in Figure 4.

3.2.1. Word Extraction

To present the words that students focus on, it is necessary to extract the words from the text where the marks are drawn or from the text of the memos created by users. Additionally, words are used to measure the similarity of e-textbook contents in the identification of most similar materials, which will be explained in the next section. In our university, we mainly use lecture materials written in Japanese, which requires word segmentation because Japanese texts do not clearly indicate word boundaries. We use the popular morphological analysis library MeCab [27] to perform tokenization and part-of-speech estimation.

Figure 4. Data generation flow for real-time feedback.

Figure 4. Data generation flow for real-time feedback.

We then remove unimportant words by employing the estimated part of speech labels. For the purpose of extracting the content of interest in the highlights and notes, only proper nouns are retained since words that represent the primary topic of the material are important. For the purpose of identifying similar materials, it is necessary to extract a set of words that covers the entire content of the material. Therefore, unlike the case of extracting the content of interest from memos and highlights, a wider range of words is targeted. In this study, only nouns, verbs, adjectives, and adverbs are retained.

3.2.2. Identifying Similar Materials and Pages

To utilize past records of learning activities, the possibility must be taken into account that lecture materials are updated from year to year and may differ slightly in content. Therefore, it is necessary to map the material used in past lectures to the material used in the current lecture. In addition, the number of pages in the materials may increase or decrease, or the order of the pages may change, so the correspondences must be established on a page-by-page basis.

First, we identify a set of similar materials. Let $\mathcal{C}$ be the set of past courses of the same subject. For each course $C\in \mathcal{C}$, we find the most similar e-textbook $d^{\prime }\in C$ by considering material titles. Let a current material and its title be d and $t_d$, respectively. Similarly, let a material used in the past course C be $d^{\prime }\in C$, and its title be $t_{d^{\prime }}$. We then use the normalized Levenshtein distance $\mathit{NLD}(t_d,t_{d^{\prime }})$ to measure the similarity between $t_d$ and $t_{d^{\prime }}$:

$$\mathit{NLD}(t_d,t_{d^{\prime }})=\frac{\mathit{LD}(t_d,t_{d^{\prime }})}{max\left\{\right|t_d|,|t_{d^{\prime }}\left|\right\}},$$

where $\mathit{LD}$ is the Levenshtein distance [28], and $\left|t\right|$ is the length of the string t. With this measure, the set of materials, $S_d$, most similar to d is defined as follows:

$$S_d=\underset{d^{\prime }\in C,\mathit{NLD}(t_d,t_{d^{\prime }})\le 0.15}{arg\; min}\mathit{NLD}(t_d,t_{d^{\prime }}).$$

Second, we make a mapping between pages of similar materials based on the text content of the pages. For the estimation of similar pages, we use words extracted from page contents. Let the set of words on the i-th page of d be $W_i^d$. The similarity between d’s i-th page of $d_i$ and $d^{\prime }$’s j-th page $d_j^{\prime }$ is defined as follows:

$$\mathit{sim}(d_i,d_j^{\prime })=\frac{|W_i^d\cap W_j^{d^{\prime }}|}{max\left\{\right|W_i^d|,|W_j^{d^{\prime }}\left|\right\}},$$

where $\left|W\right|$ is the number of elements in W. For each similar material $\widehat{d}\in S_d$, the most similar page $\widehat{j}$ to $d_i$ is computed as follows:

$$\widehat{j}=\underset{j\in \widehat{d},\mathit{sim}(d_i,{\widehat{d}}_j)\ge 0.5}{arg\; max}\mathit{sim}(d_i,{\widehat{d}}_j).$$

4. Methods

We introduced the proposed dashboard into actual online classes and conducted user evaluations. Through questionnaires, we asked students to evaluate the usefulness of the proposed dashboard system; these results are discussed later. We also discuss the impact of the dashboard on students’ learning activities and learning outcomes based on the results of BookRoll activity logs and quizzes. In the experiment, we explained to the subjects and obtained their agreement for the use of their data in the research.

Although the proposed dashboard and BookRoll are separate web systems, the student-facing user interface in this experiment displays the proposal dashboard and BookRoll side by side, as shown in Figure 5. This design decision builds on the findings of our previous study [23]. We have found that providing different independent tools at the same time can be burdensome for users, who switch between them as needed. By visually integrating and displaying the two at the same time, we can make them recognizable as a single tool and force users not to switch between them. In this way, we aim to reduce the burden on the user. For this purpose, this study employs a compositional learning environment (CLE) [29]. This is a meta-LTI tool provider that encapsulates and visually integrates other LTI tools. In this way, we can continually share the activity status of other learners while they are browsing the textbook. In this paper, we refer to the composed system as the combined learning system.

The actual procedure for using the combined learning system is as follows. First, when a user opens the combined learning system, the BookRoll and app selection menu will appear. The selection menu presents a number of research apps that we have developed, and the user must select a dashboard from them. After the selection, the BookRoll and the proposed dashboard will appear side by side on the screen. The dashboard then asks the user to select an e-textbook and once selected, the user is redirected to the feedback screen. Normally, the user selects the same e-textbook that is open in BookRoll. By default, all feedback types are hidden, and the user can toggle the display of each using the toggle buttons at the top of the screen. This is intended to avoid user confusion due to information overload by allowing users to select the type of feedback they need.

The experiment was conducted on five courses of “Information Science” for first-year undergraduates offered at our university from October 2020 to January 2021. The dashboard was introduced during an experiment period of 2 weeks out of 15 weeks. During the experiment, the same teacher taught five courses using the combined learning system. For students, the combined learning system was used only in three of the five courses, i.e., only BookRoll was used in the other two courses. The target classes were non-face-to-face online classes using a web-conference system and were conducted by the same instructor. During the classes, the instructor explained the contents using e-textbooks, and two e-textbooks were used for each week.

There were six e-textbooks on BookRoll, D-01, D-02, D-03, D-04, D-05, and D-06, and the combination of textbooks differed depending on the course. The number of students in each course and the e-textbooks and systems used in the lectures are listed in

Table 1. Experimental course groups. “# Students” in the header stands for “number of students”.

Course	# Students	Learning Materials		System
		1st Week	2nd Week
A	234	D-01, D-02	D-04, D-06	Combined
B	130	D-01, D-06	D-02, D-03	Combined
C	157	D-03, D-04	D-05, D-06	Combined
D	140	D-01, D-02	D-04, D-06	BookRoll Only
E	112	D-01, D-02	D-04, D-06	BookRoll Only

.

Next, we describe the flow of the lecture during the experiment. The teacher opens an e-textbook in BookRoll during the lecture and provides explanations while using the real-time feedback on the dashboard as appropriate. The students participate in the Web conference system and receive explanations from the instructor while opening the e-textbook on the BookRoll as well. As the screen sharing function of the web-conference system is not used, the screen of the teacher is not shared with the students, and each student browses on the BookRoll by themselves. Students in the three courses using the combined learning system were given time to view the explanatory video of the dashboard at the beginning of the first lecture.

The voluntary questionnaire was conducted at the end of the second lecture in the experiment. Using a five-point Likert scale, we asked students to rate several aspects of the entire dashboard and the usefulness of the individual feedback types. Our questionnaire for evaluating the entire dashboard was based on the LAD success questionnaire of Park and Jo [30]; its item texts were translated into Japanese, slightly modified for Japanese university students, and reorganized, as shown in

Table 2. Correspondence between our questionnaire items and those of Park and Jo [30].

Ours		Park and Jo
Item	Index	Item	Index
1	Adequacy of visual representation	5	Visual effectiveness
2		6	Visual effectiveness
3		8	Visual effectiveness
4		9	Appropriation of visual representation
5		10	Appropriation of visual representation
6		11	Appropriation of visual representation
7		12	Appropriation of visual representation
8		7	Visual effectiveness
9	Ease of use	14	User friendliness
10		15	User friendliness
11		16	User friendliness
12		17	User friendliness
13	Understandability of the presented information	18	Understanding
14		19	Understanding
15		20	Understanding
16	Relevance to learning	1	Goal-orientation
17		2	Goal-orientation
18		3	Information usefulness
19		4	Information usefulness
20	Usefulness for behavior change	22	Reflection
21		23	Motivation increase
22		24	Motivation increase
23		25	Behavioral change
24		26	Behavioral change
25		27	Performance improvement
26		28	Performance improvement
27		29	Competency development
28		30	Competency development

. Please refer to

Table 3. Questionnaire about the dashboard. Questions were originally written in Japanese.

Questions		Evaluation
		Avg.	Std. Dev.
Q1	The dashboard contains visual elements (e.g., a graph).	4.3	0.8
Q2	The size of the dashboard fits the computer display.	3.5	1.1
Q3	Visual elements (e.g., graphs) in the dashboard are intuitive and easy to understand.	3.9	1.0
Q4	The dashboard contains appropriate graphical representations.	3.9	0.9
Q5	The amount of information presented in the dashboard graphs is appropriate.	3.9	0.9
Q6	The information presented on the dashboard is concise, intuitive, and clear.	3.9	0.9
Q7	The dashboard uses appropriate attention-grabbing elements such as form, color scheme, location of information presentation, and dynamic presentation.	3.9	0.9
Q8	The dashboard displays visual information that is clearly understandable at a glance.	3.9	0.9
Q9	Dashboards are easy to access.	3.3	1.2
Q10	The dashboard is customized to fit your needs.	3.2	1.1
Q11	The dashboard is easy to use because of its intuitive interface and menus.	3.5	1.1
Q12	The dashboard makes it easy to find detailed and unseen information on the dashboard.	3.5	1.0
Q13	I know the meaning of the visual information presented on the dashboard (e.g., circles, lines, and colors in the graph).	3.7	1.1
Q14	I understand the meaning of statistical information (e.g., numbers) presented on the dashboard.	3.7	1.0
Q15	I think I can compare my own and other learners’ situations regarding overall activity patterns.	3.9	1.0
Q16	The specific information presented on the dashboard will help me set learning goals.	3.5	1.0
Q17	The dashboard will help me monitor activities related to the learning goals set.	3.6	1.0
Q18	The dashboard presents information that I need to know.	3.7	0.9
Q19	The dashboard contains only important learning-related information.	3.4	0.9
Q20	It is easy for me to focus on information related to my learning activities.	3.7	0.9
Q21	Looking at the dashboard motivates me to engage in learning.	3.5	1.1
Q22	I am willing to create my own learning plan based on the information in the dashboard.	3.3	1.0
Q23	I will use the information on the dashboard to manage my own learning activities.	3.3	1.0
Q24	I will change my learning patterns (study habits, learning methods, etc.) after monitoring the information on the dashboard.	3.3	1.0
Q25	I think the dashboard will help me achieve my learning goals.	3.6	1.0
Q26	I think the dashboard will help me improve my learning performance.	3.6	1.0
Q27	I think the dashboard will help me improve my self-management skills.	3.7	1.0
Q28	I think learning on the dashboard will help me improve my social values (for example, sense of responsibility, justice, and contribution) and networking skills (e.g., building relationships).	3.3	1.0

Notes: (N = 223); 1: strongly disagree; 2: disagree; 3: neither agree nor disagree; 4: agree; 5: strongly agree.

in the next section for the question text of the questionnaire items. In addition, several open-ended questions were included about specific utility and areas for improvement. The questionnaire also included a question to ascertain whether the respondents actually used the dashboard, to which they responded “Yes”, “I stopped using it midway”, or “No”. Students who answered that they stopped or did not use the dashboard were asked to choose the reason for their decision from a list of options.

5. Results

5.1. Evaluation by Students

5.1.1. Evaluation of the Entire Dashboard

Table 3 presents the results of the questionnaire evaluating the entire dashboard, along with the text of the questionnaire questions. The results do not include responses from students who indicated that they did not use the dashboard.

Q1–Q8: Adequacy of visual representation. The overall responses indicated that the presentation method was easy for students to understand. The relatively low response to Q2 is likely due to the fact that the dashboard user interface design is not optimized for the simultaneous, parallel display of tools. Therefore, it is considered necessary to design the layout for displaying dashboard information in consideration of screen size.

Q9–Q12: Ease of use. These items were not rated highly overall but rather moderately. The standard deviation of the ratings was only slightly larger than those for the other aspects. The reason for the relatively low score for Q9 could be that switching between teaching materials is complicated. Since the dashboard and BookRoll are not linked, switching to different materials during a lecture requires switching in each tool separately, which is more time-consuming than with BookRoll alone. The possible reason for the similarly low score for Q10 is that it does not provide personalized feedback. The scores for Q11 and Q12 were relatively high among the four, indicating that the user interface was intuitive and the ease of finding information was evaluated to some extent.

Q13–Q15: Understandability of the presented information. Overall, the ratings were higher than medium, indicating that the information was presented in a way that was easy to understand. The most highly rated of all was Q15: Students can compare their own learning status with that of the other students. This indicates that one of the goals of this study, the sharing of learning status, has been realized, and, as a result, the possibility of comparison with others is particularly appreciated.

Q16–Q19: Relevance to learning. The items related to learning goals (Q16 and Q17) and the presentation of important information (Q18 and Q19) can be divided into two categories, but they were rated somewhat better than moderate on the whole. Interestingly, Q18 is higher than Q19, suggesting that the information presented includes information that students need, although some of it is not important.

Q20–Q28: Usefulness for behavior change. While the dashboard was perceived to be useful for improving grades and self-management skills (Q20, Q21, Q25–Q27 ), students were less willing to apply it to improve specific learning plans and behaviors (Q22–Q24).This suggests that although students perceive the information presented as useful, they have difficulty actually making use of it. In order to use the information to change their own behavior, students need to read information from the dashboard that is useful for their own learning and build a process to improve their learning. However, most of the information presented is simply a summary of the learning situation, making it difficult for students to determine the specifics and timing of improvement during the class, which may explain the results. Q28 also showed that although it was not rejected that the dashboard would increase the students’ own social value or improve their networking skills, this aspect was only rated moderately.

5.1.2. Evaluation of Individual Feedback Types

Table 4. Questionnaire results on usefulness of individual feedback types.

Feedback Type		Evaluation
		Avg.	Std. Dev.
1	Browsing Statistics (Section 3.1.1)	3.5	1.1
2	Browsing Heatmap (Section 3.1.2)	3.7	1.1
3	Following Status (Section 3.1.3)	4.1	1.1
4	Comprehension Status (Section 3.1.4)	3.7	1.1
5	Highlight Heatmap (Section 3.1.5)	3.5	1.2
6	Word Ranking (Section 3.1.6)	3.4	1.1
7	Page-Wise Comprehension (Section 3.1.7)	3.5	1.1
8	Past Learning Activities (Section 3.1.8)	3.2	1.1

Notes: (N = 223); 1: not useful; 2: not very useful; 3: neither useful nor unhelpful; 4: slightly useful; 5: useful.

shows the evaluation results of the individual feedback types. The results do not include responses from students who indicated that they did not use the dashboard. Following Status was rated as the most useful. The fact that many free-text responses indicated that it was easy to return to the class even after temporarily dropping out of the class suggests that this feedback type was perceived as useful for checking the teacher’s browsing page rather than for checking the status of other learners. Respondents rated Browsing Heatmap and Comprehension Status as the next highest. In addition to these, the relatively high ratings of Highlight Heatmap and Page-Wise Comprehension suggest that learners are, in fact, interested in and concerned about the learning status of other learners. Additionally, Past Learning Activities was perceived to be less useful than other types of feedback. This implies that learners place more importance on the current learning situation than on the past.

5.1.3. Free Writing Answers

The following were the students’ most common responses regarding what they found useful during the lecture on the dashboard.

• Even in the online classes, I felt that I am learning or participating with other students in the lectures.
• I was able to objectively grasp my own learning status even in online classes.
• I was able to compare my own learning status with that of other students to see what I needed to pay attention to or understand.
• It was easy to get back on track even if I temporarily lost track of the lecture.

Among these, the most frequent response was the appreciation of being able to check the learning status of other students even during online lectures.

The following were the most common responses to what needs to be improved in the dashboard and the combined learning system.

• There was some information that I did not know how to read.
• Switching e-textbooks was time-consuming.

It seems that some of the feedback types were not clear to the students, perhaps because the user interface was not easy to understand. Another possible reason is that the prior explanation of use was insufficient. In the experiment, at the beginning of the first lecture, the students were asked to watch a video explaining how to view and use the combined learning system. However, due to time constraints, the video was too short to provide a detailed explanation. Therefore, with long-term use, these problems may be resolved as learners become accustomed to the dashboard. Since the optimal presentation of information and long-term use of the dashboard are outside the scope of this study, these issues will be the subject of future research. As for the effort involved in changing e-textbooks, this is due to the fact that the dashboard and BookRoll are not linked. In order to change educational materials, operations are required on both the dashboard and BookRoll, and using the dashboard requires extra effort. However, this is due to the design of the dashboard, which emphasizes independence as a system, and can be easily resolved technically by linking or integrating BookRoll.

The following were the most common responses to the question about the functions they would like to see in the future for the dashboard and combined learning system.

• A feature that allows users to see the specific part of the slide that the teacher explains.
• A quiz or survey function that can be used in real-time and the results shared.
• A comment or chat function that allows students to ask questions to the instructor and share their questions with other students.

These responses can be deemed requests for the functionality of existing LMS and web conferencing systems. In other words, students do not want distributed functions for learning support but rather that they be provided in a single place. We integrated the dashboard and BookRoll into a single tool using CLE, and we consider that this direction of integrating them was indirectly supported by the students.

5.1.4. Reasons for Not Using the Dashboard

Table 5. Student reasons for abandoning the dashboard.

Questions		Evaluation
		1	2	3	4	5
Q1	The information on the dashboard was not useful for my own learning.	0	3	7	5	2
Q2	I did not have time to use the dashboard during the lecture.	0	4	4	8	1
Q3	I wanted to concentrate on the lecture.	1	1	3	8	4

Notes: (N = 17); 1: strongly disagree; 2: disagree; 3: neither agree nor disagree; 4: agree; 5: strongly agree.

and

Table 6. Student reasons for never using the dashboard.

Questions		Evaluation
		1	2	3	4	5
Q1	I had no interest in or concern for the dashboard.	3	3	6	9	8
Q2	I felt that the information available on the dashboard was not necessary for my learning.	3	7	11	5	3
Q3	I did not have time to use the dashboard during the lecture.	0	2	12	8	7
Q4	I wanted to concentrate on the lecture.	1	2	13	8	5

Notes: (N = 29); 1: strongly disagree; 2: disagree; 3: neither agree nor disagree; 4: agree; 5: strongly agree.

show the results for students’ responses as to why they stopped using ($N=17$) or did not use the dashboard at all ($N=29$). The most frequently selected reasons were “I wanted to concentrate on the lecture” and “I did not have time to use the dashboard during the lecture”. This suggests that the use of the dashboard was perceived as burdensome by some students. Indeed, it is not easy to monitor the dashboard while listening to the teacher’s explanation during a lecture. Therefore, it is important to have a mechanism that prompts learners to use the dashboard at appropriate times so that they do not have to constantly check it. However, to realize such a notification mechanism, it would be necessary to determine the timing based on an analysis of the teacher’s teaching situation and the learning situation of each individual learner, and, for this, more advanced activity log collection and analysis methods would be needed.

5.2. Impact on Learning

5.2.1. Learning Outcomes

Table 7. Comparison of the average total quiz scores of dashboard users and non-users in each class. A dashboard user is defined as a student who had the same e-textbook open on the dashboard as the teacher for more than half of the teacher’s explanation time throughout all two lectures.

Course	Dashboard Users			Non-Dashboard Users			t-Value	p-Value
	N	Avg.	Std. Dev.	N	Avg.	Std. Dev.
A	56	14.61	3.18	93	14.75	4.00	0.230	0.818 ${}^{*}$
B	36	12.97	2.69	99	11.81	3.52	−1.774	0.078 *
C	56	13.11	4.43	39	12.77	3.21	−0.403	0.688 *
D	-	-	-	99	13.33	3.14	-	-
E	-	-	-	127	14.04	4.03	-	-

Notes: * p < 0.1.

shows the quiz results for each course. In the table, the average of the total quiz scores is compared between the groups of students who utilized the dashboard during the lecture (dashboard users) and those who did not (non-dashboard users). We defined a dashboard user as a student who had the same e-textbook open on the dashboard as the teacher for more than half of the teacher’s explanation time throughout both lectures. As described earlier, the dashboard does not appear by default when students open the combined learning system; they must explicitly launch it. Hence, even if a student has the combined learning system open and is using BookRoll to participate in class, the dashboard is not necessarily open as well. From the table, we can see there was no significant difference in the cases of courses A and C. However, in course B, students who used the dashboard tended to have slightly higher total scores than those who did not.

Table 8 presents a detailed breakdown by e-textbooks, displaying the difference between the average total score of dashboard users and non-dashboard users as relative percentages for comparison. Specifically, letting ${\mathit{score}}_{\mathrm{user}}$ and ${\mathit{score}}_{\mathrm{non}-\mathrm{user}}$ be the average total score of dashboard users and non-dashboard users, respectively, the percentage was calculated as $100({\mathit{score}}_{\mathrm{user}}-{\mathit{score}}_{\mathrm{non}-\mathrm{user}})/{\mathit{score}}_{\mathrm{user}}$. The most significant differences were evident for the combination of Course B with materials D-2, D-3, and D-6, and for the combination of Course C with materials D-5. In these cases, the mean scores of students who utilized the dashboard were higher than those who did not. In other cases, students using the dashboard may have lower scores, but in most cases, the difference is not that great.

These results suggest that there is little negative impact of dashboard use, leading to lower scores. On the other hand, the fact that the use of the dashboard led to relatively high scores in some cases suggests that the use of the dashboard to monitor other learners’ learning situations may have a positive impact on learning.

Table 8. The detailed breakdown of Table 7 by e-textbooks. The difference between the average total score of dashboard users and non-dashboard users in relative percentages are shown for comparison.

Course	D-1			D-2
	User	Non-User	Diff.%	User	Non-User	Diff.%
A	4.21	4.41	−4.75	3.66	3.76	−2.73
	(1.14)	(1.16)		(1.05)	(1.32)
B	3.75	3.8	−1.33	3.58	3.19	10.89
	(1.00)	(1.20)		(1.11)	(1.32)
C
D		4.14			3.71
		(1.07)			(0.99)
E		4.24			3.69
		(1.26)			(1.29)
Course	D-3			D-4
	User	Non-User	Diff.%	User	Non-User	Diff.%
A				3.96	3.72	6.06
				(1.29)	(1.56)
B	3.28	2.68	18.29
	(1.49)	(1.55)
C	3.57	3.69	−3.36	3.04	3.13	−2.96
	(1.45)	(1.26)		(1.40)	(1.28)
D					3.12
					(1.28)
E					3.4
					(1.44)
Course	D-5			D-6
	User	Non-User	Diff.%	User	Non-User	Diff.%
A				2.77	2.86	−3.25
				(1.24)	(1.64)
B				2.36	2.15	8.90
				(1.31)	(1.40)
C	4.05	3.56	12.10	2.45	2.38	2.86
	(1.43)	(1.21)		(1.82)	(1.33)
D					2.36
					(1.37)
E					2.7
					(1.71)

Table 8. The detailed breakdown of Table 7 by e-textbooks. The difference between the average total score of dashboard users and non-dashboard users in relative percentages are shown for comparison.

Course	D-1			D-2
	User	Non-User	Diff.%	User	Non-User	Diff.%
A	4.21	4.41	−4.75	3.66	3.76	−2.73
	(1.14)	(1.16)		(1.05)	(1.32)
B	3.75	3.8	−1.33	3.58	3.19	10.89
	(1.00)	(1.20)		(1.11)	(1.32)
C
D		4.14			3.71
		(1.07)			(0.99)
E		4.24			3.69
		(1.26)			(1.29)
Course	D-3			D-4
	User	Non-User	Diff.%	User	Non-User	Diff.%
A				3.96	3.72	6.06
				(1.29)	(1.56)
B	3.28	2.68	18.29
	(1.49)	(1.55)
C	3.57	3.69	−3.36	3.04	3.13	−2.96
	(1.45)	(1.26)		(1.40)	(1.28)
D					3.12
					(1.28)
E					3.4
					(1.44)
Course	D-5			D-6
	User	Non-User	Diff.%	User	Non-User	Diff.%
A				2.77	2.86	−3.25
				(1.24)	(1.64)
B				2.36	2.15	8.90
				(1.31)	(1.40)
C	4.05	3.56	12.10	2.45	2.38	2.86
	(1.43)	(1.21)		(1.82)	(1.33)
D					2.36
					(1.37)
E					2.7
					(1.71)

5.2.2. Learning Activities

Learners’ use of the dashboard requires them to devote time to other activities besides listening to the teacher’s explanations; the introduction of dashboards may thus interfere with learners’ other learning activities. Therefore, we evaluate the negative impact on learning activities in terms of the activity of following the teacher’s explanations.

Using the activity log on BookRoll, we define a student as following the teacher’s explanation if he or she opens the same page within a certain tolerance time after the teacher opens a new page. Figure 6 shows the fraction of students who are judged to be following under various tolerance times, grouping students according to the course and dashboard use. In the chart, the filled bars represent groups of dashboard users, and the patterned ones represent groups of non-users. As can be seen from the figure, the follow-up ratios of dashboard users are not bad but rather higher than the groups of non-users.

6. Discussion

• RQ1: What is the effective representation of activities on the e-textbook in the real-time LAD?

According to Table 4, of the eight proposed feedback types, Following Status (described in Section 3.1.3), Browsing Heatmap (described in Section 3.1.2), and Comprehension Status (described in Section 3.1.4) were the feedback types that students rated most useful. This means that nearly half of our proposed feedback types were recognized by the learners as useful. The average scores for the other feedback types were also above 3, indicating that a few people appreciated the usefulness of these feedback types. It was found that the overall dashboard was found to have an appropriate visual representation of the information and was easily understood, as shown in Table 3. The information presented was also rated as useful for their own learning. From comments in the open-ended responses to the questionnaire, participants indicated that they were able to experience a sense of togetherness, objectively look at their own learning, and grasp important points. On the other hand, the fact that some students did not use the dashboard indicates that not all students necessarily want other learners’ learning status to be continuously shared. Therefore, it is important that the dashboard provide a variety of means of providing information so that learners can choose the appropriate method according to their needs. For example, there could be a method that provides information quickly in response to student requests or that notifies them only of important information at appropriate times. In summary, this study proposed several feedback types and successfully identified the most effective of them in terms of their subjective usefulness to students.

• RQ2: How does live sharing of learning activities affect students’ learning in online classes?

The analysis of quiz scores suggested that the use of the dashboard may enhance learning. Scores may drop, if at all, only slightly; on the whole, however, scores may notably increase. In addition, the dashboard not only did not prevent students from following the class but increased the follow-up ratio. As mentioned earlier, the free writing responses to the questionnaire indicated that the dashboard enabled the students to be objective about their own learning and to grasp important points, suggesting that the dashboard had a positive impact on their learning in these aspects as well. In summary, this study examined the impact of dashboards on learners’ learning through an analysis of differences in quiz scores and class following ratio, and it confirmed that there was a positive impact. In addition, although quantitative analysis was not possible in this study, the results suggest that a dashboard may enable learners to objectively view their own learning and grasp important content, and it would be beneficial to examine these effects in future research. The study also found that it is important to introduce a system that makes it easier for students to follow the lessons in order to prevent dropout due to the use of the dashboard.

7. Limitations

In the experiment of this study, the use of the dashboard was not required for all students. Considering the reasons given for not using the dashboard, it is possible that students who used the dashboard were able to easily understand the course content and could afford to use the dashboard. Therefore, it is unclear what impact requiring the use of the dashboard would have on learners who need great effort to understand the course content. Therefore, the results of this study only concern students who used the dashboard voluntarily.

Although this study focuses on activities in the e-textbook system, the experiment only concerns the use of presentation slides as lecture materials. In the case of text-based lecture materials, the amount of information per page, layout, etc., are very different. Therefore, feedback types such as the Browsing Heatmap provide only rough information and may be less useful. On the other hand, feedback types such as Word Ranking may be more useful because the materials are text-based. Therefore, the findings of this study are limited to lecture materials in slide format.

The findings of this study were not obtained through long-term application. This study was conducted over only two 90-min lectures, and it is likely that users are not fully familiar with dashboards and dashboard-based learning. Therefore, the potential use of dashboards in their own learning would not have been fully examined, and the long-term impact of dashboards on learning activities may differ significantly from the findings of this study. Therefore, the findings of this study are only applicable to short-term use.

8. Conclusions

In this study, we addressed the problem that it is difficult for learners to grasp the learning status of other learners in online classes. We focused on the learning analytics dashboard (LAD) as a means of informing learners about the learning activities of others in real-time and on a continuous basis and proposed a LAD specifically for activities on an electronic textbook system, which has rarely been investigated. We verified in actual online classes what constitutes an effective representation of learning activities and how learning status sharing affects learners’ learning. Of the eight feedback types provided by our LAD, Following Status (described in Section 3.1.3), Browsing Heatmap (described in Section 3.1.2), and Comprehension Status (described in Section 3.1.4) were found to be the most useful displays. As for the impact of LAD use on learning, it was found to have a positive impact on quiz scores and the class following ratio. Currently, there are almost no dashboards focused on e-textbook activities, so this paper is the first to report on a useful real-time feedback representation of e-textbook use, especially in online classes. The findings contribute to sustainable learning as well as enhanced learning by maintaining a way for students to interact with each other even under special circumstances such as COVID-19.

Future studies should conduct experiments long enough for students to become proficient with the dashboard. Although our findings were obtained from a short experimental period of only two weeks, we were able to confirm positive effects. Another important future task will be to devise a less burdensome information delivery mechanism that is not restricted to the dashboard format so that more learners can make use of the information provided in a limited amount of time.