The Critical Factors of Student Performance in MOOCs for Sustainable Education: A Case of Chinese Universities

Abstract

COVID-19 has drastically changed human life, both in the current era and, no doubt, in the years to come. One of the hot topics during this period is whether and how COVID-19 will change education patterns in the next few years. This study aimed to investigate the relationship between massive open online courses (MOOCs) and students’ learning performance. To that end, this study combined the gamification concept with the modified information systems (IS) success model and the expectation–confirmation model (ECM) for building a research model. We surveyed 586 students through an online survey and tested the hypotheses. The research results indicated that MOOCs’ information quality (IQ), system quality (SQ), and service quality (SEQ) had positive influences on the confirmation, which then affected MOOCs’ usefulness, satisfaction, and gamification. Students’ continued usage intention and course performance were positively influenced by usefulness and gamification. Students’ continued usage intention of MOOCs also positively affected course performance. The relationship between gamification and satisfaction, however, was not proved. The research results also showed that gamification, which was composed of entertainment, challenge, and social interaction, played a critical role in improving students’ continued usage intention of MOOCs and their course performance.

1. Introduction

The outbreak of COVID-19 spread worldwide at an unprecedented rate, becoming the first global pandemic over the past 100 years, causing severe harm and threatening human lives. However, the impacts and damages of the pandemic on the economy, politics, culture, education, and ideology remain unassessed. At the beginning of 2020, for the first time in history, worldwide universities closed offline classes as an emergency measure to prevent the increase in the virus infection rate. Thanks to the massive open online course (MOOC), hundreds of millions of students worldwide can continue their studies. Universities could work in a usual fashion during the lockdown period by providing temporary access to learning on MOOC platforms. During the pandemic, Chinese universities had to organize teaching and learning online. In December of 2020, Tsinghua University and the UNESCO Institute for Information Technologies in Education co-hosted the Global MOOC Conference. The minister of the Ministry of Education of the People’s Republic of China (MOE) pointed out that MOOCs in China started in 2013, after 7 years of rapid development, with more than 32,000 courses online. The number of courses and the application scale currently rank first among countries in the world. Leading Chinese universities created and offered MOOCs, and students from other universities could enroll in these courses. Teachers evaluated available courses and chose the most appropriate course for students first. Typically, each course lasts 11–14 weeks. Students must pass quizzes and a final exam to receive a certificate from the MOOC provider (e.g., https://www.icourse163.org/ accessed on 21 June 2021) and eventually obtain credit from their university. Teachers usually provide extra instruction and consultation as a supplement to ensure students’ learning outcomes.

Education institutions have made tremendous efforts, and teachers have created online courses and delivered them to students through the Internet. The “virtual semester” has started in many universities, and various classes are launched and provided online in a well-organized manner. In the field of education, many countries and regions, including China, have accumulated experience in the past efforts and exploration to combat COVID-19 outbreaks, but many unprecedented problems, meanwhile, have been revealed. Humans are bound to coexist with COVID-19 for a long time to come. In order to restore and rebuild the ecosystem of education, maintain the vitality of the education system, and promote the healthy and sustainable development of education in the post-pandemic era, it is not only necessary to reflect on and examine the experience accumulated in the fight against COVID-19 but also to conduct systematic and comprehensive plans from the perspective of education sustainability in the future. As an important mode of online education, MOOC gradually occupies an important position in traditional education, has its own unique advantages, and tends to take over the sole dominance of traditional education. MOOC has emerged from the rapid development of technology and the urgent need for education [1]. The MOOC’s distinguishing feature is its convenience and extensive sharing, making it a tool for improving the quality of education with high efficiency and proper education at a low cost. In addition to recorded lectures and assignments, many MOOCs provide tools that enable students to interact with the teachers and communicate with online classmates through social media and online communities. MOOCs can also provide immediate feedback, online examinations, and assignments. Interaction between students and professors can occur within an online community through synchronous and asynchronous online learning activities, and students’ social, interactive, and collaborative skills will improve [2]. More than this, students can select their schedule for studying with the assistance of their teacher. After COVID-19, MOOC has been widely discussed in academic and practical areas as an innovative tool for sustainable education.

There is a debate because people wonder whether online classes offered by schools will be attractive to students. With the outbreak of COVID-19, most of our daily activities have been transferred from offline to online. Many educational institutions worldwide provide MOOCs, including edX, Moodle, CourseSites, Udemy, Versal, etc. According to Class Central, a search engine and reviews site for free online courses popularly known as MOOCs, by 2019, MOOCs had reached 110 million learners. Most importantly, the top three MOOC platforms (Coursera, edX, and FutureLearn) registered as many new learners in April 2020 as in the whole of 2019. Worldwide platforms provide more than 2500 courses, 11 online degrees, and 170 micro-credentials. The use of MOOCs during COVID-19 has allowed educational institutions to ensure teaching work runs smoothly on the rails. It enables students to have greater autonomy in preparing and conducting their learning. MOOCs differ significantly in terms of their purposes and the mechanisms that power them. Main categories of MOOCs usually include: First, connectivist MOOCs (cMOOCs) are a relatively loose online community for learners, mainly focusing on a shared learning interest. cMOOCs, on the other hand, use various social media tools and personal interaction tools for learning and sharing knowledge among the participants. Second, eXtended MOOCs (xMOOCs) are additional supplementary courses at universities. Third, cooperative MOOCs often combine characteristics of both xMOOCs and cMOOCs. Lastly, gamification cooperative MOOCs (gcMOOCs) [3,4] add gamification designs to the cooperative MOOCs.

The focus of previous research on MOOCs has been varied. Some key topics include barriers and intentions for using MOOCs [5], MOOC scholarship [6], challenges and motivation for teachers and students [7], and engagement and retention [8]. Lee et al. [9] discussed MOOCs from the perspective of self-regulated learning. Furthermore, many studies have discussed the ECM in explaining and exploring learners’ continued usage intentions. For instance, Dai et al. [1] extended the ECM and proposed a theoretical model including cognitive and affective variables to discuss and explain users’ intentions to stay in MOOC learning. Zhou [10] added “social influence” as a new construct to the original ECM in the MOOCs’ learning context.

Gamification is a prevalent concept in mobile and technological systems that adopts a game mechanism to facilitate desired behaviors and bring positive performance outcomes. It has been proven that gamification in MOOCs positively influences students’ motivation [11]. Among the diverse research theories that were used for discussing MOOCs, the gamification theory was a notable one. Antonaci et al. [12] provided a gamification design process with six phases. In designing gamified MOOCs, learner, social, openness, course design, and technology were considered. Antonaci et al. [13] summarized the effects of game elements on learners’ behavior in an online learning environment, such as performance, motivation, engagement, collaboration, and social awareness. Dominguez et al. [14] and Dicheva et al. [15] indicated that gamification of online learning could increase users’ motivation for studying if adequately designed and implemented. Thus, gamification is a favorable mechanism that can motivate users of information systems. Malone [16] emphasized the concept of gamification. This study focused on stimulating more people to consider game-based elements in education because game mechanisms influence users’ learning motivation and engagement. As gamification gains increasing attention in education research, many scholars have emphasized the benefits that gamified designs bring to learners. Their findings provide breakthroughs for future studies in online learning and gamification research [17,18,19]. In order to maintain and increase students’ interest and engagement, the gamified design of MOOCs should include emotional incentives by giving students challenging cases, entertaining functions, and social interaction opportunities. Although those studies noted that gamification affected students’ positive experience with the MOOC, to the best knowledge of the authors, no study has been arranged to examine the relationship between gamification and overall effects of MOOCs. To address this research gap, the current study adopted an integrated framework based on the IS success model, expectation–confirmation model (ECM), and gamification concept to investigate the relationship between MOOCs and students’ course performance empirically. For the analysis and research hypothesis tests, we used covariance-based structural equation modeling (CB-SEM). This paper is organized as follows. The description of MOOC and the theoretical basis of this study are systematically reviewed and arranged in Section 2.1, Section 2.2, Section 2.3, Section 2.4 and Section 2.5. Then we propose a research model and hypotheses in Section 2.6. Next, we describe the data collection and research methods in Section 2.7 and Section 2.8, followed by data analysis results in Section 3. Finally, we discuss the major findings, implications, and limitations in the final part of the paper.

2. Materials and Methods

2.1. MOOCs (Massive Open Online Courses)

The media used in MOOCs is open to the public. Learners are encouraged to share and contribute materials. Course modules are often 5–10 min, and course content is edited when needed. As to traditional online courses, most institutions have their own online course platform and restrict the media. Students search for courses within an institution’s library database. Course modules are typically 45–60 min long. In MOOCs, lectures are usually pre-recorded, and courses are open-ended with no due dates. As for traditional online courses, in addition to recorded lectures, the instructors sometimes conduct live lectures. Courses frequently end on the due date. The learners of MOOC include not only school students but also on-job students. As with traditional online courses, learners are often students of the institution. In addition, unlike online courses, which focus more on content, MOOCs focus more on context.

Wu and Chen [20] suggested that MOOCs are attractive to students who have strong motivation and who hold the perception that MOOCs are useful for individual users. To further examine and discuss the factors influencing students’ perceptions and intentions to use MOOCs, many studies have paid attention to the antecedents of learners’ behavioral intentions from various perspectives [5,10,21,22]. Although past research has successfully revealed various valuable and interesting implications and provided views that support the importance of an actively engaged MOOC, it is necessary to investigate MOOCs’ effect on students’ continued usage intentions and course performance.

2.2. Overview of a Chinese MOOC Platform

In MOOCs, each course runs periodically, and the entire learning process consists of multiple sessions: attending lectures, participating in discussions, submitting assignments, pop quizzes, and a final exam. Courses are managed and operated by the Academic Affairs Office of each university. Universities create courses and designate the teachers to manage the courses.

Usually, there are 9 sections involved in one course, from course selection to certificate issuance, as shown in Figure 1.

In addition, students should finish authentication first so that teachers can manage the students in an organized manner, as shown in Figure 2:

The design and structure of a course include 4 parts: a bulletin board, courseware (video and PPT files), assignments and quizzes, and a discussion board. Consider the course Financial Risk Management, as shown in Figure 3.

The first figure on the left shows the bulletin board of Financial Risk Management, which includes the course outline, description of each class, and final exam description. The second figure presents the courseware of the course, where students can find lecture videos from chapter 1 to the last chapter and obtain PPT and/or PDF files. The third figure indicates the assignments and quizzes of the course. Students should finish and pass these assignments and quizzes before the deadline. The last figure demonstrates the discussion board of the course. The discussion board is divided into three sections: general discussion, Q&A, and the Learning Exchange Zone. On the discussion board, students can communicate with the teachers and their classmates.

2.3. IS Success Model

The implementation success of an information system can be influenced by a lot of factors, such as environmental factors, system factors, end-users’ characteristics, etc. [23] DeLone and McLean [24,25] proposed the IS success model, an integrated model that comprehensively explains IS success. The model presents, identifies, describes, examines, and explores the relationships between the essential dimensions. For decades, research of information systems and technology has adopted the IS success model. From 1995 to 2019, various studies have applied the IS success model and provided beneficial results [16,26,27,28,29,30].

In online education, well-established research models and theories such as the technology acceptance model (TAM), the theory of reasoned action (TRA), the theory of planned behavior (TPB), and the unified theory of acceptance and use of technology (UTAUT) have been discussed so far. Rogers’ [31] innovation diffusion theory (IDT) was also a remarkable one. In addition, some researchers examined students’ behaviors in online education by employing the IS success model. For example, Freeze et al. [32] and Aparicio et al. [33] investigated e-learning system success using the IS success model. Cidral et al. [34] proposed a research model integrating the IS success model and success factors in e-learning systems. Albelbisi [35] verified the role of quality factors in self-regulated learning in MOOCs, and Albelbisi [36] also developed and validated MOOCs’ success factors using Malaysian samples.

Information quality (IQ), system quality (SQ), and service quality (SEQ) are three independent variables in the IS success model. IQ deals with the supplied information from the information system with accuracy, correlation, and conformance. SQ usually refers to the usefulness, stability, function, reliability, complexity, compatibility, and demonstrability of the IS. SEQ discusses whether information systems provide assured, responsive, and empathetic service. Most previous studies usually included IQ, SQ, SEQ, intention to use, user satisfaction, and net benefits when researching information technology and systems [37,38,39,40,41]. MOOC platform provides content courses, a learning environment, and services for potential learners through the Internet. IQ of the MOOC platform refers to information provided on the MOOC platform having good quality when contents are perceived as useful, reliable, complete, and understandable. SQ of the MOOC platform refers to good system technical performance, accessibility, availability, and usability. SEQ of the MOOC platform refers to the responsiveness, competence, and empathy of the MOOC platform service representatives. From previous studies using these IS success variables (IQ, SQ, SEQ) [39,41,42], we presume that this model is often adopted in a variety of fields since it provides a comprehensive understanding of the context in which the users use the system. Hence, it is also important to explore and examine how these factors influence success on MOOC platforms. MOOC platforms are attracting attention as a new online education tool, and it is important to keep these platforms available to students. This study focused on improving students’ course performance by continuously using the MOOCs platform. We proposed that the information system success model can be adopted since it is useful to measure how students perceive the usefulness and gamified designs of a MOOC platform, students’ satisfaction with a MOOC platform, their continued use intention toward a MOOC platform, and their learning performance resulting from the involvement. MOOCs presuppose the importance of their content, scheme, and instant service as a representative online education tool. These attributes have been neglected in the evaluation of MOOCs. As a result, it is critical to explore and explain MOOCs’ qualities, which may significantly influence learners’ perceptions and behaviors.

2.4. Expectation–Confirmation Model

When considering post-purchase or post-adoption satisfaction, previous studies have used various constructs. Some representative antecedents of continued usage intentions include perceived usefulness, satisfaction, perceived ease of use, trust, etc. Many studies have discussed users’ continued usage intentions of information technologies through the expectation–confirmation model (ECM). Hsu and Lin [43], Leung and Chen [44], and Gupta et al. [45] investigated the relationship between pre-adoption experiences and post-adoption continued usage intentions in the field of paid mobile apps and e-health technology. Wu et al. [40] explored online impulse buying through the expectation–confirmation model, an epistemic research model that explores post-adoption satisfaction and continued usage of the information system. Bhattacherjee [46] demonstrated that expectations formed by accepting a specific information system would lead to confirmation by comparing the characteristics and performance. When users confirm the performance, their satisfaction will be followed. Thus, in turn, continued usage intentions will be induced.

2.5. Gamification

In IS research, gamification is an emerging concept that uses gamified designs and non-gamified technologies to make tasks more engaging. Gamification has a profound impact on many different types of information systems, both in theory and action. British-born computer programmer and inventor Nick Pelling initially proposed the concept of gamification in 2002. It started to gain attention and hit mainstream academic circles around 2010. Research in the health system, crowdsourcing, education, etc., has been applied to gamification. One of the predominant problems of MOOCs is students’ low retention rates and completion rates [47,48]. This phenomenon emphasizes the importance of introducing a gamification mechanism to increase the retention and completion rate. Levels, badges, leaderboards, points, feedback, challenges, likes, and social features, communication channels, narratives, etc., are the most widely used gamified elements in MOOCs [13,33]. Figure 4 shows a “Lucky Card” game provided by a MOOC provider during exam week, and Figure 5 shows a badge.

The Lucky Card game is a “Will Not Fall” game with a lucky meaning. By engaging this game, students can draw different cards randomly. Each card represents a “weapon” for the final examination, such as the “memory bread” card, which means you will be able to memorize all the contents of the exam. “Energy milk” card means you will be full of energy. “Not fall spray” card means you will outdo yourself in the exam. These cards are supposed to be lucky in Chinese culture. Furthermore, students can share this game with their friends on social networks and have various discussions together. These entertaining games can help students reduce the pressure of studying psychologically. In addition to that, students will be rewarded (points gained) when they complete predefined activities of a weekly mission, such as watching video lectures and finishing quizzes. The digital badges are also incentives to motivate students to continue learning.

Dominguez et al. [14] indicated the benefits and consequences of gamified elements and designs in an e-learning platform, and Dicheva et al. [15] prepared systematic mapping research to discuss gamified elements in education. An et al. [49] found that learners had various attitudes concerning the gamification of MOOCs. Academics and industry are becoming more interested in the gamification of MOOCs. What influence does gamified design have on MOOCs? Among the effects, the most remarkable one must be an increase in participation. When students participate in the discussion forum, they have an opportunity to gain “likes” from classmates and teachers. This is a way to increase students’ participation. Furthermore, group interactions are available for students as a tool for getting feedback and making contributions. In addition, time pressure is another gamified element for encouraging students’ participation. Time pressure means MOOC platforms give students a time limit to finish a certain activity to push them to complete a required task. For students, creating time pressure can encourage them to participate in learning positively because the time pressure is connected to their learning goals (e.g., finishing the current learning stage and moving to a higher level). By adding the gamification concept to MOOCs, students’ completion rate will increase, and students will be willing to spend more time on the MOOCs. Ortega-Arranz al. [19] and Khalil et al. [50] showed that students’ retention increased by employing gamification, and their motivation and engagement were improved. Saputro et al. [51] found that students’ intrinsic motivation in MOOC increased under the framework of fulfilling rules and suitability of the instruments and game elements. The current study tried to add the gamification concept to the IS success model and expectation–confirmation model framework based on the previous studies and findings.

2.6. Research Model and Hypotheses

2.6.1. MOOC Qualities and User Confirmation

DeLone and McLean [24] proposed a research model explaining information system success. The original research model includes six constructs, system quality, information quality, use, user satisfaction, and individual and organizational impact. The updated IS success model was proposed by them later in 2003. The updated model [25] added a new construct, service quality, to the original model. In addition, the “use” in the basic model was divided into two separate variables: “intention to use” and “use” for measuring and explaining IS success. In online education, Hassanzadeh et al. [52], and Marjanovic et al. [53] applied the IS success model for exploring the e-learning platform. Lin [54] indicated that the platform’s SQ, IQ, and SEQ significantly influenced actual online learning platform use through users’ satisfaction and intention to use. The ECM is built on consumer behavior studies and integrates theoretical and empirical findings from previous IS literature. Bhattacherjee [55] suggested that users’ continued usage intention (CI) is influenced by their satisfaction with the IS and users’ perception of its usefulness. The users’ dissatisfaction will result in the discontinuance of the IS if performance disconfirms the expectations.

By using MOOCs and evaluating their performance, learners constitute perceptions in their minds and compare MOOCs with other learning tools, corresponding to their expectations. The qualities of MOOCs are connected with potential users’ expectations of MOOCs. The higher the users’ perceived qualities of the MOOCs and confirmed expectations, the more positive perceptions and satisfaction will be induced. The study of Roca et al. [56] proved that the IQ, SQ, and SEQ of an e-learning system positively affected expectation–confirmation of the e-learning system.

Similarly, Chung et al. [57] also suggested that an information system’s confirmation was positively influenced by its IQ, SQ, and design quality (DQ). In addition, in smartphone classes, Shin et al. [58] showed that the system’s perceived content quality and perceived service quality positively influenced users’ confirmation. Based on previous studies, this paper hypothesizes that MOOCs’ information quality (IQ), system quality (SQ), and service quality (SEQ) have positive influence on MOOCs’ quality confirmation:

Hypothesis 1 (H1).

Information quality of the MOOC positively affects students’ confirmation of MOOC qualities.

Hypothesis 2 (H2).

System quality of the MOOC positively affects students’ confirmation of MOOC qualities.

Hypothesis 3 (H3).

Service quality of the MOOC positively affects students’ confirmation of MOOC qualities.

2.6.2. User Confirmation, MOOC Platform Usefulness, User Satisfaction, and Gamification

Bhattacherjee [46] showed that users’ confirmation of their original expectations is a critical predictor of users’ usefulness perception formation. Information system confirmation deals with the degree to which the information systems satisfy the user’s expectations. The perceived usefulness of an information system implies the user’s expectation of that information system, influencing users’ post-adoption behavior. To be more specific, if any user confirms his/her expectations, their perceived usefulness will be formed. In addition, ECM proposed that users’ satisfaction with an information system is induced by their confirmation of experience and the perceived usefulness of the system. Previous studies have proved the relationships between users’ confirmation, perceived usefulness, and users’ satisfaction [1,45,59,60]. Pozón-López et al. [61] showed that satisfaction is affected by MOOC quality and other qualities such as usefulness. In MOOCs, if students find that the actual performance is better than their original expectations, their confirmation of the platform will be positive. In turn, the positive confirmation will result in a higher level of user satisfaction. On the contrary, if students believe that the system’s performance falls short of their expectations, their confirmation will be negative. Consequently, the negative confirmation will lead to a lower level of user satisfaction in the subsequent stages. According to Bhattacherjee and Lin [55], user satisfaction is an emotional evaluation of an information system based on previous experience.

In addition to the above constructs, gamification is a new critical component of the information system. Although gamified elements’ influences have been explored and discussed in several fields, studies on gamification in MOOCs are still limited. Some researchers have recently addressed the relationship between gamification and expected benefits [2,51,62,63]. Extrinsic motivation could be considered and explained in terms of gamified elements in MOOCs. While learning, learners engage in gamified MOOCs to get points and obtain badges and other prizes. Recently, studies discussing the effectiveness of the gamified design of the online educational system have been growing. However, it is necessary and valuable to conduct empirical studies to explore the relationship between gamification (e.g., badges, points, forum participation, “likes”, time pressure, social networking interaction, etc.) and learners’ behaviors and learning performance. Students’ perceptions of gamified elements of the MOOCs are their expectations of the system, and the expectations will influence their post-adoption behaviors. If students’ expectations can be confirmed, the perception of the gamified design will be formed as well. Based on previous gamification literature, we hypothesized that:

Hypothesis 4 (H4).

Students’ confirmation of MOOC qualities positively affects MOOC usefulness.

Hypothesis 5 (H5).

Students’ confirmation of MOOC qualities positively affects MOOC gamification perceptions.

Hypothesis 6 (H6).

Students’ confirmation of MOOC qualities positively affects their satisfaction with the MOOC.

Hypothesis 7 (H7).

MOOC usefulness positively affects students’ satisfaction with the MOOC.

Hypothesis 8 (H8).

MOOC gamification positively affects students’ satisfaction with the MOOC.

2.6.3. MOOC Usefulness, Satisfaction, Gamification, and Continuance Intention

In MOOCs, the platform’s perceived usefulness deals with the degree to which MOOCs can be useful and improve students’ learning. Ajzen [64] believed that users’ subjective perceptions influence their behaviors, even though the perceptions they hold are tendentious or incorrect. As a result, instead of an objective assessment, the perceived feeling is supposed to be relevant. According to Bhattacherjee [46], users’ satisfaction with an information system is an active, neutral, or passive feeling. Expectation–confirmation studies state that users’ satisfaction with an information system is a prerequisite for building users’ continued usage intentions. Bhattacherjee [46], Joo et al. [65], Li and Fang [66], and Bölen and Özen [67] proved the positive relationship between perceived usefulness and IS continued usage intention, as well as satisfaction and IS continued usage intention. Hence, in MOOCs, students are expected to use MOOCs continuously if satisfied with the platform. On the other hand, the direct relationship between gamification and users’ continued usage intentions has not been empirically verified yet. However, several studies have proved that users’ benefits and continued usage will increase in a gamified environment [68,69]. Hence, we hypothesized that students’ perceptions of MOOCs’ gamified design will positively influence their continued usage intentions in the MOOC context.

Hypothesis 9 (H9).

MOOC usefulness positively affects students’ continued usage intention of MOOC.

Hypothesis 10 (H10).

MOOC gamification positively affects students’ continued usage intention of MOOC.

Hypothesis 11 (H11).

Satisfaction with the MOOC positively affects students’ continued usage intention of MOOC.

2.6.4. Usefulness, Gamification, Continued Usage of MOOCs, and Course Performance

Course performance in MOOCs refers to the “perceived outcome” of learning through MOOCs. Specifically, it deals with the ability and possibility to learn well with higher efficiency and less effort and time. Tam and Oliveira [70] investigated individual performance in the context of m-banking. However, insufficient empirical research has studied the salient relationship between the IS’ perceived usefulness and individual performance and the IS’ continued usage intention and individual performance. Usefulness keeps ahead of user satisfaction, and users’ positive experience with usefulness will induce greater user satisfaction. Increased IS user satisfaction leads to increased IS usage intentions, which improves individual performance [71]. Increased use of MOOCs may raise student awareness of the benefits of MOOCs. In a MOOC learning context, students who adopt digital systems and have digital skills can improve their learning performance. This study investigated whether students could gain a clear understanding of the classes through using MOOCs, measured whether students could easily achieve the learning goals asserted by the course via MOOCs, and discussed if it is easier to accomplish the assignments by using MOOCs.

Gamified design is another critical element of MOOCs. Gerdenitsch et al. [72] examined how gamification affects work enjoyment, productivity, and the moderating role of leadership responsibility. Feng et al. [73] discussed whether gamification could increase consumers’ engagement with fitness apps. Groening and Binnewies [74] examined the relationship between digital achievements, motivation, and performance. The research results show that digital achievements could improve motivation and enhance performance. The findings provide significant inspiration for future IS research. MOOCs employ gamified design to induce and increase students’ engagement and participation. A gamified MOOC may change students’ study patterns and communication patterns. Thus, if students feel satisfied with the novel studying tools and the interaction patterns, their motivation and engagement will increase. In turn, students’ learning performance will be improved compared with those online education tools without gamified design. Based on previous studies, we hypothesized that:

Hypothesis 12 (H12).

MOOC usefulness positively affects students’ course performance.

Hypothesis 13 (H13).

MOOC gamification positively affects students’ course performance.

Hypothesis 14 (H14).

Students’ continued usage intention of the MOOC positively affects students’ course performance.

2.6.5. Gamification: Sociality, Entertainment, and Challenges

As discussed above, gamification is a way to increase students’ motivation and engagement and, finally, affect students’ course performance. The gamification concept has also gained attention from researchers and practitioners. Legaki et al. [75] proved that the challenge-based gamified elements in an online learning platform positively affect students’ learning compared to conventional teaching tools. The concept of gamification has been employed in many research areas. For example, Tobon et al. [76] examined the relationship between gamified features and online customer behavior, and Rodrigues et al. [77] explored gamification in the field of e-banking. Specifically, Rodrigues et al. [77] investigated how bank users face gamified smart applications, mainly focused on entertainment and ease of use elements. Their findings reveal that the system’s ease of use and entertainment had an impact on e-banking usage. Social interaction is another critical component of gamification. Fang et al. [21] stated that online learning platforms’ social interaction design positively influences users’ study focus. Students could feel a sense of entirely participating or breathtakingly engaging in the MOOCs’ learning [78]. If applied to MOOCs, social interaction may stimulate frequent use of the platform and possibly improve students’ learning performance. In addition, the social interaction design may improve the students’ MOOC user experience. Without face-to-face communication, students still maintain social connections with professors and classmates. Thus, we propose that gamification may include vibrant designs by providing students with challenges, entertaining atmosphere, and social interaction in MOOCs.

Hypothesis 15 (H15).

Gamification is a second-order construct that is comprised of socialness, challenge, and entertainment.

Based on the hypotheses described above and previous studies, this study presents a research model as shown in Figure 6.

2.7. Data Collection

This study conducted a questionnaire survey to collect data and investigate the relationship between the constructs. After completing an extensive review of IS literature, we created the survey items and translated the original ones into Chinese. Then, we translated them into English again to ensure the questionnaire’s accuracy and consistency. We distributed the survey questionnaires to universities in China and collected 629 questionnaires. All of the students attended MOOCs in the spring semester of 2020. After excluding 43 questionnaires with incomplete responses, 586 valid questionnaires were used for analysis.

The respondents’ demographic characteristics are summarized in

Table 1. Demographic distribution.

Variables	Classification	Number	Percentage
Gender	Male	209	35.7
	Female	377	64.3
Education background	Undergraduate	482	82.3
	Graduate school (M.A.)	20	3.4
	Graduate school (Ph.D. and above)	84	14.3
Experience	Less than 1 month	136	23.2
	1–3 months	127	21.7
	3–6 months	117	20
	6–9 months	64	10.9
	9–12 months	35	6
	12 months and above	107	18.3
Average using time/week	1–4 h	260	44.4
	5–9 h	151	25.8
	10–14 h	87	14.8
	15 h and above	88	15
Residency	Western	160	27.3
	Central	215	36.7
	Eastern	211	36

. Table 1 shows 209 (35.7%) males and 377 (64.3%) females. In terms of educational backgrounds, most of those who responded were college students (482, 82.3%). In terms of past MOOC user experience, 136 (23.2%) have used them for less than a month, 127 (21.7%) have learned in them for 1 to 3 months, and 107 (18.3%) have used them for more than 12 months. When it comes to the average amount of time spent on MOOCs each week, most participants (260, 44.4%) spend 1 to 4 h per week on them. Finally, 160 (27.3%) respondents came from the country’s western region, 215 (36.3%) from the country’s central region, and 211 (36%) from the country’s eastern region.

2.8. Measurement Items

The measurement items were developed through an extensive literature review (see Appendix A). The survey aimed at measuring students’ perceptions of MOOCs positively or negatively, which includes the following variables: information quality (IQ), system quality (SQ), service quality (SEQ) of the MOOC platform, students’ confirmation, perceptions of the MOOC platform’s gamified characteristics, perceived usefulness of MOOCs, students’ satisfaction with MOOCs, students’ MOOC platform continued usage intentions, and students’ course performance through learning on MOOC platforms.

This study employed a 5-point Likert scale, ranging from 1 to 5 (1 represents an inclination to strongly disagree, 5 represents strongly agree). We analyzed the collected data in 3 steps. First, outliers, normality, and missing values were checked. Second, we tested the validity and reliability of the measurement items. Finally, we examined correlations between the constructs, tested the model fit, checked common method bias, and tested research hypotheses.

3. Analysis and Results

This study analyzed the data using the covariance-based structural equation model (CB-SEM). The measurement model and structural model were evaluated, respectively.

3.1. Measurement Model

We evaluated the measurement model by checking reliability and convergent and discriminant validity. In

Table 2. Factor loading, CR, AVE, and Cronbach’s α values.

Constructs	First-Order Constructs	Item	Item Loading	CR	AVE	Cronbach’s α
MOOC platform quality confirmation	N/A	MCON1	0.814	0.838	0.632	0.837
		MCON2	0.777
		MCON3	0.794
Information quality	N/A	IQ1	0.771	0.839	0.565	0.838
		IQ2	0.731
		IQ3	0.725
		IQ4	0.779
Service quality	N/A	SEQ1	0.784	0.859	0.605	0.858
		SEQ2	0.763
		SEQ3	0.75
		SEQ4	0.811
System quality	N/A	SQ1	0.818	0.862	0.61	0.861
		SQ2	0.767
		SQ3	0.752
		SQ4	0.784
MOOC usefulness	N/A	MUSE1	0.845	0.854	0.662	0.853
		MUSE2	0.768
		MUSE3	0.825
MOOC satisfaction	N/A	MSAT1	0.799	0.813	0.592	0.809
		MSAT2	0.7
		MSAT3	0.805
Gamification perceptions	Social interaction	SOC1	0.834	0.881	0.65	0.88
		SOC2	0.754
		SOC3	0.794
		SOC4	0.839
	Challenge	CHA1	0.826	0.831	0.623	0.827
		CHA2	0.709
		CHA3	0.827
	Entertainment	ENT1	0.814	0.85	0.587	0.849
		ENT2	0.722
		ENT3	0.733
		ENT4	0.792
MOOC continued usage intention	N/A	MCI1	0.795	0.833	0.624	0.831
		MCI2	0.756
		MCI3	0.818
Course performance	N/A	CP1	0.781	0.841	0.57	0.841
		CP2	0.739
		CP3	0.739
		CP4	0.761

, all values of standardized item loadings for the indicator were ≥0.7. All of the composite reliability (CR) values were greater than 0.8. All values of Cronbach’s α for each construct were ≥0.8. All average variance extracted (AVE) values employed for assessing the convergent validity were ≥0.5. Consequently, Table 2 indicates that all values met the minimum requirement, and the measurement model has good reliability and convergent validity.

Pearson’s correlation coefficients are shown in

Table 3. Discriminant validity test.

	IQ	SQ	SEQ	MCON	MUSE	GAM	MSAT	MCI	CP
IQ	0.752
SQ	0.433 **	0.781
SEQ	0.457 **	0.418 **	0.778
MCON	0.392 **	0.338 **	0.366 **	0.795
MUSE	0.318 **	0.288 **	0.263 **	0.394 **	0.814
GAM	0.366 **	0.312 **	0.323 **	0.328 **	0.178 **	0.767
MSAT	0.353 **	0.307 **	0.325 **	0.532 **	0.514 **	0.284 **	0.769
MCI	0.356 **	0.352 **	0.364 **	0.405 **	0.435 **	0.425 **	0.501 **	0.790
CP	0.403 **	0.361 **	0.414 **	0.393 **	0.454 **	0.506 **	0.414 **	0.575 **	0.755

NOTE: ** p < 0.01. MCON = MOOC platform quality confirmation; IQ = Information quality; SQ = System quality; SEQ = Service quality; MUSE = MOOC usefulness; MSAT = MOOC satisfaction; GAM = Gamification; SOC = Social interaction; CHA = Challenge; ENT = Entertainment; MCI = MOOC continued usage intention; CP = Course performance.

. Next, we compared the square root of AVE with the variables’ correlation coefficients to assess the discriminant validity. As shown in Table 3, the AVE’s square root is significantly larger than each variable’s correlation coefficient. Consequently, the scale has good discriminant validity.

Podsakoff and Organ [79] and Podsakoff et al. [80] suggested that there were possibilities for common method variance (CMV) in the self-reported data. We applied the Harman one-factor statistical analysis to examine the common method bias in the collected data. First, we checked if the measurements were affected by common method variance through Harman’s single factor test and examined whether relationships between constructs were observed to inflate or deflate. This test includes a factor analysis of all the involved items. In addition, we performed exploratory factor analysis without rotation and checked whether all factors were returned. Previous research suggested that if one factor’s total variance exceeds 50%, it shows common method bias in the study. However, as shown in

Table 4. Common method bias test.

Component	Initial Eigen Values
1	Total	% of Variance	Cumulative %
	11.591	27.719	27.719
	Extraction Sums of Squared Loadings
	Total	% of Variance	Cumulative %
	10.881	27.899	27.899

, one factor explained only 27.899% of the total variance. Thus, we can conclude that there is no common method bias in the collected data.

Finally, we compared the recommended values and actual values of model fitness.

Table 5. Measurement model fit.

Fit Indices	χ2/df	GFI	AGFI	NFI	CFI	RMR	RMSEA
Recommended Value	<3.0	>0.9	>0.8	>0.9	>0.9	<0.08	<0.08
Value indices	1.506	0.920	0.909	0.918	0.971	0.069	0.029

shows the values of χ², degrees of freedom (df), goodness-of-fit index (GFI), adjusted goodness-of-fit index (AGFI), normed fit index (NFI), comparative fit index (CFI), parsimony goodness fit index (PGFI), root mean square residual (RMR), and root mean square of approximation (RMSEA). All actual values met the standard of recommended values, which indicated a good model fit.

3.2. Structural Model

This section examined the R² of MOOC platform confirmation, usefulness, satisfaction, continued usage intention, course performance, gamification (social interaction, entertainment, and challenge), which suggested the explanatory power of the independent variables. The path coefficients and R² showed how well the data supported the proposed research model. It is recommended that the R² value of 75%, 50%, and 25% may indicate the substantial, moderate, and weak level of the construct’s explanation power (s).

Information quality (β = 0.312 and p-value < 0.001), system quality (β = 0.172 and p-value = 0.001), and service quality (β = 0.217 and p-value < 0.001) have significant positive effects on MOOC quality confirmation. Three qualities explained MOOC quality confirmation with R² = 33.8%. Thus, Hypotheses 1, 2, and 3 are supported. MOOC quality confirmation significantly positively affected MOOC usefulness (β = 0.489 and p-value < 0.001) with R² = 23.9%. Thus, Hypothesis 4 is supported. MOOC quality confirmation significantly positively affected MOOC gamification perceptions (β = 0.463 and p-value < 0.001) with R² = 21.5%. Therefore, Hypothesis 5 is supported. In addition, MOOC quality confirmation (β = 0.453 and p-value < 0.001) and MOOC usefulness (β = 0.375 and p-value < 0.001) have significant positive effects on students’ satisfaction with MOOCs (R² = 55.3%). However, gamification did not have a statistically significant effect on satisfaction. Thus, Hypothesis 6 and Hypothesis 7 are supported, but Hypothesis 8 is not. Regarding students’ continued usage intention of the MOOCs, MOOC usefulness (β = 0.23 and p-value < 0.001), MOOC gamification perception (β = 0.365 and p-value < 0.001), and MOOC satisfaction (β = 0.335 and p-value < 0.001) have significant positive effects on students’ continued usage intention (R² = 51.8%). Therefore, Hypotheses 9, 10, and 11 are supported. In addition, MOOC usefulness (β = 0.275 and p-value < 0.001), MOOC gamification perception (β = 0.392 and p-value < 0.001), and MOOC continued usage intention (β = 0.331 and p-value < 0.001) significantly affected course performance with R² = 62.1%. As a result, Hypotheses 12, 13, and 14 are supported. Lastly, as a second-order construct, gamification explained 67.5% of the variation of social interaction, 53.3% of the challenge, and 55.3% of entertainment. To summarize, based on these findings, gamification was proven to be a reflective second-order construct of social interaction, challenge, and entertainment. Thus, Hypothesis 15 is supported. Details can be found in

Table 6. Hypothesis test results.

Hypothesis	Path			β	p-Value	R2	Remarks
H1	IQ	→	MCON	0.312	***	0.338	Supported
H2	SQ	→	MCON	0.172	0.001		Supported
H3	SEQ	→	MCON	0.217	***		Supported
H4	MCON	→	MUSE	0.489	***	0.239	Supported
H5	MCON	→	GAM	0.463	***	0.215	Supported
H6	MCON	→	MSAT	0.453	***	0.553	Supported
H7	MUSE	→	MSAT	0.375	***		Supported
H8	GAM	→	MSAT	0.062	0.189		Not supported
H9	MUSE	→	MCI	0.23	***	0.518	Supported
H10	GAM	→	MCI	0.365	***		Supported
H11	MSAT	→	MCI	0.335	***		Supported
H12	MUSE	→	CP	0.275	***	0.621	Supported
H13	GAM	→	CP	0.392	***		Supported
H14	MCI	→	CP	0.331	***		Supported
H15	GAM	→	SOC	0.675	***	0.675	Supported
		→	CHA	0.553	***	0.533
		→	ENT	0.533	***	0.553

NOTE: *** p < 0.001.

and Figure 7.

4. Discussion

Campus closure is an important intervention that helps safeguard public health and respond to the spread of epidemic infections. According to the Policy Brief: Education during COVID-19 and beyond provided by the United Nations, the COVID-19 pandemic has created the largest disruption of education systems in history, affecting nearly 1.6 billion learners in more than 190 countries all continents. Closures of campuses and other learning spaces have impacted 94% of the world’s student population, up to 99% in under-developed countries. The sudden outbreak of COVID-19 triggered a global education crisis and, at the same time, forced governments and schools to take the initiative to change education. In response to the ongoing situation of COVID-19, many countries have adopted a variety of flexible teaching and learning approaches in their education systems, among which MOOC is one of the main approaches. The MOOC, as a subset of distance education, has been focused on providing a more flexible and innovative educational experience in terms of time and space than school education by utilizing various types of technology. During the epidemic prevention and lockdown period, many countries closed schools and changed to MOOC mode and thereby accelerated the process of informatization and modernization of education.

The COVID-19 epidemic will not be eliminated for a long period in the future. Human beings are forced to move into and survive in the post-pandemic era with the coronavirus. The way of production, living, and survival also needs to be changed accordingly. Education is crucial to the sustainable development of human society. We have a mission and a task to adapt to coexist with the epidemic, work toward its improvement, and build a better world for future generations. Education in the digital information era is critical to the future development of the quality of human life and the world’s sustainability. New digital technologies in education have far-reaching effects on formal education and informal learning regarding educational goals and objectives, educational ecology and the environment, learning processes, teaching and learning processes, and educational governance and policies.

We surveyed university students who had taken massive open online courses (MOOCs) during the COVID-19 pandemic to investigate the relationship between MOOCs and students’ course performance. All hypotheses were supported except Hypothesis 8 (gamification satisfaction). It indicates that, among the three MOOC platform qualities, the MOOC platform’s information quality was the most critical factor in affecting the quality confirmation of the MOOC platform. In addition, the MOOC platform’s quality confirmation significantly influenced students’ perceived usefulness of the MOOC platform, satisfaction with MOOCs and students’ perceived gamified features. Moreover, only when students perceive MOOCs’ usefulness and gamification, and when they are satisfied with the MOOC platform will they consider using MOOCs for future studies. Furthermore, with students’ perceived usefulness and gamified MOOCs’ features increasing, their learning performance will improve significantly. Finally, if students intend to use MOOCs for their future learning, their learning performance will also considerably improve. The results indicate that students’ continued use intention and learning improved a positive attitude toward MOOCs. It is worth mentioning that we combined gamification with the IS success model and ECM in this study. The findings support the notion that the gamification of MOOCs consists of social interaction, entertainment, and challenges. Furthermore, gamification has a significant impact on students’ intentions to continue using MOOCs and improve course performance.

4.1. Theoretical Implications

The results of this study could provide future studies with some theoretical implications. Firstly, our study utilized an integrated framework combined with DeLone and McLean’s [24,25] IS success model and Bhattacherjee’s ECM [17] and introduced MOOCs’ gamified designs to the research model. Next, we introduced information quality, system quality, and service quality to define MOOC platform qualities. Then, we investigated the overall MOOC platform quality confirmation through the ECM framework. The integrated framework ensures that the MOOC platform quality confirmation links students’ perceived characteristics of the MOOC platform, students’ post usage intentions, and learning performance.

Secondly, most previous research on MOOCs has primarily focused on the users’ intention to adopt or the users’ intention to re-adopt. This study expanded the research model by syncing the MOOC platform’s continued usage intention and course performance and empirically testing the influence of MOOCs’ cognitive perceptions on course performance. The results show that MOOCs significantly impacted students’ course performance through various factors. This study measured the characteristics of MOOCs and users’ behaviors. Another difference between this study and previous studies is that it attempted to consider MOOCs as a learning tool and a combination of technology and a learning option during COVID-19. To the best of the authors’ knowledge, this study may be novel research in measuring how MOOCs influence students’ course performance under the context of a public health emergency.

4.2. Practical Implications

This study also makes several contributions to MOOC platform managers about how to improve their platforms and services. First, the results indicate that MOOC platform qualities had significant influences on the MOOC platform quality confirmation. Information quality had the most significant effect, followed by service quality and system quality. From this perspective, we concluded that the information quality of the MOOC platforms exceeded students’ expectations. For this reason, MOOC platform managers should improve information quality continuously. MOOC platform managers should pay attention to online service or call center service construction as MOOCs’ service quality is lower than students’ expectations. One of the reasons for this phenomenon might be found in the increasing number of users during COVID-19, which meant that users could not get consulting services in a timely fashion. The results also indicate that system quality significantly influences the confirmation of a slight estimate. This emphasizes the importance of system quality in the management of the MOOC platform. MOOC platforms are frequently paralyzed during COVID-19 because tens of millions of students attend online classes at the same time. Improvements to the MOOC platform’s quality will increase students’ pre-expectations.

Second, the results show that students’ perceived usefulness and confirmation of the MOOC platform significantly influenced students’ satisfaction with the MOOC platform. If students believe MOOCs are simple to grasp, useful, well-managed, and beneficial to their studies, it will influence the level of students’ satisfaction with MOOC platforms. MOOC platform managers should improve and upgrade the system and update course materials to provide a better user experience. By increasing the usefulness of MOOC platforms, students will be more likely to use MOOCs for continuous learning. Furthermore, useful course materials, well-constructed content, and an easily navigated system will significantly improve students’ course performance.

Third, the results prove that MOOCs’ continued usage intentions positively influenced students’ course performance. Thus, MOOC platform managers should develop mechanisms to stimulate students’ course completion rates. Lastly, the results indicate that gamification was composed of social interaction, entertainment, and challenges. Gamified mechanisms significantly influence students’ continued usage intentions and their course performance. During COVID-19, students have to take all of their classes online and spend 6~8 h per day on online learning. Therefore, it is necessary to make MOOCs more exciting and attractive to students. As previous research has shown, one of the primary issues with MOOC learning is the low completion rate of students [81]. Gamification mechanisms such as redeemable points, team leaderboards, badges, peer grading, emoticon feedback, peer interaction, etc., will increase the course completion rate. In addition to the current gamified features, MOOC managers should develop other attractive and useful gamified designs to improve students’ MOOCs’ continued usage intentions and course performance.

5. Conclusions

Despite the theoretical and practical implications, this study has some limitations, as follows. First, this study’s results and implications should not be generalized because the survey was conducted in China only. We consider that comparative research in another cultural context could contribute more effectively in the future. Second, we developed a questionnaire based on previous studies and obtained empirical data from online learners. Personal interviews or focus group interviews may be chosen for future studies to create new constructs and items. Third, we did not consider control variables in this study. Future research may examine the influences of control variables such as demographic variables and personality traits, which might influence the research results. Lastly, this study showed that gamification did not significantly influence students’ satisfaction with MOOCs. However, previous studies have reported that gamification in IS is a critical attribute to users’ satisfaction. Thus, we need to conduct further investigations into the relationship between gamification and satisfaction in future studies.