The Impact of the COVID-19 Pandemic on User Experience with Online Education Platforms in China

Abstract

During the COVID-19 pandemic, social education has shifted from face to face to online in order to avoid large gatherings and crowds for blocking the transmission of the virus. To analyze the impact of virus on user experience and deeply retrieve users’ requirements, this paper constructs a reasonable evaluation index system through obtaining user reviews about seven major online education platforms before and after the outbreak of COVID-19, and by combining the emotional analysis, hot mining technology, as well as relevant literature. At the same time, the variation coefficient method is chosen to weigh each index based on the difference of index values. Furthermore, this paper adopts the comprehensive evaluation method to analyze user experience before and after the outbreak of COVID-19, and finally finds out the change of users’ concerns regarding the online education platform. In terms of access speed, reliability, timely transmission technology of video information, course management, communication and interaction, and learning and technical support, this paper explores the supporting abilities and response levels of online education platforms during COVID-19, and puts forward corresponding measures to improve how these platforms function.

1. Introduction

The global spread of COVID-19 resulted in the suspension of classes for students from more than 60 countries, disrupting the original teaching plans of schools in these countries and regions [1]. As the first country to detect the spread of the virus, China was also deeply affected by it [2]. Under the influence of the COVID-19 pandemic, schools were forced to suspend in China. However, the massive suspension of school is found to affect the teaching progress [3]. In order to minimize the impact of the pandemic on education and control the spread of the pandemic, online teaching has become a necessary strategy to restore the normal teaching order in this special period. In the face of massive demand, some office-meeting software, supported by the strong technical ability and the keen insight into market opportunities, have also developed and improved relevant functions, and have become a platform for teachers and students to realize online teaching together with many education platforms. Among many online teaching platforms, the office-meeting software represented by DingTalk has provided service such as online classroom and online teaching functionalities. However, due to some technical and functional defects, these online teaching platforms have been criticized by millions of students. For instance, the satisfaction score of DingTalk dropped from 5 stars to 1 star quickly [4].

The above phenomenon shows that although the online education industry has a broad application prospect, it exposes some existing problems in its development process, especially during a public emergency period. Since MOOC (Massive Open Online Course) was widely used in 2013, online teaching modes have gradually been known. At the same time, online education has attracted more and more attention due to its advantages such as breaking the spatio-temporal limitations and improving the fairness of education [5]. Furthermore, research shows that the MOOC teaching model can develop teachers’ careers and improve teachers’ teaching skills [6]. However, at present, due to technology restrictions, online education has been focused on vocational classes and tutorial classes without involving basic courses and professional courses. Most schools still adopt the traditional teaching ways [7]. As a result, school suspensions are a common risk aversion measure in the education sector when facing public health emergencies such as COVID-19.

The current pandemic prompts the reform of existing teaching modes in ways that make online education suddenly transit from an auxiliary method to the key way. This change brings obvious difficulties and challenges to the online education platforms, including the changes of users’ concerns regarding online education, new requirements for online education, the satisfaction of these requirements, and the live broadcast. All above issues need to be further thought and discussed.

These platforms provide strong support and help for education during the pandemic period, and bring users a new experience, but also bring a lot of controversy. In order to better understand the disputes from the source, it is necessary to analyze the changes of users’ concerns on these platforms before and after the epidemic. Based on this, in terms of user experience on online education platforms during COVID-19, this paper first collected user comments from seven representative and widely-used mainstream online education platforms, scored comment emotion, and constructed an online education platform evaluation index system. While setting the index weight, the coefficient of variation method and entropy method were used for calculation, and the results of these two methods were compared and the former was finally selected. Finally, based on the obtained index weights, the user experience of each platform before and after the outbreak of COVID-19 was evaluated, so as to analyze the impact of the pandemic on user experience.

On the one hand, this paper aims to see the emergency response level of online education in response to public health emergencies, as well as to explore the development level of online education and the technical support capacity of these platforms. On the other hand, in view of the current disputes regarding online education, this paper explores the source of disputes from the perspective of user experience, and understands the defects of online education platforms, so as to help online education achieve longer-term development. Therefore, we hope that some of the research in this paper can help the world know the impact of the pandemic on education, and make the following research have a clearer understanding of the current level and disadvantages of online education. In addition, the mining of user reviews and the processing methods of user experience are expected to provide enlightenment and experience for other scholars and related companies, and help other industries such as education or tourism to get closer to users’ real feelings and feedback in user experience related research.

The structure of the paper is organized as follows: Section 2 is a literature review to provide a basis for the subsequent establishment of the platform evaluation system; Section 3 selects seven representative online education platforms and collects their user comments. Section 4 completes emotional scores for each platform based on the initial comments and digs out hot issues from comment data and ranks their popularity. Section 5 constructs and quantifies the evaluation index system of online education platforms, and then evaluates user experience. Section 6 provides the summary of the whole paper.

2. Literature Review

This paper focuses on the impact of the pandemic on the user experience of online education platforms. Due to the outbreak of COVID-19, there is very limited research in this area. However, we can still learn from the existing international research on online education. The research method of this paper is to establish an evaluation index system for online education platforms. Therefore, we have reviewed and sorted out the existing international evaluation systems, whose evaluation indexes are mainly based on the traditional teaching evaluation system indexes. Secondly, the purpose of this paper is to study user experience, so the following is a review of the literature related to user experience; and build our indicators based on the influencing factors of user experience studied by previous people. This section will analyze the literature from the above two aspects.

2.1. The Study of Education Evaluation Systems

Due to the severity and urgency of the COVID-19 outbreak in late 2019, school suspension is one of the most common ways of pandemic prevention and control [8]. However, the suspension of classes affects the progress of courses, but the resumption of classes may cause a second spread of the pandemic in schools [9]. Especially in the severe pandemic period, the best way to solve the contradiction between the two is to shift classroom teaching activities from in-person to online, which can effectively control the crowd gathering. Therefore, the online education mode under COVID-19 will become an important way to prevent and control the pandemic and ensure the teaching progress.

Unlike online platforms in other industries, the evaluation of online teaching platforms is not only reflected in technology, but also needs to be measured in terms of the quality of online courses, teaching levels and so on. At present, the traditional educational evaluation system indicators are mainly based on expert scores and industry-developed standards. Representative studies are as follows: Tochot et al. [10] constructed a measurement model consisting of conceptual use, symbolic use, legitimate use, and instrumental use through external evaluations by The Office for National Education Standards and Quality Assessment (Public Organization) (ONESQA). Daytion and Vaughn [11] designed a network course quality assurance system with 7 first-level indicators and 25 s-level indicators based on educational practice goals. From the perspective of the quality of online courses, Lin [12] had established an evaluation system consisting of 4first-level indicators, namely system quality, information quality, service quality and attractiveness, and 16 s-level indicators. The above evaluation indexes are mainly aimed at the traditional offline teaching mode, but do not apply to the online mode.

In recent years, with the popularization of online education, more and more online education platforms have emerged one after another. Meanwhile, higher requirements have been put forward for the teaching/learning outcomes and technical standard that the platforms need to achieve. Therefore, more consideration should be given to the technical and interactive features of online education when evaluating these platforms and courses. For example, Kimberley et al. [13] combined the characteristics of MOOC platform, and compared several assessment methods to find their respective advantages and limitations. Wong and Billy [14] compared four MOOC platforms—namely Coursera, edX, FutureLearn and OpenLearning, and the results show course differences in terms of their duration, learning activities, assessment, social interaction and instructors’ participation. Based on the evaluations in terms of theory foundation, principle of evaluation, evaluation index system, evaluation system and platform design, and implementation over the Moodle network learning system, Xia [15] constructed a relatively complete evaluation system with the characteristics of scientific and practical education. Combined with course quality evaluation indicators, Jin [16] established the open quality evaluation index system of online education through the capability maturity models such as reliability, effective testing, and expert consultation verification. Additionally, in the field of mobile news training, Cervi et al. [17] used the data collected through the post-evaluation survey to prove that MOOCs were effective training tools by evaluating the structure, function, and opinions of participants. Moreover, factors such as policies would also have an impact on the results of education. For example, in the field of media quality education, Caprino et al. [18] described the connection between the media education policies and media literacy levels in 27 member states of the European Union.

Most of the above researches on the evaluation system of online education platforms are from the perspective of curriculum and platform, but less related to user requirements and experience. For example, Wong and Billy [14] scored the platform based on the traditional quality evaluation system, the indicators and scores were improved or given by themselves on the basis of the past studies. However, this paper will show an evaluation system in which the user experience indexes are not only based on the previous indicator system, but also from the user comments. More importantly, under the influence of public health emergencies, the course content, teaching methods of online education and even students’ learning mentality are different from the past. While constructing the indicator system, users’ experience and feedback behaviors should also be further considered.

2.2. The Study of User Experience

User experience (UX) [19] refers to the feelings of users’ feelings before, during and after using a product or system, including emotions, beliefs, preferences, etc. Zahidi et al. [20] pointed out that the factors that affect user experience were the driving factors that triggered user satisfaction and dissatisfaction. Moreover, user satisfaction depends on user needs, expectations, and existing user experience.

Current research on user experience and satisfaction over the platforms is conducted through questionnaire surveys or online reviews to obtain basic data, so as to complete statistical analysis or natural language processing. UX is widely used in the evaluation and optimization research of commercial platforms, but rarely in online education. Fox example, Pappas [21] conducted the research with regards to the influencing factors of purchase intention, according to the real-time experience feedback of 185 users with online shopping experience. Lohse et al. [22] used the user experience questionnaire to evaluate the music, cutting and flickering of the learning video. According to the analysis of customers’ online shopping comments’ vocabularies, Li et al. [23] obtained 11 indicators which constitute to the satisfaction index system of e-commerce platforms. Yang [24] calculated the evaluation score of a certain online courses by establishing a word library with positive/negative emotional tendency, and combining the proportion weight of comments in each category and the gap between the number of positive and negative comments. Compared with the questionnaire data, the analysis based on online user comments is more realistic and objective. The objects of questionnaire analysis are directional and not globally representative. Moreover, the questions set in the questionnaire analysis are subjective. The data of network comments generally contain user objects with various ages and levels, and there is no need to set some specific questions to obtain the data, so the data obtained are more objective. Meanwhile, the evaluation index model built on this basis can help the platform receive continuous evaluation feedback and dynamically understand its advantages and disadvantages.

Current research on online education platforms is mainly focused on the satisfaction evaluation results, willingness for continuous use, and learning influencing factors. Kamali et al. [25] made the conclusion that the help of electronic devices and resources on learning and education is limited. In order to give better effect, the priority should be given to provide a network environment that students can adapt. Tawafak et al. [26] found that the continuance intention depends on the type of technology. Roca et al. [27] verified that users’ continuance intention is determined by satisfaction, which in turn is jointly determined by perceived usefulness, information quality, confirmation, service quality, system quality, ease of use and cognitive absorption. Kravvaris et al. [28] and Mackness et al. [29] found that learners’ autonomy played an important role in learning through the empirical study of MOOC. Ayşenur et al. [30] took the following factors into consideration: perceived usefulness, flexibility, reliability, active participation, instructor response time and consultation. Asarbakhsh and Sardars [31] made the conclusion that system stuck and failed video connection affect user satisfaction through analyzing learning demand, technological design and intervention content. Roth et al. [32] verified that students receiving the course through video conference had lower final grades and were less satisfied with the course and the instructor. Perceval and Tejedor [33] portrayed an overview of the five degrees of communication in education: oral gestural, writing, audio, audiovisual, and digital, which highlighted the changes introduced by the online scenario in the educational process, reflecting on the character of the student, the teacher and the relationship between them. Chen et al. [34] used a questionnaire survey and web crawler to collect comment data of online and offline users, constructed a customer satisfaction index system by analyzing emotion and the existing literature for quantitative analysis, and then forecasted user satisfaction. Most of the above researches focus on platform satisfaction or curriculum setting without evaluating and analyzing the teaching content combined with platform technology. However, they still have reference value for the establishment of an online education platform evaluation system in this paper.

To sum up, there are still some problems in the research on the evaluation system of online education platforms, such as the lack of specific indicators, unclear evaluation objects, and unreasonable weight distribution. The indicators in existing evaluation systems mainly focus on the two aspects of online education curriculum and platform function; that is, setting fixed evaluation indicators from the aspects of course quality, teaching effect, teaching technology, system quality, etc., and realizing the evaluation through qualitative or quantitative methods. However, few people evaluate the level of online education platforms from the perspective of user experience, and they do not consider COVID-19 and the possible impact of the outbreak. Based on this, we expect to realize the evaluation of user experience before and after the outbreak of COVID-19 from the perspective of user evaluation. However, user experience is mostly used in the research of business platforms. Considering that many research results from the factors affecting online education satisfaction can be used as evaluation indicators that impact user experience, this paper will refer to the results of these articles to establish our indicators. Additionally, in the analysis of user satisfaction and experience, more and more researches focus on users’ comments and feedback, so as to score and evaluate the platform and courses. By using the methods of processing and capturing these comments, the index system of online education platforms can be built more accurately and more perfect.

Based on this, in terms of the effects of COVID-19, this paper optimizes the indicators used in previous research, constructs an evaluation system for the online education platform, and finds out users’ focus in order to analyze the impact of the COVID-19 pandemic on the user experience of online education.

3. Data Acquisition

Before analyzing the data, comment data should be captured. This section first selects the mainstream online education platforms during COVID-19, and then grabs the user comment data of the selected platforms before and after the outbreak of the pandemic.

3.1. The Choice of Online Education Platform

At present, there are a large number of online education platforms in China, including MOOC, Tencent Classroom, Xuetang Online, Yu Classroom, etc. However, the quality of these platforms varies greatly, so it is necessary to select a representative platform from many online education platforms. Qimai data is a professional mobile promotion data analysis platform launched by Beijing Qimai Technology Co., Ltd. (Beijing, China), which supports data query of IOS, Android application market, WeChat, small programs, etc. This paper selects data samples of online education platforms on the Qimai platform.

The download volume through online education platforms, comments rank, as well as the platform’s popularity are taken as measurement criteria. During COVID-19, more business software has shifted to developing online education functions and has been widely used. This paper has screened the ranking of “business” and “education” in the Qimai data subcategories (updated on 17 March 2020). The results are shown in

Table 1. Rank of online education platforms.

Application	Application List	Classification Ranking	Keyword Coverage
DingTalk	first	First (business)	18,799
Tencent Meeting	second	Second (business)	8526
Zoom Cloud	eighth	Eighth (business)	5333
Chaoxing	seventh	Seventh (education)	2043
MOOC	thirteenth	Thirteenth (education)	7031
TIM	fourth	Fourth (business)	4467
WeChat Work	third	Third (business)	6157

below. Application refers to the platform object selected in this study. Classification ranking refers to the user usage ranking of the selected platform in the two subcategories of “business” and “education” on Qimai website. Application list refers to the total user usage ranking of the selected platform on Qimai website. The “Keyword Coverage” mainly means that users can find our APP (APP is the abbreviation of Application, which means Application program. It mainly refers to software installed on a smart phone. Moreover, the original system can improve the shortcomings and is personalized to meet the different needs of users) even if they search for more keywords.

It can be seen from Table 1 that in the category of “business”, DingTalk, Tencent Meeting, TIM, and WeChat rank in the top four. In addition, Zoom Cloud, which ranks 8th in this paper, is selected because the business software ranked 5th to 7th have not been transformed into online education platforms. In the category of “education”, though Chaoxing Learning, MOOC did not rank at the top, since the education ranking also contained a non-online learning platform, so Chaoxing Learning and MOOC ranked top relatively on the premise of their network teaching platform. Therefore, Chaoxing Learning and MOOC are selected here as the representative platforms of the online teaching and education industry, while DingTalk, Tencent Meeting, ZoomCloud, TIM, and WeChat are selected as the representatives of the transformation from business software platforms to online education platforms during the pandemic.

3.2. Acquisition of Comment Data on Online Education Platforms

When collecting the data of user comments from DingTalk, Tencent Meeting, ZoomCloud, TIM, WeChat Work, Chaoxing Learning, and MOOC, the moments (time nodes in this paper) of comment collection and the types of comments should be considered. Since this paper mainly compares the user experience of the online education platform before and after the outbreak of the pandemic, the time nodes of comment collection are divided into two stages: 17 February 2020 to 17 March 2020 and 16 November 2019 to 16 December 2019. In addition, there are four types of user comments on Qimai website, namely: all comments, developer reply comments, deleted comments and undeleted comments. The deleted comment data cannot be collected and the developer’s reply to the comment has nothing to do with user experience. Additionally, the number of comments per platform does not represent the number of users on the platform because the number of comments is not a proportion of the number of users. In order to make the comment more timely, the undeleted comment is selected here as the comment data.

Through the analysis of the user usage and comment popularity of the online education platform, the collected data were manually and simply processed, and the invalid words such as emoticons, repeated words and mood words were removed to obtain the number of comments on the seven platforms before and after the outbreak of COVID-19, as shown in

Table 2. Number of comments collected by the seven platforms before and after the outbreak ofCOVID-19.

Teaching Platform	Before the Outbreak of COVID-19	After the Outbreak of COVID-19
DingTalk	1061	74,981
Tencent Meeting	1438	3970
WeChat Work	1022	959
Chaoxing	800	5319
MOOC	930	1819
TIM	500	400
ZoomCloud	560	800

below.

4. Analysis on the Characteristics of User Comments on Online Education Platforms

Since users’ comments are relatively redundant and scattered, ROST CM5.8.0 software is used to classify the emotional tendency of users’ comments, and NLPIR-Parse software is used to score the emotional tendency of the comments, so as to analyze the online education platform with good user experience and the emotional tendency of users. According to the statistical method of text similarity and the visual analysis technology of semantic network, the hot issues concerned by users are mined as the basis for the construction of the subsequent user experience index.

4.1. Emotion Analysis

Emotional analysis of text is also called opinion mining, tendentiousness analysis, etc., allowing a large number of internet users and objects to participate, such as people, events, products, and other valuable comments on information. These comments’ information expressed by people of all kinds of emotional color and emotion tendentiousness, as well as text sentiment analysis, is the subjectivity of color with emotional text analysis, processing, summing up and reasoning process. The actions in the text, with the help of semantic rules to each user, reviews data assignment by value to determine the size of the emotion and value of positive emotions, with said negative emotion value is negative, neutral emotional value is zero said, and the higher the value, the more positive attitude, with said the lower value, said the more negative attitude. Thus, the corresponding emotional score and emotional tendency of user comments are obtained.

By rating users’ comment emotional tendency and emotion scores, this paper finds out users’ preference for the online education platform before and after the occurrence of COVID-19 and regards it as the initial grading of the platform. Due to the partial limitation of the emotion dictionary in ROST CM5.8.0, NLPIR-Parser software is used to score the comments on emotion, so as to find the online education platform with good user experience.

4.1.1. ROST CM Analysis

As a digital research platform for humanities and social sciences based on content mining, ROST CM [35] is a large, free social computing platform developed by Professor Shen in Wuhan University. At present, it is the only one in China for the purpose of supporting humanities and social science research and is widely used in many fields such as network public opinion [36,37], personalized recommendation [38], emergency rescue decision making [39,40] and so on. The software can realize Weibo, chat, the entire network analysis, site analysis, through analysis, word segmentation, word frequency statistics, English word frequency statistics, traffic analysis, clustering analysis, and a series of text analysis. Text mining does not need a basic computer using this software to conduct text mining does not require a computer foundation, as long as it can be carried out in accordance with the operation step. ROST CM software is competitive in semantic network emotion analysis. In this paper, ROST CM5.8 is used for emotion analysis, and the analysis results are used to integrate the ratio of positive, medium, and negative comments of seven platforms before and after the outbreak ofCOVID-19, as shown in

Table 3. The proportion of comments with emotional orientation before the outbreak of COVID-19.

Teaching Platform	DingTalk	Tencent Meeting	WeChat Work	Zoom Cloud	TIM	Chaoxing	MOOC
Proportion of positive comments	50.05%	52.92%	40.72%	44.35%	45.21%	0.00%	60.33%
Proportion of neutral comments	1.31%	1.32%	2.99%	2.09%	1.27%	0.00%	1.31%
Proportion of negative comments	48.64%	45.76%	56.29%	53.56%	53.52%	100%	38.36%

and

Table 4. The proportion of comments with emotional orientation after the outbreak of COVID-19.

Teaching Platform	DingTalk	Tencent Meeting	WeChat Work	Zoom Cloud	TIM	Chaoxing	MOOC
Proportion of positive comments	52.26%	47.86%	48.78%	42.43%	40.88%	22.40%	30.29%
Proportion of neutral comments	1.32%	1.34%	1.22%	0.73%	0.73%	1.43%	1.32%
Proportion of negative comments	46.43%	50.81%	50.00%	56.84%	58.39%	76.18%	68.39%

below.

(1) From the above analysis, during COVID-19, DingTalk receives better user experience response with positive reviews. It suggests that duringCOVID-19, with plenty of technology and reliable software support, DingTalk can easily cope with the dramatic increase of users. As a business software, it can meet the requirements of teachers and students while responding to various requirements of online education. The following contents will further explore the advantages of this platform and use it as a reference for other platforms. Compared with DingTalk, Chaoxing Learning needs to think of the reasons why users have poor experience. Generally speaking, as an online education software, users will not consider using this platform if there is no technological innovation or functional improvement. A lot of technical and reliable software support is constructive for the later improvement of the online platform.

(2) Compared with before the occurrence of COVID-19, users’ positive comments on Tencent Meeting and MOOC decreased somewhat. The proportion of Tencent Meeting decreased by 5.06%, but the proportion of MOOC positive comments decreased as high as 30.04%. As a self-study software providing learning resources, MOOC prefers to provide electronic books, video, etc., and gives priority to self-study. During COVID-19, in order to improve the quality of students’ learning outcomes, by offering the video teaching, audio conferencing, etc., most colleges and universities carry out the teaching by strongly interactive methods to achieve the same quality with in-person teaching. However, in the face of COVID-19, MOOC did not develop new functions timely enough to improve their disadvantages, resulting in a sharp decline in user experience. For the later evaluation of the online education platform, evaluation criteria can be provided. If the platform can provide video teaching, audio conference and teaching in a highly interactive manner, users will have high commendation on the platform.

(3) The positive and negative comments of WeChat Work, TIM and Zoom Cloud platforms before and after the outbreak ofCOVID-19 did not change significantly. Zoom Cloud’s positive comments decreased by 1.92%, WeChat’s increased by 8.06% and TIM’s decreased by 4.33% during the pandemic. These online education platforms have not been greatly improved on the whole, but compared with Chaoxing Learning and MOOC, their user experience is relatively better. The positive and negative emotional evaluation of the platform by users has reference opinions on the later evaluation of the platforms. Generally speaking, if the negative emotional evaluation of the platform by users is too heavy, the overall evaluation by users on the platform will be lower. For the enterprises that belong to the platform, software upgrade is of vital importance.

4.1.2. NLPIR Emotional Score

NLPIR-Parser [41] big data semantic intelligent analysis platform, which has been developed for more than 20 years, integrates core technologies such as network data collection, natural language processing, text mining and text retrieval. NLPIR emotion analysis mainly focuses on the automatic recognition and weight calculation of emotion words, and adopts the co-occurrence relationship and bootstrapping strategy to generate new emotion words and weight through repeated iteration. In bibliometrics, the common word method of keywords is often used to determine the relationship between the themes in the discipline represented by the document collection. For example, co-occurrence relationship can be used to analyze the relationship between characters in a novel or play. In statistics, bootstrapping strategy is a uniform sampling with a return from a given training set; that is to say, every time a sample is selected, it is equally likely to be selected again and added to the training set again.

The emotional scores of the comments before and after the outbreak of the pandemic (including total emotional scores, positive scores and negative scores) are shown in

Table 5. Emotional rating of NLPIR-Parser before the outbreak of COVID-19.

Teaching Platform	DingTalk	Tencent Meeting	WeChat Work	Chaoxing	MOOC	TIM	Zoom Cloud
Total emotional score	−50	−68	−33	1	−1	−38	0
Positive score	52	32	28	4	12	4	3
Negative score	−102	−100	−61	−3	−13	−42	−3

and

Table 6. Emotional rating of NLPIR-Parser after the outbreak of COVID-19.

Teaching Platform	DingTalk	Tencent Meeting	WeChat Work	Chaoxing	MOOC	TIM	Zoom Cloud
Total emotional score	105	−267	−14	−341	−43	−20	530
Positive score	563	114	29	152	79	0	758
Negative score	−458	−381	−43	−493	−122	−20	−228

by analyzing the comments data of the seven online education platforms.

According to the above emotional score data, before the pandemic, the feedback of the seven platforms was generally negative, indicating that most users had a poor experience when using the online education platform. From the previous section, the conclusion can be made that during COVID-19, total emotional scores of DingTalk and Zoom Cloud are positive; and Zoom Cloud is the highest, indicating that user experience of Zoom Cloud in general is the best, followed by DingTalk. These two platforms show more advantages after the pandemic outbreak, which indicates that technical response capacity of DingTalk and Zoom Cloud is a worthy reference. In addition, after the occurrence of COVID-19, the negative score of Chaoxing Learning is much higher than its positive score, and its disadvantages are more obvious, leading to a poorer overall user experience.

4.1.3. Visual Analysis Based on Semantic Network

Based on ROST CM’s analysis of the positive and negative proportion of user comments and NLPIR’s rating of user comments’ emotions, it can be seen that DingTalk gets unanimous praise and provides the best user experience, while Chaoxing Learning and MOOC have poorer user experience. In order to obtain users’ main concerns on online education platforms, semantic network visualization analysis was adopted before and after the outbreak of COVID-19 for platforms such as DingTalk, Chaoxing Learning and MOOC.

Semantic network is one of the representations of an artificial intelligence program, which expresses human knowledge construction in the form of a network. It consists of arcs between nodes, where nodes represent concepts (events or things), and arcs represent relationships between them. The semantic network consists of four related parts. The lexical part: Determining which symbols are allowed in the vocabulary, which involves nodes and arcs, and extracting key words from user comment information. The structure part: Describe the constraint conditions of symbol arrangement, specify the node pairs connected by each arc, that is, establish the connection line through the causal relationship between keywords commented by users. The process section: Explain the access process, which can be used to establish and modify the description, and answer the related question, that is, the direction between related words indicates cause and effect. The semantic part: The method to determine the meaning related to description, that is, to determine the arrangement of relevant nodes and their possessive and paired arcs.

The software ROST CM5.8.0 analysis was used to obtain the semantic network relationship shown in Figure 1 and Figure 2. The main purpose is to show the semantic network diagram of user comments to get the similarity of users’ comments on the platform, so as to dig out the most concerned points of users.

From Figure 1 and Figure 2: the “software” is the important node, and “live” is the closest node, reflecting that the major teaching method of DingTalk is live, which has increased the interaction, and better enlivened the learning atmosphere, improving the teaching quality. In addition, before and after the outbreak ofCOVID-19, “five star”, “payment by installments” and other kinds of nodes appear, because DingTalk congested for a while, public relation (PR) explained timely, which caused a topic of “payment by installments to the five-star”. It is a timely response that makes users give five stars and high commendation to DingTalk.

At the same time, the “office” before the pandemic was closer to the central node. The description of the relevant nodes involved two aspects: office and education, among which “mobile phone”, “holiday”, “powerful” and “Internet” nodes were the main evaluation of the business function of DingTalk. This is because before the outbreak of the pandemic, DingTalk was more of an important software for business clocking and office work. However, after the outbreak of the pandemic, most of DingTalk’s nodes focused on educational evaluation, such as “live broadcast”, “school”, “children” and “homework”, while the business experience evaluation was relatively rare. This indicates that DingTalk adjusted its software functions in time to meet the needs of users of online education courses before and after the outbreak ofCOVID-19.

In addition, the semantic relations of Chaoxing Learning and MOOC are as shown in Figure 3, Figure 4, Figure 5 and Figure 6: setting “software”, “rubbish”, “learning”, “class” as the key nodes; “submit”, “server”, “log in”, “collapse”, “waiting” are the closest nodes, demonstrating that Chaoxing Learning has more problems, such as: server crash, unable to log in or submit the learning time due to system failure. In addition, setting “learning”, “rubbish”, “curriculum” and “software” as important nodes, the nodes close to each other include “failure”, “connection”, “duration”, “server” and “progress”, etc. It can be seen from these nodes that connection failure, unsubmitted learning duration, server crash and other problems often occur in the MOOC platform. At the same time, there are also some independent nodes in the figure, including “account” and “homework”. It can be learned that the platform may fail to register an account, submit homework, or refresh website, etc., while “delay”, “serious” and other nodes indicate that there is a serious delay in the MOOC platform.

4.2. Acquisition of Hot Issues in User Comments

In order to dig out the hot issues that users pay attention to from a large number of data, so as to build a subsequent user experience evaluation system, the similarity heat statistics method is used to analyze the similar information from the descriptive information of the text, and the information is ranked according to the frequency of the occurrence of the similar information.

4.2.1. Introduction to Algorithm Principle

In this section, Python is used for text similarity heat statistics. The input is Excel, and only platform comment content is included here. The specific implementation steps are as follows:

(1) Data preprocessing

Pandas, the data analysis package of Python, are used to obtain input information, and the words of the input comment sentences are segmented through the Python Chinese phrase fragment Jieba to form a two-dimensional array.

(2) Dealing with dictionaries

Gensim is a Python library for automatically extracting semantic topics from documents that can be used to process unstructured and numeric text. The corpora.Dictionary method in the library is invoked to generate dictionaries from two-dimensional arrays of words, thus creating a dictionary based on input comment text information that uniquely identifies a word with a numerical number.

(3) Corpus processing

The Bag-of-Words model means to pack all words into one bag without considering their morphology and word order, that is, each word is independent. If you create a dictionary [Jane, wants, to, go, Shenzhen, Bob, Shanghai], the sentence “Jane wants to go to Shenzhen” can be expressed as (1,1,2,1,1,1,0), which is the number of occurrences of the corresponding word in the dictionary. Therefore, a two-dimensional array is converted into a sparse vector through the Doc2Bow method used in Python to build the Bag-of-Words model to form a corpus.

(4) Calculating text similarity

The LSI (Latent Semantic Indexing) model uses SVD (Singular Value Decomposition) to decompose the word-document matrix. SVD can be seen as discovering unrelated index variables from the word-document matrix and mapping the original data into the semantic space. Documents that are not similar in the word-document matrix may be similar in the semantic space. The text topic matrix obtained by LSI can be used for text similarity calculation.

TF-IDF is a statistical method used to assess the importance of a word to one of the documents in a document set or corpus. The importance of a word increases proportionally with the frequency of its occurrence in the document, but decreases inversely with the frequency of its occurrence in the corpus.

In practice, models-used Latent Semantic Indexing (LSI) model calculates the TF-IDF in a corpus, then uses keywords acquisition method to obtain the number of features in the dictionary, finally takes the corpus in the words of the number of features in TF-IDF and dictionary into Sparse Matrix Similarity calculation methods, establishes a sparse matrix similarity and an index.

(5) Calculating the similarity between test data and sample data

Each user comment text was segmented through Jieba, sparse vector of test data was calculated through doc2bow, and finally, the similarity between test number and sample data was calculated, and data with similarity of more than 0.6 was classified as a class of problems.

(6) Statistics on hot issues

The amount of data in each type of problem is calculated and regarded as problem heat. The questions are ranked according to their heat.

4.2.2. Results Analysis

Data from seven online education platforms before and after the occurrence of COVID-19 were statistically analyzed for text similarity and heat. The results are shown in Figure 7 and Figure 8.

The comparison between Figure 7 and Figure 8 shows that before and after the outbreak of COVID-19, the hot issues that users were concerned with on the same platform changed greatly. For example, before the outbreak of COVID-19, users’ biggest comment on the MOOC platform was the lack of landscape function, but after the occurrence of COVID-19, users’ biggest comment on the platform was the problem of crash back. This shows that in different periods, users have different concerns and requirements for the platform. In addition, by comparing the hot issues presented by different platforms, it is not difficult to find that there are certain similarities in these issues, such as stuck and crash back, which are hot issues existing in almost every platform. Therefore, the existing problems of all platforms can be further integrated to establish a unified evaluation index system.

5. Evaluation of User Experience of Online Education Platforms before and after the Outbreak of the Pandemic

In this section, by analyzing the characteristics of comments on online education platforms, the comments of each platform are classified according to the similarity degree, so as to obtain the main problems of the platforms in user experience. According to the word frequencies of user comments, the related word frequency of each type of problem is counted. Then, based on the existing literature and hot issues of user comments, the evaluation index system of user experience of the online education platform is constructed and quantified. Finally, the changes of user experience before and after the outbreak of the pandemic are found out and analyzed.

5.1. Setting of Index System

Based on the Network Learning Evaluation System over the Moodle platform proposed by Xia [14] and referring to the evaluation index system of open online course teaching quality by Jin [15], user evaluation is selected as the analysis basis. Data selection and evaluation methods of Li [20] and Yang [21] are adopted, comment results are analyzed, problems of each platform are integrated to construct the user experience evaluation index system of online education platforms, which contains 5 primary indexes and 15 secondary indexes, as shown in

Table 7. Setting of index system.

First Level Indicators	Secondary Level Indicators	Definition of Indicator	Index Evaluation Method
Platform system characteristics	System stability	The website can be successfully accessed at any time, there will be no errors, the system does not appear flashback, do not crash	Establish different word banks for the secondary indicators. Through the enumeration of the index thesaurus, the frequency of the occurrence of the words is taken as the evaluation method of this index.
	System compatibility	The online education platform can be used by different clients and can be received by different terminals
Background customer service support	Technical and academic support	Provide the use of online education platform and learning-related technical support services, and provide valuable guidance and help for the focus and difficult content of the course
Platform video quality	Video picture quality	The video picture is clear, and the students will not have visual discomfort and unpleasant mood when watching the video
	Video sound quality	The video sounds clear
	Timeliness of video information transmission	Teachers’ PPT and blackboard pictures can be displayed on the students’ interface in a timely manner
Platform technical requirements	Interface design	The interface layout is reasonable, the text expression standard, the font size color suitable for reading
	Access speed	Platform page load speed
	Navigation link	Content navigation is convenient, clear, in line with the browsing rules
	Security	The platform shall keep confidential the personal information of users
	Reliability	The whole system runs stably for a long time, and there are few cases of lag and flashback
Platform teaching support system	Exchange interaction	Through live broadcast, group work, homework, and other modules; promotes interaction between students and teachers, between students
	Teaching functions	This platform can carry out learning courses through live broadcast, recorded broadcast, PPT display and other ways
	Course management	Can achieve synchronous or asynchronous discussion, that is, through real-time video or audio conference. In addition, it can specify the permissions of relevant personnel of the course, such as teachers and students. Assignments can only be released through teachers, and teachers can only join the group through teachers. Remind students to sign-in in time
	Student status management	Online registration information function, and records each student’s study time

.

In order to quantify the above-mentioned primary and secondary indicators, setting subsequent indicator weight should be based on an indicator library that contains synonyms, opposites, and related words by user comments. The index library is traversed and the frequency of the words is used as the evaluation method. The next section will explain, in detail, the indicator library corresponding to each level indicator.

5.1.1. The Characteristics of a Platform

Most collected user reviews are focused on the evaluations for system compatibility, stability, response speed and so on. The system characteristic is the basic index for the evaluation of the online education platform quality, which reflects that the network hardware environment of the online education platform still has room for improvement in technical support and support services for learners. Accordingly, under the characteristics of the platform system, two secondary indexes, stability and compatibility, are set here. The stability of the system means that the website can be accessed successfully at any time without errors. System compatibility means that the online education platform can be used by different clients and received by different terminals. Examples of word libraries for stability and compatibility of the system are shown in

Table 8. Examples of word library for system stability.

Name	Example 1	Example 2	Example 3
Good evaluation of system stability	fluency	runs smoothly	stabilization
Poor evaluation of system stability	system crash	poor stability	Unable to jump to other interface

and

Table 9. Examples of word library for system compatibility.

Name	Example 1	Example 2	Example 3
Good evaluation of system compatibility	stabilization	well	diversity
Poor evaluation of system compatibility	at question	non-support	bad
A word relating to system compatibility	applicable version	suitability	fit measure

.

5.1.2. Support Service

Constructing the systematic evaluation index for an online education platform should consider the teaching environment which directly affects the user experience on the platform: whether the platform can provide students with valuable learning information or guidance to solve their problems in learning; and whether it can guide students to reasonably use the platform and the related learning tools. Based on this, a secondary index of technical and academic support is set under the characteristics of the platform system, which means that relevant customer service personnel provide technical support and learning guidance materials for learners. The examples of relevant word libraries are shown in

Table 10. Examples of word library for technical and academic support.

Name	Example 1	Example 2	Example 3
Good evaluation of technical and academic support	beneficial	efficient	excellent
Poor evaluation of technical and academic support	perform practically no function	no customer service	rubbish

.

5.1.3. Platform Video Quality

The problem of “video” was repeatedly mentioned in user comments after the outbreak of COVID-19. For example, while using video conference of DingTalk, there are problems such as “unsmooth picture”, “low picture quality”, “unable to adjust the volume”, “delayed video uploading and downloading” and so on. It can be seen that the continuous development of media technology has put forward higher requirements for the quality of course video on online education platforms. In terms of video information transmission, if students cannot receive information in time, their learning enthusiasm will be greatly reduced. Based on this, this paper sets three secondary indexes to assess video quality, namely, the quality of the video picture, the quality of the video sound, and the timeliness of the video information transmission. Examples of relevant word libraries are shown in

Table 11. Examples of word library for video picture quality.

Name	Example 1	Example 2	Example 3
Good evaluation of video picture quality	fluency	distinct	free from inhibitions
Poor evaluation of video picture quality	slur	low quality	blank screen

,

Table 12. Examples of word library for video sound quality.

Name	Example 1	Example 2	Example 3
Good evaluation of video sound quality	resonant	clear	fluency
Poor evaluation of video sound quality	noise	the sound is serious	off and on

and

Table 13. Examples of word library for video information transmission.

Name	Example 1	Example 2	Example 3
Good evaluation of timeliness of video information transmission	good real-time	not kartun	fluency
Poor evaluation of timeliness of video information transmission	kartun	motionless	delay

.

5.1.4. Requirement of Platform Technology

The network technology of the platform refers to the way of information communication between learners and the online education platforms, among which the effective and reliable Internet technology is the evaluation standard for the advantages and disadvantages of the online education platform, and the simple and user-friendly interface design is an important factor determining the quality of online education and affecting the user’s sense of experience. Therefore, the second-level indexes required by platform technology include interface design, access speed, navigation link, security, reliability, etc., and the examples of relevant word libraries are shown in

Table 14. Examples of word library for interface design.

Name	Example 1	Example 2	Example 3
Good evaluation of interface design	concise	practical	as plain as print
Poor evaluation of interface design	clutter	anti-human	esthetic effect

,

Table 15. Examples of word library for access speed.

Name	Example 1	Example 2	Example 3
Good evaluation of access speed	good speed	amazed	fluency
Poor evaluation of access speed	unable to load	slowing	load along while

,

Table 16. Examples of word library for navigation link.

Name	Example 1	Example 2	Example 3
Good evaluation of navigation link	logical	meticulous	distinct
Poor evaluation of navigation link	laborious	a mere skeleton	unable to browse

,

Table 17. Examples of word library for security.

Name	Example 1	Example 2	Example 3
Good evaluation of security	covert	Privacy	safety enhancement
Poor evaluation of security	insecure	violate human rights	lack of privacy

and

Table 18. Examples of word library for reliability.

Name	Example 1	Example 2	Example 3
Good evaluation of reliability	awesome	steady	good
Poor evaluation of reliability	no one with a hundred	crash	login not on

.

5.1.5. Platform Teaching Support System

Jun and Yong [42] described the general structure and main functionalities of the open source web-based teaching management system, and incorporated student data, course management, teaching data collection and configuration management of the system into the contents of the online teaching management system. This paper draws on this literature and combines the questions raised by user comments to set four second-level indicators, namely, communication and interaction, teaching function, course management and student status management. The examples of relevant word libraries are shown in

Table 19. Examples of word library for exchange interaction.

Name	Example 1	Example 2	Example 3
Good evaluation of exchange interaction	convenience	speediness	can video
Poor evaluation of exchange interaction	shade screens	reply is swallowed	automatically open mic

,

Table 20. Examples of word library for teaching functions.

Name	Example 1	Example 2	Example 3
Good evaluation of teaching functions	convenience	stabilization	waste
Poor evaluation of teaching functions	troublesome	waste	mess around

,

Table 21. Examples of word library for course management.

Name	Example 1	Example 2	Example 3
Good evaluation of course management	covert	privacy	clear course classification
Poor evaluation of course management	don’t remind	time box	harass

and

Table 22. Examples of word library for student status management.

Name	Example 1	Example 2	Example 3
Good evaluation of student status management	attention	switching frequency	the background records
Poor evaluation of student status management	cannot modify	not sure	discrepancy

.

5.2. Indicator Scoring Method

In the previous section, different word libraries were established for the second-level index. In this section, the occurrence frequency of words was taken as the initial score of this index.

Generally speaking, a second-level index consists of two word libraries: positive and negative comments [43]. The second-level index of the system is evaluated by “(frequency of occurrence of positive words—frequency of occurrence of negative comments)/total number of comments on the platform”. However, when the universality of evaluated words is relatively high, the related words library is added, and the form of “related words + adjectives” is taken as the indicator library. For example, the word describing the poor system compatibility is “problematic”, which has strong universality and can also be used to describe other indicators. Therefore, the related word, “version adaptation” is added. Only when two words, “version adaptation” and “problematic” appear in the comment, it can be judged that the comment describes the poor system compatibility. Therefore, when a second-level index contains three vocabularies: positive comment, negative comment and related word, the second-level index of the system is evaluated by “(positive comment frequency containing related word—negative comment frequency containing related word)/total comment number of the platform”.

After obtaining the scores of each indicator on each platform through the above methods, the data is processed and the preliminary scores are obtained.

5.3. Index Weight Setting

After obtaining the preliminary scores, it is necessary to set the weight of each indicator, so as to obtain the total score of each platform. The setting methods of index weights are divided into subjective and objective weighting methods. Subjective weighting methods include expert scoring method and Delphi method, etc., while objective weighting methods include entropy weighting method and coefficient of variation method. Since the experimental data are directly obtained from the network with large amount of data and poor hierarchy, and the objective weighting method does not rely on subjective judgment of human beings and has strong mathematical theoretical basis, the objective weighting method is adopted here to set the index weight. By comparing the weight results of two objective weighting methods—coefficient of variation and entropy method—the coefficient of variation method is used to weight the experimental results.

5.3.1. Coefficient of Variation Method

Coefficient of variation method [44] is a method to calculate the variation degree of each index in the system by directly using the information contained in each index, and it is an objective weighting method. In the evaluation index system, the greater the difference in the value of the same index between different platforms, the greater the weight of the index; the smaller the difference in the value of the same index between different platforms, the smaller the weight of the index.

Suppose that the target vector and each index vector are constructed into a matrix M = (index 1, index 2..., index n) = {X₁, X₂, ..., X_n}. The coefficient of variation of the ith evaluation index is calculated as follows:

$$C{V}_i=\frac{D}{\overline{x_i}}$$

(1)

where D is the standard deviation of the i evaluation index. The coefficient of variation was normalized, and then the weight of coefficient of variation of each evaluation index can be obtained.

The weights of secondary indexes before and after the outbreak of COVID-19 by coefficient of variation method are shown in

Table 23. The weights of secondary indexes before the outbreak of COVID-19 by coefficient of variation method.

First-Level Index	Second-Level Index	Index Weight
Platform system characteristics	System stability	0.056780
	System compatibility	0.062532
Background customer service support	Technical and academic support	0.057641
Platform video quality	Video picture quality	0.050599
	Video sound quality	0.050884
	Timeliness of video information transmission	0.083637
Platform technical requirements	Interface design	0.063798
	Access speed	0.135036
	Navigationlink	0.048805
	Security	0.059239
	Reliability	0.112449
Platform teaching support system	Exchange interaction	0.070685
	Teaching functions	0.048894
	Course management	0.048957
	Student status management	0.050066

and

Table 24. The weights of secondary indexes after the outbreak ofCOVID-19 by coefficient of variation method.

First-Level Index	Second-Level Index	Index Weight
Platform system characteristics	System stability	0.061711
	System compatibility	0.054056
Background customer service support	Technical and academic support	0.082499
Platform video quality	Video picture quality	0.075169
	Video sound quality	0.056147
	Timeliness of video information transmission	0.077557
Platform technical requirements	Interface design	0.082348
	Access speed	0.062215
	Navigationlink	0.000000
	Security	0.067109
	Reliability	0.068368
Platform teaching support system	Exchange interaction	0.082622
	Teaching functions	0.063522
	Course management	0.102191
	Student status management	0.064485

, and the numbers marked in red are the three secondary indexes with the highest weights before and after the outbreak of the pandemic.

5.3.2. Entropy Weight Method

Entropy was first introduced into information theory by Shannon [45], and has been widely applied in engineering technology, social economy and other fields. Information measures the degree of system order when entropy measures the degree of system disorder. For an index, its dispersion degree can be judged by entropy value. The smaller its information entropy is, the greater its dispersion degree is, the more information it provides, and the greater its role and weight it can play in the comprehensive evaluation.

Suppose there are k indicators and n sets of data. First, the target vector and each indicator vector are constructed into a matrix M = (indicator 1, indicator 2..., the indicator k) = {X₁, X₂,..., X_k}, where X_i = {x₁, x₂,..., x_n}. According to the definition of information entropy in information theory, the information entropy of a group of data is calculated as follows:

$$E_j=-ln{\left(n\right)}^{-1}\cdot {\displaystyle \sum _{i=1}^n{p}_{ij}}\cdot ln{p}_{ij}$$

(2)

where p_ij is the proportion of standardized data in this index data. Then the weight of each index can be determined according to the information entropy.

The weights of secondary indexes before and after the outbreak of COVID-19 by entropy weight method are shown in

Table 25. The weights of secondary indexes before the outbreak of COVID-19 by entropy weight method.

First-Level Index	Second-Level Index	Index Weight
Platform system characteristics	System stability	0.035310
	System compatibility	0.030881
Background customer service support	Technical and academic support	0.050035
Platform video quality	Video picture quality	0.044019
	Video sound quality	0.031988
	Timeliness of video information transmission	0.051665
Platform technical requirements	Interface design	0.049066
	Access speed	0.036606
	Navigationlink	0.383216
	Security	0.039662
	Reliability	0.041621
Platform teaching support system	Exchange interaction	0.052509
	Teaching functions	0.037286
	Course management	0.079421
	Student status management	0.036716

and

Table 26. The weights of secondary indexes after the outbreak of COVID-19 by entropy weight method.

First-Level Index	Second-Level Index	Index Weight
Platform system characteristics	System stability	0.052743
	System compatibility	0.061028
Background customer service support	Technical and academic support	0.052945
Platform video quality	Video picture quality	0.046264
	Video sound quality	0.046546
	Timeliness of video information transmission	0.101660
Platform technical requirements	Interface design	0.059589
	Access speed	0.145680
	Navigationlink	0.044719
	Security	0.055884
	Reliability	0.128727
Platform teaching support system	Exchange interaction	0.068772
	Teaching functions	0.044794
	Course management	0.044847
	Student status management	0.045801

. The numbers in red are the three secondary indicators with the highest weights before and after the outbreak of COVID-19.

5.3.3. Method Comparison

The coefficient of variation method and entropy weight method are both objective. According to the calculation results, users’ attention to the platform changed before and after the outbreak of the pandemic. However, there are differences in index weights between the two approaches.

The coefficient of variation method calculates the weight according to the coefficient of variation of the index. Table 23 shows the weights of secondary indicators before COVID-19. Three secondary indicators with the highest weight are access speed, reliability, and timeliness of information transmission, indicating that users are more concerned about the smooth use of the platform and basic technical problems. Table 24 is the secondary index weight after the outbreak of COVID-19. The highest three secondary index weights are course management, interaction, learning and technical support, respectively, indicating that user attention is shifting from basic technical problems to the learning function.

The entropy weight method calculates the weight according to the information entropy of the index. Table 25 shows the weights of secondary indicators before the outbreak of COVID-19. Three secondary indicators with the highest weight at this time are navigation links, course management, and communication and interaction, among which the weight of navigation links is much higher than that of other indicators. However, only one comment about navigation links appeared in WeChat Work platform; other platforms have not mentioned this indicator. This is because the entropy weight method amplifies the differences between the index data, leading to extremely high weight of navigation links, but its weight does not match common sense. In general, the pros and cons of the navigation links are unlikely to be the key factors for user experience. In addition, the high weights of course management, and communication and interaction, indicate that users pay more attention to the learning function of the platform. Table 26 shows the weights of secondary indicators after the occurrence of COVID-19. At this time, three secondary indicators with the highest weights are access speed, reliability, and timeliness of video information transmission, indicating that users pay more attention to the technical problems of the platform at this time. Compared with before and after the outbreak of the pandemic, the attention of users under the entropy weight method shifted from the learning function of the platform to the technology of the platform.

In fact, since the outbreak of COVID-19 disrupted normal study plans, the original face to face learning has shifted to online. More and more teachers and students are using online education platforms now, so users pay more attention to a platform’s functionalities, good teaching environment, and effective implementation of learning programs. Based on this, the entropy weight method is not reasonable, while the coefficient of variation method is reasonable. Therefore, the coefficient of variation method is finally selected as the setting method of index weight.

5.4. The Analysis of Evaluation Results

5.4.1. Overall Result Analysis

Due to the low market demand, the number of users for online education platforms was relatively small before the outbreak of COVID-19. DingTalk and Tencent Meeting did not develop an online education function. Therefore, compared with learning functions of the platform, users are more focused on smooth use and technical problems. After the outbreak, because market demand of online education and the number of online platform users rose rapidly, mobile office platforms such as DingTalk and Tencent Meeting also successively introduced the function of online teaching. Due to their competitive parent companies, DingTalk and Tencent Meeting have little technical problems, so users pay attention to the functionalities of online study. In terms of teaching support system, which represents the learning function of the platform, it can be further seen that users pay more attention to communication, interaction, and course management than teaching functionalities and school role management. Teaching functionalities refer to live broadcast, video broadcast, etc. All the platforms selected here have played an online education role during the pandemic period and all have basic functionalities such as live broadcast, video broadcast, etc. Therefore, there is little difference in this index and the index weight is naturally low. Student status management refers to the modules such as online registration, recording lecturing, etc. If the duration of video playing cannot be accurately calculated, it may lead to low scores of students and other situations damaging the user experience. Among the seven platforms, only MOOC and Chaoxing Learning are conducted through course recording, and the learning time needs to be recorded by calculating the video playing time. The other five platforms mostly teach through live streaming, so the demand for learning time recording and the weight of this index is relatively low.

Interaction is effective to improving teaching quality. Group discussion, attendance check by asking questions, and raising hand to speak, promote students’ learning enthusiasm and help instructors understand students’ learning degree and disabuse students of questions. Due to the space distance of online teaching, the interaction between teachers and students is difficult to carry out. Therefore, in order to ensure the quality of online learning, the design of communication and interaction functions is extremely important for the user experience of the online education platform during the pandemic. If the platform can provide rich interactive functionalities, it will be favored by more teachers and students.

Classroom management ensures the orderly development of the teaching plan. The setting of permissions in course management ensures the difference between the identities of online users and clarifies the difference between teachers and students. In course management, the function of reminding students to check in restores the offline scene of class bell ringing and improves the learning atmosphere. Therefore, course management is relatively important.

To sum up, after the outbreak of the pandemic, there is a huge market need for online education platforms. Course management, communication and interaction are the key factors affecting user experience.

5.4.2. Results Analysis of Each Platform

Based on a set of evaluation index system of the user experience, 5 primary indexes including platform system characteristics, customer service support, video quality, technical requirements, the platform teaching platform support system, and 15 secondary indexes including stability and compatibility of system, and user comments are weighted through variation coefficient method, and scores and ranks of user experience are shown in

Table 27. Scores and ranks of user experience before the outbreak of COVID-19.

Rank	1	2	3	4	5	6	7
Platform Name	Zoom Cloud	Tencent Meeting	DingTalk	MOOC	TIM	WeChat Work	Chaoxing
System stability	0.84	0.75	0.79	1.00	0.00	0.43	0.79
System compatibility	0.90	0.77	1.00	0.80	0.27	0.84	0.00
Technical and academic support	0.89	1.00	0.51	0.78	0.00	0.69	0.55
Video picture quality	1.00	0.87	0.91	0.74	0.76	0.85	0.00
Video sound quality	0.71	1.00	0.95	0.91	0.80	0.84	0.00
Timeliness of video information transmission	0.72	0.46	1.00	0.54	0.70	0.02	0.00
Interface design	1.00	0.72	0.43	0.63	0.00	0.38	0.77
Access speed	0.19	0.20	0.19	0.00	1.00	0.12	0.19
Navigationlink	1.00	1.00	1.00	1.00	1.00	0.00	1.00
Security	1.00	0.85	0.72	0.86	0.00	0.39	1.00
Reliability	1.00	0.34	0.33	0.10	0.08	0.00	0.49
Exchange interaction	1.00	0.77	0.27	0.00	0.36	0.69	0.63
Teaching functions	1.00	0.95	0.99	1.00	0.95	1.00	0.00
Course management	1.00	0.93	0.95	0.96	0.91	0.94	0.00
Student status management	0.78	0.89	1.00	0.86	0.96	0.82	0.00
Total Score	99.78	70.21	65.32	46.14	33.48	20.22	0

and

Table 28. Scores and ranks of user experience after the outbreak of COVID-19.

Rank	2	3	1	5	6	4	7
Platform Name	Zoom Cloud	Tencent Meeting	DingTalk	MOOC	TIM	WeChat Work	Chaoxing
System stability	1.00	0.60	0.97	0.00	0.54	0.76	0.73
System compatibility	1.00	0.85	0.98	0.85	0.00	0.94	0.84
Technical and academic support	1.00	0.78	0.67	0.34	0.00	0.37	0.30
Video picture quality	1.00	0.58	0.74	0.36	0.35	0.53	0.00
Video sound quality	0.71	0.00	1.00	0.84	0.84	0.70	0.74
Timeliness of video information transmission	0.90	0.72	1.00	0.13	0.86	0.64	0.00
Interface design	1.00	0.47	0.94	0.33	0.00	0.41	0.43
Access speed	0.93	0.96	1.00	0.42	0.94	0.96	0.00
Navigationlink	1.00	1.00	1.00	1.00	1.00	1.00	1.00
Security	0.58	0.79	0.87	1.00	0.37	0.00	0.97
Reliability	0.95	0.84	1.00	0.28	0.82	0.73	0.00
Exchange interaction	0.40	1.00	0.56	0.72	0.00	0.94	0.19
Teaching functions	0.41	0.00	0.91	1.00	1.00	0.73	0.92
Course management	1.00	0.65	0.67	0.39	0.01	0.32	0.00
Student status management	0.46	0.55	1.00	0.00	0.70	0.57	0.70
Total Score	91.18	54.44	99.79	17.07	9.18	44.69	0

. In the matrices, the green font indicates the data with the highest score, while the red font represents the data with the lowest score in each row.

As can be seen from Table 27, before the occurrence of COVID-19, the ranking of user experience of each platform from high to low is Zoom Cloud, Tencent Meeting, DingTalk, MOOC, TIM, WeChat Work, and Chaoxing Learning. It can be seen from Table 28 that after the occurrence of COVID-19, the ranking of user experience of each platform by descending order is: DingTalk, Zoom Cloud, Tencent Meeting, WeChat Work, MOOC, TIM, and Chaoxing Learning.

This change is closely related to initiatives taken by education platforms during the pandemic. As an online office software, DingTalk quickly identified its new position, providing a complete set of solutions for online education of all kinds of schools. It also developed functions such as online classroom and health tasks to respond to “school suspension without class suspension”. Although the application of teaching functions was opposed by students who were tired of learning at the beginning, compared with other platforms, its performance during the pandemic was more prominent. The platform had higher stability and compatibility, relatively simple interface design, complete teaching functions, and was favored by teachers and students. Similarly, WeChat Work was originally an online office software, but it also helped with online teaching during the pandemic. WeChat’s group broadcast supports online teaching, while the pandemic prevention collection form of micro-documents helps schools manage students’ health information. The improvement of its teaching functionalities and the reliability of the platform have also won many positive comments. In contrast, as an educational software, Chaoxing Learning could have played a greater role during the pandemic, but its performance was always atthe bottom in terms of user experience score. The reasons for this situation are closely related to the imperfect learning function and low access speed of the platform. The other four platforms also showed slight changes in their rankings, but the differences were not significant.

The performance of each platform on each indicator will be analyzed in detail below.

(1) Zoom Cloud

Zoom Cloudperformed well overall, ranking the 1st before the outbreak ofCOVID-19 and 2nd after, respectively. Before the pandemic, Zoom Cloud’s access speed was the only disadvantage compared with other platforms. Users had a poor experience in this indicator. After the outbreak of the pandemic, Zoom Cloud has greatly improved its access speed, and users’ attention has been transferred to its teaching functionalities. However, Zoom Cloud needs to improve its communication and interaction, teaching functionalities, and student status management. Due to its relatively stable system, simple interface, and easy operation, Zoom Cloud was highly praised during the pandemic period. However, Zoom Cloud did not develop teaching functionalities, instead, it was still a mobile office software for video conferencing and content sharing. It is more suitable for team meetings, where everyone can express and share their views. However, the interaction between teachers and students in teaching is still dominated by teachers’ teaching and supplemented by students’ participation. Therefore, the communication and interaction between teachers and students do not exactly meet the needs of teaching. In addition, users of Zoom Cloud do not distinguish between teachers and students, so it is deficient in teaching functionalities and school role management.

(2) Tencent Meeting

Tencent performed better overall, ranking the 2nd before the outbreak ofCOVID-19 and 3rd after, respectively. Before the pandemic, its access speed, reliability, and timeliness of video information transmission were poor, indicating the existence of problems such as lag and video fluidity. After the outbreak of the pandemic, Tencent Meeting has significantly improved in the above three indicators, but the quality of video sound and the teaching functionalities are deficient. In terms of the quality of video sound, many users put forward problems such as “there is no sound on the recording screen” and “the sound volume cannot be adjusted”. In terms of teaching functionalities, Tencent Meeting has the similarity with Zoom Cloud: as video conferencing software, both of them can only satisfy the need of video teaching, and are unable to count the number of students entering the meeting and record the time of entering meeting, without distinguishing students’ and teachers’ type, resulting in low score in the interaction, teaching functionalities, and student status management.

(3) DingTalk

DingTalk performed better overall, ranking 3rd before the outbreak ofCOVID-19 and 1stafter, respectively. Before the outbreak of the pandemic, the three lowest indexes were communication and interaction, reliability, and interface design. After the outbreak of the pandemic, the reliability and interface design scores increased, but the communication and interaction scores were still low. Although DingTalk was originally a mobile office software, it was successfully transformed and introduced functionalities related to teaching during the pandemic. Thanks to Alibaba Group, 140,000 servers were deployed to ensure the normal operation of the system during the pandemic. In terms of communication and interaction, when the teacher gives a live lecture, students can express their views through the message window. However, the message window is somewhat delayed and the interaction process is not smooth enough. Therefore, although DingTalk performs well compared with other platforms, it is still deficient in communication and interaction.

(4) MOOC

MOOC performed mediocrely overall, ranking 4th before the outbreak ofCOVID-19 and 5th after, respectively. Before the pandemic, MOOC was ranked last in terms of speed, interaction, and reliability. After the outbreak of the pandemic, the interaction and communication have been improved, but the stability of the system, the timeliness of video information transmission, reliability, and school role management have been criticized. Positioning an online education platform, however, MOOC does not aim to provide online learning services to students in school, but to provide an equal learning platform for all, including students and office workers. As a result, students on the platform are learning via video classes from existing elite schools, rather than via live streaming. With video recording method, it has rare interaction. After the outbreak of COVID-19, a large number of students are using MOOC to watch course video. The watching time becomes the important factor to measure student learning conditions for schools. However, due to frequent unsynchronized learning record and low sensitivity, it greatly impacted on the student status management and reduced user satisfaction. The technical problems of the platform have existed before and after the outbreak of the pandemic.

(5) TIM

TIM performed poorly overall, ranking 5th and 6th before and after the outbreak ofCOVID-19. Before the pandemic, it ranked the last in terms of system stability, platform learner support, interface design and security, but it performed well in access speed. After the outbreak of the pandemic, the platform had serious problems in five aspects: system compatibility, platform support for learners, interface design, communication and interaction, and course management. TIM is known as the concise version of Tencent QQ, which can sync QQ friends with low entry threshold. However, its positioning is also suitable for teamwork office software, which has the same problems as Zoom Cloud and Tencent Meeting. At the same time, TIM also performs poorly in terms of technology and system compatibility, which is lack of a suitable version of the tablet and delays in updating, etc. As one of Tencent’s office software, with no advertising push, simple design interface, and practical function, TIM enables Tencent to provide an opportunity to capture the market in the field of mobile office. Without clear profit as well as appearance of pure mobile office software such as DingTalk, TIM is declining.

(6) WeChat Work

The overall performance of WeChat Work is mediocre, ranking 6th and 4th before and after the outbreak ofCOVID-19. Before the pandemic, it ranked last in reliability and navigation links, and performed poorly in timely video information transmission. WeChat Work is the only platform that has problems on navigation links, with critics pointing out that it contains too many advertising links. After the outbreak of the pandemic, WeChat has improved its scores in various indicators. The main problem has shifted to security, and its privacy protection for users is insufficient. Like TIM, WeChat Work is equivalent to office WeChat. Besides live conference, it also undertakes daily information communication, clocking, approval, and other functions. Therefore, as a teaching software, its teaching functionalities are not complete enough. In addition, most users believe that WeChat is a private space, while the chat records of staff or students on WeChat can be viewed by superiors, and the moderator of the meeting can open the microphone of others at will. The setting of these functions is a kind of privacy interference behavior, so users have raised many doubts about the security of this platform.

(7) Chaoxing Learning

Chaoxing Learning had the lowest overall performance, ranking last before and after the outbreak ofCOVID-19. Before the pandemic, Chaoxing Learning performed poorly in three first-level indicators, including system compatibility and video picture quality, including platform system characteristics, video quality and teaching support system. After the outbreak of the pandemic, its teaching functionalities have been improved, but it still ranks the bottom in five secondary indicators, including the quality of video images, timeliness of video information transmission and course management, and there are still many limitations. Chaoxing Learning is a mobile learning platform, which provides electronic literature review, course learning and group discussion functionalities. However, technical problems often occur on the platform, and the platform has been complained of many times due to the system crash during the pandemic. In addition, when the system crashed, the official reply of Chaoxing Learning was that “efforts are being made to repair it, and we call on more students to learn from the wrong peak, so that the stability will be significantly improved”, which shows the weakness of the technical support behind Chaoxing Learning. Due to the limitations of the technology, it also performs poorly in teaching functionalities such as school role management. Many users commented that they were “unable to sign in” and were defined as absent by the system, as “failure to sign in after the wrong peak is still considered as absenteeism, and the homework notification cannot be received”, etc.

6. Conclusions

Through this research, we hope to make up for the few literatures on the differences of online education platforms in case of emergencies, so as to improve the education level from more perspectives. This paper collected online user experience data of the online education platforms before and after the outbreak of COVID-19, and obtained the change of user experience focus by analyzing the data before and after the outbreak. On one hand, the relevant characteristics analysis of online user reviews before and after the outbreak of COVID-19 is carried out, emotional tendencies of user reviews in the two stages are compared, and the method of text similarity statistics are used to retrieve the hot issues in the user reviews. The study found that during COVID-19, the users of the platforms have different concerns and requirements, and there are some similar problems between each platform such as kartun and flashback. According to the existing literature and the characteristics analysis results for the comments from online education platforms, an evaluation index system was constructed. Meanwhile, the coefficient of variation weighting method was used to comprehensively evaluate the 15 secondary indicators. The results show that before the outbreak of the pandemic, users were concerned about the access speed, reliability, and timeliness of video information transmission of the platform, and the user experience of the Zoom Cloud platform was the best. After the outbreak of the pandemic, users mainly focused on course management, communication and interaction, learning and technical support services of the platform, and the user experience of the platform was the most important. Overall, Chaoxing Learning had the poorest user experience and DingTalk performed best.

Based on the above analysis, this paper summarizes the following suggestions, which are expected to improve the user experience of the online education platform during COVID-19:

(1) Improving support service

Providing comprehensive, timely, convenient and fast support services for learners is conducive to the maintenance of learners’ positive learning attitude. In online education courses, there are several problems: untimely video information transmission, slow platform access, and untimely response to questions in class, students’ hope to get corresponding guidance after class, and teachers’ opinions in a timely manner so as to improve their learning effect. Therefore, the delay of feedback should be shortened and the variety of feedback forms should be ensured. In addition, in the online class, there will be more accidents, such as the failure of voice connection, the teachers’ unfamiliarity with the software interface, unable to skillfully operate the software, and diagnosis problems. Moreover, the hardware problems cannot be solved. For example, the microphone cannot be heard or spoken, and even lead to classroom suspension, which wastes a lot of time. Therefore, the platform should simplify the software design and configuration, and provide corresponding customer service to solve user problems at any time. On the other side, the network problem of the platform needs to improve its own technology.

(2) Improving the convenience of interactive communication

While conducting online classroom teaching on the platform, sometimes it is not convenient for learners to interact and communicate with teachers. It is suggested that the platform should be designed in a split screen so that users can simultaneously interact with the platform, thus timely and effectively share and interact with information resources.

(3) Optimizing ease of use

For the problems regarding stability, security and compatibility of the platform, and the invisibility for homework assignment and submission for mobile learning, as well as the block of pop-up questions for video learning, it is suggested that the platform should continue to be optimized.

(4) Enriching platform resource

It is suggested that the platform should provide extended learning resources for users to ensure that the resources cover all disciplines. In addition, more course activities can be added to the platform to continuously improve the enthusiasm of learners.