The State of the Art in Methodologies of Course Recommender Systems—A Review of Recent Research

Abstract

In recent years, education institutions have offered a wide range of course selections with overlaps. This presents significant challenges to students in selecting successful courses that match their current knowledge and personal goals. Although many studies have been conducted on Recommender Systems (RS), a review of methodologies used in course RS is still insufficiently explored. To fill this literature gap, this paper presents the state of the art of methodologies used in course RS along with the summary of the types of data sources used to evaluate these techniques. This review aims to recognize emerging trends in course RS techniques in recent research literature to deliver insights for researchers for further investigation. We provide a systematic review process followed by research findings on the current methodologies implemented in different course RS in selected research journals such as: collaborative, content-based, knowledge-based, Data Mining (DM), hybrid, statistical and Conversational RS (CRS). This study analyzed publications between 2016 and June 2020, in three repositories; IEEE Xplore, ACM, and Google Scholar. These papers were explored and classified based on the methodology used in recommending courses. This review has revealed that there is a growing popularity in hybrid course RS and followed by DM techniques in recent publications. However, few CRS-based course RS were present in the selected publications. Finally, we discussed future avenues based on the research outcome, which might lead to next-generation course RS.

1. Introduction

Recently, we have witnessed an enormous research expansion in Recommender Systems (RS). RS are information filtering systems which provide facilities to perform predictions. They are beneficial to many industries, such as book recommendation (Amazon), Google Play Store Apps suitable for the user, YouTube videos and music recommendation of Apple Music [1]. Among these, RS for course selection are more complex, which will provide support for students to find an optimal path to accomplish their learning goals. The primary focus of this research article is to investigate different approaches used to develop and assist students to select the most appropriate degree programs or subjects which suit their preferences. Effective career counselling services received by a student are crucial to their overall academic performance and it can make a countless impact on their career success. These counselling decisions should incorporate several factors such as perceived employment opportunities, student interests, academic results, attitudes and aptitudes and therefore, course selection is challenging.

Challenges to Course Selection

The problems associated with course selection may vary from situation to situation. For example, in a university setting there are varieties of courses available and problems arise if students have limited knowledge of available courses. Similarly, course selection should also need to deal with a student’s career interests and aptitude. Additionally, the system should need to handle changes in course structures and circumstances of a specific student [2,3,4]. Furthermore, academic institutions may offer different course modules with overlaps and prerequisite. Hence, course RS should address these issues by defining suitable constraints to course selection functions.

Every academic organization has career guidance counsellors or advisors who have several years of experience in the field. However, curriculum reviews and changes in course structures require all human advisors to be adept in all foreseeable future changes. Furthermore, this academic advising is time-consuming, mentally exacting and often inundated with bundles of student requests with the growing number of students and courses. Moreover, academic requirements at universities are complex, students find it difficult to select their preferences just by reading the course titles or descriptions provided in course catalogues. Moreover, awareness of the prerequisite in course structure is also critical to student’s achievement in the course progress [5,6,7]. Hence, it is desirable to have an effective RS that will enhance the quality of academic advising and the student satisfaction by minimizing the cognitive load and the time spend by an academic advisor to deliver the course recommendation. From the perspective of the university, enhancement of student’s satisfaction and maintaining strong relationship between the student and the university, are the long-term goals which contribute mainly to intensify the student retention. This manifests the importance of having a system which advises and supports a student to select a course that was taken successfully by the students who had the similar preferences and academic performance. It is important and challenging to design a system that matches preferences and academic performance of a potential student user with the existing students in the system [8,9]. These RS will fill the gap between the information collection and presenting the most important information to the active users by filtering the required information.

The other major challenges in implementing course RS are sparsity and scalability of the rating matrix [8,9,10]. Sometimes, students will not provide rating or preferences to these systems which may lead to weak recommendation due to missing values/ correlation in the rating matrix. The scalability and diversity are other issues when the numbers of users and items/courses are increasing. The cold-start issue is another problem in RS, which occurs when a new item or new user is added to the matrix and insufficient rating data is available in the system. Another challenge for this system is a lack of similar users or few users exist in the system who liked similar items, which is called neighbor transitivity. Hence, recommendations or predictions will not be able to generate from the systems based on the similar users or items. Farzan and Brusilovsky [11] explored challenges to course RS especially in the context of user participation which can be encouraged by interpersonal factors; liking, affiliations and individual factors; rewards, personal needs and intrinsic motivation. Furthermore, authors developed community-based RS called CourseAgent to address these challenges [11].

The remainder of the paper is organized as follows. Section 2 reviews the techniques used in RS in general and related research. Section 3 outlines the systematic review process we followed during this review work. Section 4 summarizes outcome of our systematic review. Section 5 concludes the study with some suggestions to future avenues.

2. Background and Related Work

RS are information filtering systems which support users in decision making process on products, services or information that they may be interested in [12]. Inputs to RS are mainly:

• Demographic data: age, gender, education, and location of the user
• Content of documents: textual analysis
• User ratings: opinions of users on specific items

In general, several approaches, have been used to develop RS such as: Collaborative Filtering (CF) [10,13,14,15,16,17,18], content-based [1,19,20], knowledge engineering [21,22,23,24,25,26,27,28], hybrid approaches [27,29,30,31,32,33,34,35], Associative Rule Mining (ARM) [36,37,38], fuzzy logic-based [39,40,41,42,43,44,45], Machine Learning (ML) approaches [46,47] and Conversational RS (CRS) [26,48,49,50,51,52,53]. However, not much research has been conducted on systematic reviews of recommendation techniques applied to course RS. In the following section, we present a detail review of RS approaches in general followed by some examples to show how they are applied in course RS. We will start the review with Content-Based RS (CBRS).

2.1. Content-Based Recommender System (CBRS)

CBRS recommends items to the user based on the description of the item and items that are interest to the user or similar to those the user liked in the past. Generally, user preferences are compared against the contents of the items. Characteristics of the user interest can be extracted explicitly from questionnaires or implicitly by learning user’s transactional behavior over time [12,20]. However, in the research literature some systems describe complex item representations; for example, Watters and Wang [54] represented a news item with set of headers, content and behavior attributes, for online news article recommendation and components and data flow in a CBRS is described in [55]. Contemporary content-based course RS incorporates hybrid techniques to build user profiles [32,56,57] and statistical analysis or ML techniques [58] to make recommendations. CBRS also uses Term frequency, Inverse Document Frequency and Rocchio’s method to determine the relative importance of a document or item-profile, but these techniques fail to keep semantics in the description [12].

Since these systems use user’s own rating as a factor to influence the future recommendation, a pure content-based approach has several fundamental drawbacks [12,20]. Some of the problems associated with CBRS are limited content analysis, difficult to apply non-textual data, overspecialization by recommending only items that bear strong resemblance to those the user already knows and sparsity of data [10,20]. In some complex domains such as movies, music, and restaurants, extraction of useful features is not possible with current technologies. Even from text documents certain aspect of the text can be extracted, many other factors that influence the user’s experience are missing [12,59]. To overcome problems associated with over specification, some systems discard high-relevance items from the list of recommendations.

Content-based course RS for liberal arts education is presented in [56]. The proposed approach applied topic model called Latent Dirichlet Allocation (LDA) to infer the studentś academic interests from the key words they entered into the system and used the regression models to predict the grades. However, this system is suitable for recommending courses for existing students as system needs the studentś past academic performance and level of academic expertise on the topic to produce recommendations. Content-based filtering and good learner rating have been applied in e-learning recommender system framework proposed in [2]. In this course RS, the average rating of the good learners was incorporated to improve the system performance. Content-based course RS using KNN, Naïve Bayes classification algorithms and user profiles have been proposed in [60].

2.2. Collaborative Filtering-Based Recommendation System (CFRS)

CF technique is the most mature and the most commonly implemented in RS [11,61]. MovieLens [62], Netflix prize data [63] and Jester [64] are the common databases where CF techniques are implemented. CFRS are based on the assumptions that people have similar preferences and interests that we can predict their choice according to their past preferences [10,13]. These systems need collaboration from different users to build a database (user-item matrix) of preferences for various items in the collection [17]. This user profile or rating matrix [10] is used to identify other users with similar taste in the past and use their opinion to recommend items to the active user with the similar taste.

Figure 1 demonstrates CF process with prediction and recommendation [14] by using a rating matrix. The first step of CF is to develop the user profile or rating matrix. Rows in the rating matrix represent users (users, $U=[u_1,u_2,\dots ,u_m]$ ) and column represents items ($I=[i_1,i_2,\dots ,i_n]$), and user rating value ($r_{u_i}$) is recorded at the intersection; $r_{u_i}$ represents score given to item I by the user u [10]. Rating matrix is sometimes sparse, as there may be no user rating values entered for certain items. User can explicitly enter, or the system can implicitly record by analyzing the user’s online activities, user logs or web links. The second step is to calculate the similarities between users and the neighbors to find similar users. Finally, calculate the rating for a particular item based on the weighted average of the ratings by the neighbors. Users can be grouped based on their preferences by using methodologies such as clustering [3,28]. Recommendations are based on the correlation between these clusters with the active user preferences. Active user $u_a(\in U)$ likeliness can be predicted or recommended as follows [14].

• Prediction: predicting a value, to item $I_j$ within the given scale, which describes the active user $u_a$ likeliness to the item $I_j$.
• Recommendation is recommending top N items $I_r(r$= $1,2,\dots ,N)$ to the active user, where $I_r\in I$. These N items should not include the items active user liked previously, $I_r\bigcap I_{u_a}=\varnothing$

In the literature, CFRS have been implemented in different application areas; GroupLens [65] is one of the earliest implementations which recommends netnews articles that the user will like, Large retailer Amazon.com recommends personalize items from their online store [35]. In addition, [17] provides a survey about recommendation algorithms and comparison of their performances and proposed CFRS can be improved by integrating natural language reviews as side information [15]. Almahairi et al. [15] introduced a model of product reviews to regularize rating prediction on the Amazon Reviews dataset.

CF systems have several drawbacks, such as cold-start problem, sparsity problem and gray sheep problem. Cold-start problem occurs when a new user tries to use the system, as the new user does not have rated items in the system or the new item that has not been rated. Gray sheep problem refers poor prediction due to lack of similar users with similar preferences [3,66]. The authors in [66] proposed a solution using K-means clustering algorithm offline, with different centroid selections and different distance measures. CFRS can be grouped into two general classes: memory-based and model-based [14,17]. Next section briefly presents these two categories of CFRS.

2.2.1. Memory-Based Collaborative Filtering Algorithm

There are two types of memory-based CF algorithms discussed in the literature, namely: user-based and item-based. This categorization depends on how user-item rating matrix is formed to obtain similarities [17]. Different item-based recommendation algorithms are analyzed in [14] and they suggest that item-based algorithms provide dramatically better performance than user-based algorithm.

Pearson correlation coefficient, Spearman’s rank correlation coefficient, Euclidean and mean squared differences can be used to calculate item-item similarity, to obtain the extent to which two items linearly relate with each other. In user-based systems, it has been reported that Pearson coefficient outperforms other measures and in item-based systems cosine similarity measure outperforms other measures [67]. Performances of these algorithms are mainly based on time takes to search for neighbors among a large population of potential neighbors. In the following section, we will focus on user-based and item-based collaborative systems.

User-Based Collaborative Filtering

User-based CF recommends items to the active user based on the user’s past preferences or on the opinions of their closest neighbors or similar users [10,14]. Next, user-based CF systems calculate similarities among users using some measures to find the similar users. There are several techniques that have been used to calculate similarities to extract neighbors, such as: Pearson correlation, cosine vector similarity, adjusted-cosine vector similarity, mean squared difference and Spearman correlation [14]. The Pearson’s correlation or Pearson R formula measures how strong the relationship is between two vectors of ratings [13]. Cosine similarity calculates the similarity between users and find their nearest neighbors [14].

The next step is to select user neighbors based on these similarities and then combine the neighbor users’ rating scores [10]. K-Nearest Neighbor (KNN) algorithm can be used to find the K-most similar users to the active user. By using these neighbor’s similarities and ratings, RS will recommend top-N most similar neighbors or calculate the weighted average for the prediction to the active user [17]. Sparsity and scalability are the potential challenges for user-based collaborative systems [14]. One possible solution to scalability issue is to use Latent Semantic Indexing, reduced dimensional space to increase the density of the rating matrix [28,68].

Item-Based Collaborative Filtering

In these systems, two items are considered to be similar when they received similar ratings from different users. The target item is compared with the set of similar items rated by the target user and compute how similar they are to the target item. Hence, the recommendation is a two-step process. First, find similar items by calculating similarity $S_{ij}$ between the co-rated item i and item j, by considering all users who rated both i and j [14,69]. Here two items are considered to be similar if many users rated two items similarly. There are several ways to calculate the similarity between 2 rating vectors, such as correlation-based similarity [14] and adjusted-cosine similarity. Most of the time rating matrix needs to be normalized to obtain optimal results. Once similar items are identified, next step is to compute the predictions by taking weighted average of rating given to most similar items by the active user.

User-based collaborative course RS, which uses the history of student course selection records and user behavior records is proposed in [70]. Farzan and Brusilovsky [11] highlighted importance of user participation in course RS and developed community-based course RS called CourseAgent. To encourage user participation, authors proposed interpersonal factors such as liking, affiliations and individual factors such as rewards, personal needs and intrinsic motivation.

2.2.2. Model-Based Collaborative Filtering Algorithm

Model-based approach use algorithmic approach to performs recommendations. These algorithms have been developed to overcome some limitations in data-intensive commercial RS such as prediction speed and over-fitting in the memory-based algorithms [68,71]. The model-based CF algorithms compress huge databases/dataset of ratings into a model which will positively contributes to the efficiency of prediction. These algorithms can be Data Mining (DM) algorithms including Machine Learning (ML), Deep Learning (DL) and Association Rule Mining (ARM) [3,68,72,73]. In the research literature these DM approaches are applied in course RS to personalize the recommendation after learning data relevant to the given user. These approaches incorporate mathematical or statistical analysis to draw conclusions. For example, model-based CFRS use DM techniques for model building process [61]. The model-based CFRS use dimensionality reduction approaches to generate recommendations such as; clustering, Matrix Factorization (MF), Singular Value Decomposition (SVD), DL algorithms, to alleviate data sparsity issue in these models [74]. If the user ratings are categorical, the model can be built by using classification algorithms, when ratings are numerical regression models can be used.

In recent years, ARM is one of the popular DM techniques use to extract frequent patterns, associations or correlations among the data items in large databases. Association rules represent knowledge in RS. The framework for ARM techniques applied in DM setting first introduced by Agrawal et al. [36]. These techniques are used in collaborative-based RS [3,38,75]. Collaborative RS incorporate ARM to improve the scalability, which may arise due to missing rating or preferences in the user-item matrix.

Laghari, et al. [76] proposed an Automated Course Advising System (ACAS) to undergraduates to plan their course registration. It provides visualization map of sequences of courses on which they need to register. Profile building is completed by the students by entering Grade Point Average (GPA), ID, degree major, current date and decide average credit hours of load per semester, then system chooses the semester load accordingly. Alternatively, a student is given a choice to calculate for a slow, normal, or a fast track of degree completion. Several test cases were discussed but it requires more test cases to discuss the performance. In [77] authors present user-based and item-based CFRS for recommending elective courses by predicting grades for these courses. It has been observed that more accurate results can be obtained from both methods when the neighborhood size is greater than 10.

2.3. Knowledge-Based Recommender System (KBRS)

CBRS and CFRS are easy to understand and widely applied in e-commerce [21,23,78]. However, precision of CFRS is very sensitive to the number of ratings in rating matrix and CFRS do not consider personal characteristics of the user or the domain knowledge. The knowledge-based approaches use domain knowledge (knowledge about users and products) to generate recommendations which meet user preferences [21,22,25]. The main drawbacks in KBRS are difficulty of acquisition and representing the domain knowledge in a machine-readable structure and it needs extensive efforts to extract the knowledge. However, KBRS are more appropriate for designing course RS which need complex knowledge domain to make recommendations.

There are two types of retrieval modes in KBRS [21]; similarity-finding and tweak application. One of the earliest approaches, Case-Based Reasoning (CBR) comes under similarity-finding which adopts solutions that were used to solve similar problem before; e.g., Entree [21,79] recommends similar restaurants the user likes. Only difference in tweak application is, the best available options are filtered based on the tweak. There are number of research studies conducted using pure CBR; such as music [80], urban planning [81], discharge planning in hospitals [82] and tourism [83].

There is not much applied pure CBR research for course RS. The authors in [84] present CBR-based KBRS to recommend suitable major by comparing the historical cases where each student is a case. The case properties are student personal information and features of each course taken. Similarity between the new case and the existing case is measured by using cosine similarity. Moreover, users often need to be educated about courses and new updates in the market before recommending suitable courses which can be partly overcome by building ontologies to incorporate concepts in a particular domain. Ontologies are used to model human knowledge, and support the recommendation process in knowledge intensive systems. It encompasses concepts in a particular domain as objects, attributes, relations between individuals and classes, and represent using ontology representation languages such as Ontology Web Language, Resource Description Framework, DARPA Agent Markup Language and Ontology Inference Layer [85,86]. In the context of e-learning course RS, domain ontology is used to represent user preferences, relationships between users, represent knowledge about the learner and the learning resources [22]. Course Advisory Expert System (CAES) proposed in [87], uses CBR and rule-based reasoning for decision making. It has 3-tier architecture: web application server (Apache Tomcat), Rule-Based Engine (RBE) (Java Expert System Shell) and the relational database (courses). Knowledge base contains the facts and rules that is used by RBE. Main drawbacks are system expects students and advisors to have adequate understanding of the procedure and system administrator must continually update the case base to ensure that course information and rules in the knowledge base are current.

2.4. Data Mining (DM) Approaches

We discussed how DM techniques are applied in CFRS in Section 2.2.2. Previous course RS research has primarily relied on CBRS, CFRS and hybrid techniques. In recent years, ML techniques have been applied [88] or incorporated into course RS [89] to enhance performance and the quality of the final recommendation. In this section, we briefly discuss how DM approaches including ML, DL and ARM are used independently in RS and followed by some examples for course RS with DM [46,90].

2.4.1. Machine Learning (ML) and Deep Learning (DL) Approaches

In [47,88,91,92,93,94,95,96], DM techniques are applied independently in RS and reviewed in detail. A comprehensive review of the use of ML and DL for health informatics can be found in [91]. A RS based on DL approach for choosing publication venue is introduced in [92]. Additionally, a comprehensive review of the use of DL in RS can be found in [93,95]. The authors in [96] conducted systematic review to investigate which ML algorithm support RS and identified that Bayesian, Decision Trees, Neighbor-based, Neural networks, Clustering and Gradient descent-based algorithms are applied mostly in RS.

Furthermore, K-means clustering algorithm is the common algorithm used in research literature in course RS to generate recommendations by clustering large data sets [97]. A ML-based RS for improving student learning experiences is introduced in [94]. The researchers in [46,90] applied K-means clustering to group learners with similar preferences and behavior to recommend the learning objects which are most similar to the active learners based on the selected similarity metrics.

Fuzzy logic has also been introduced into course RS to improve the accuracy of predictions by considering ambiguity in user preferences or human behavior [39,40,41,45,46]. The fuzzy tree-structured model is proposed to express the uncertainty in user preferences [98]. Fuzzy tree matching-based personalized e-learning RS is presented in [43]. The authors in [45] applied, Item response theory and fuzzy logic to represent the learner’s level of understanding of the learned course ware. The algorithm recommends appropriate online course ware to the users based on their learning abilities. Fuzzy c-mean clustering is introduced in [41] to recommend next web page to the target user based on the usage patterns of web pages.

2.4.2. Association Rule Mining (ARM) Approaches

ARM approaches have three categories; traditional ARM, Adaptive-Support ARM (ASARM) and evolutionary ARM. A Priori, Predictive A Priori and Tertius algorithms are a few traditional ARM algorithms applied in research literature [99] to extract association or frequent patterns between data items [38]. In ASARM approach, the system updates the minimum support threshold and generates new rules by traditional ARM algorithm. The authors in [37] discuss how to optimize ARM by using evolutionary algorithms and outlines the limitations of those algorithms. In [75], ASARM is applied on movie data and the results are compared with the other three collaborative recommendation techniques (correlation-based method, neural network paired with information gain and neural network paired with SVD) on similar data on movie. The authors claimed that the performance achieved from the proposed ASARM is better than correlation-based method and neural network paired with information gain. A comparative study of how different types of ARM algorithms applied for course RS in e-learning environment is presented in [99]. The authors claim that the performance of A Priori Association Rule is better than other algorithms.

2.5. Hybrid Techniques

Hybrid RS combine different recommendation techniques to overcome the limitations in single technique. Researchers have focused on incorporating different recommendation techniques in a single course RS to improve quality of final recommendation. These hybrid techniques can be formulated differently and we summarize as follows:

• Combining CFRS and CBRS: Literature indicates that CBRS and CFRS can be combined to achieve better performance [61,100]. Madadipouya and Chelliah [101] proposed four ways to create hybrid approaches by combining CF and content-based filtering. This approach combines CFRS and CBRS recommenders and perform final recommendation at the end [61]. The combination can be performed by various methods including weighted [102,103], switched, mixed and cascade combination [31].
• Developing a course RS combining CBRS or CFRS with Ontologies: This type of hybrid RS technique incorporate ontologies into CFRS or CBRS to improve final course recommendation to users [33].
• Developing a course RS combining DM and statistical methods with CFRS: This type of hybrid RS techniques use DM and statistical methods as the main technique and combines with CFRS. These systems apply features of DM to analyze and discover hidden patterns in the large databases to uncover student behavior or learning patterns to provide personalize course recommendation to users [104,105].

2.6. Other Recommender Systems

In this section, we provide a brief overview of recommender techniques that are not covered in Section 2.1, Section 2.2, Section 2.3, Section 2.4 and Section 2.5. Conversational RS (CRS), statistical-based RS and package recommendation-based RS are discussed here which are applied in course RS in research literature. CRS takes a different approach to produce recommendations [48,49]. According to the definition given in [49], CRS supports its users in achieving recommendation-related goals through a multi-turn dialog. Due to the increased use of new voice-controlled smart devices, and emergence of chatbot technology the popularity of CRS has been increased. The main advantage of CRS is that it clarifies and supports to understand the user needs by asking questions [52,53]. In [50], authors introduce a CRS-based framework for course RS for e-learning environment.

Some statistical-based techniques such as Bayesian Classifier [106] and Naïve Bayes classifiers are prevalent in model-based and user-based collaborative systems, and in Content-Based RS [107,108]. These statistical-based RS attempt to enhance the performance by especially addressing the cold-start situations. Another area of interest of researchers was to apply pure statistical-based techniques to course RS [109,110,111].

Another technique used for RS is package-based recommendation which extends the scope of the general RS. These RS recommends packages of items to a single user which is beneficial in various domains such as: travel packages to tourist, weekly personalized diet (recommending a starter and main course) and recommendation of packages of academic courses to students [112,113,114]. The authors in [112] proposed package recommendation framework by using collaborative filtering predictions and a graph-based model. Several of previous studies applied package recommendation in course RS [5,115]. Proposed algorithm in [5,115] recommends courses with high score, which comply with the given prerequisite conditions.

3. Systematic Review Process

As we discussed in Section 2, there are various recommendation techniques proposed and applied in the field of RS. However, there are several challenges and opportunities in implementing those techniques in the context of course RS as we described in Section 1. Although there are several systematic review papers [116,117,118] published on specific learning platforms such as; MOOCs (Massive Open Online Courses) and e-Learning RS, still lacks systematic literature reviews for course RS in general [119]. Hence, primary focus of this research is to investigate current and emerging approaches applied in course RS in the recent research literature and also to outline the avenues for the future research.

To ensure a systematic review process, we followed guidelines in [120,121,122]. The steps of our review process are illustrated in Figure 2. It involves four major steps: Research Questions Formulation; Setting Protocol of Systematic Review; Review and Analyze Literature; and Data Analysis and Report Findings. This systematic review addresses the following research question:

• What are the cutting-edge techniques applied in recently published course RS?
• What type of data sources have been used to evaluate the techniques in recently published course RS?

3.1. Setting Protocol of Systematic Review

Many research papers have been published in last few years on RS in general and on course RS [61]. Since we were after the most recent and cutting-edge works in the course RS, we selected publications from the last five years; 2016–June 2020. Furthermore, this systematic review aims to discover emerging trends in course RS techniques to deliver insights for the researchers for further investigation. This is a limitation of this review and it may have omitted papers on course RS published before 2016 and after June 2020. In this step of the process, we outline the objectives of the review followed by setting search terms and selection of publications.

The main contributions of this review are:

• First, we investigate techniques applied in RS in general to obtain applicability and usability of these techniques in the context of course recommendation domain.
• Secondly, we analyze and classify course RS in selected publications from 2016 to June 2020. Then we present an overview of state-of-the-art course recommender techniques, by emphasizing which technique applied mostly in this domain.
• Finally, we discuss future avenues which might lead towards next-generation course RS.

3.1.1. Data Sources

This systematic review conducted on the following three repositories and publications from 2016 to June 2020: Google Scholar, IEEE Xplore (https://ieeexlore.ieee.org, accessed on 22 June 2020), ACM digital Library (https://0-dl-acm-org.brum.beds.ac.uk/, accessed on 22 June 2020). We selected Google Scholar since research literature highlighted that Google Scholar is one of the databases that should be there to guarantee adequate and efficient coverage [123,124,125,126].

3.1.2. Setting the Search Terms

Next step is to formulate the search terms to conduct a comprehensive search for research publications. As illustrated in Figure 3 step 1, we searched Google Scholar using the search terms “course” AND “recommender systems” in general without date limit to identify quality search terms for our systematic review. This search retrieved 29,500 results and searches were taken in June 2020. We selected four full text articles [3,11,33,127], which satisfied the selection criteria of our research questions as a guidance for building the search strategy. By analyzing the title and abstract of the first 30 documents retrieved, we noticed that these query terms; “course AND recommender systems” failed to retrieve research publications on course recommender systems. To retrieve more articles which satisfy the selection criteria, we refined the search string and repeated the process and finally changed the search terms as follows.

Boolean operators OR was used to formulate the search strings with words; “course”, “recommender” and word stem “recommend” as wildcard character (*) symbol could not be used in the Google Scholar. Our final selected search terms were “course recommender” OR “course recommend”. An extensive search was performed to identify the suitability of the selected search terms and to collect research papers for our review of current state-of-the-art course recommendation approaches.

The first search was executed on Google Scholar as illustrated in step 2 of Figure 3. It was conducted using the search string “course recommender” OR “course recommend”, with advanced search option ”with at least one of the words“ and “Where my word occur anywhere in the article”. This search returned 1300 records. Since this search retrieved too many records and after reading first 30 documents, we realized some of them are not related to course RS so we refined the search string. We then added another option under advanced search on Google Scholar and selected only the title of the articles (step 3) and search returned 102 documents.

The second database search was on the ACM (step 4) using the same search terms, and it returned 33 documents. Finally, search was conducted on IEEE database (step 5) and resulted 19 documents. All 154 (102 + 33 + 19) papers were included in the preliminary investigation.

3.2. Review and Analyze Literature

Next step is to evaluate the quality/validity of the selected publications. The primary sources were reviewed based on the concepts discussed in the abstract, title of the paper and the following factors listed under inclusion and exclusion criteria. Identification of the sources for the study was conducted by the primary reviewer. Since we focused only on course RS, we excluded articles with other RS.

Inclusion Criteria:

The following criteria was applied to include the publications in the selected collection.

• Publications from 2016 until June 2020
• Conference proceedings and journals
• Research focusing on course RS
• Research papers which shows well-defined methods and evaluations

Exclusion criteria:

The following criteria was applied to exclude the publications in the selected collection.

• Articles in books, textbooks, standards and courses
• Articles written in language other than English
• Technical reports, white papers and thesis—Master and PhD
• Recommendation for digital learning resources or learning activities
• Articles without full paper—only abstract is available
• RS without evaluating their results and duplicates

3.3. Data Analysis and Report Findings

Figure 4 illustrates different phases of the document selection process of this systematic review, which is developed according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement presented in [128]. All 154 papers selected from three repositories; Google Scholar, IEEE and ACM were checked for duplicate records and 54 duplicate records were removed. After primary review based on the abstract, title of the article and inclusion and exclusion criteria, 54 papers were selected for final detail review. In the detail review, researchers examined the full text of the papers. During the detail review, only 43 papers; 6 papers from IEEE Xplore database, 14 papers from ACM and 23 papers from Google Scholar, had been identified as the sources which satisfies our selection criteria for course RS. Then, the researchers extracted techniques used/ application area of the research, type of the dataset or source used in these 43 articles. These papers were explored and classified based on the technique used in recommending courses. Research papers with ontology-based course RS analyzed in detail to identify the main methodologies or factors considered for building ontology and recommending courses such as curriculum design, learner knowledge learning domains and user similarity. Next, we will present the findings from the systematic review.

4. Systematic Review Findings

We conducted a systematic review as discussed in Section 3. The findings from the systematic review are tabulated in

Table A1. A summary of course recommender techniques from systematic review.

Paper Title	Technique Used/Application Area	Dataset / Data Source	Group
Filtering Graduate Courses based on LinkedIn Profiles [129]	BoW (Bag-of-Words) model to represent the studentś LinkedIn profile and course features in vector space	8249 users(professional profiles in LinkedIn) and 163 courses	CBRS
A Personalized “Course Navigator” Based on Student Goal Orientation [57]	Based on Self-reported goal orientation and students past course performance	510 students and eight courses	CBRS
Recommender System for E-Learning through Content and Profile Based Approach [130]	Content-based using pedagogical content and query expansion	50 student and 2–3 queries	CBRS
Domain-Aware Grade Prediction and Top-n Course Recommendation [131]	Nearest neighbor and MF-based CF applied for grade prediction and course recommendation	University of Minnesota—10,000 (13 years) courses, 570 subjects, 60,000 students	CFRS
Guess You Like: Course Recommendation in MOOCs [132]	CF and topic model, based on student interest, demographic profiles and course prerequisites for MOOCs	Online and offline data with 114,303 users and 1242 courses	CFRS
A Course Recommender System based on Graduating Attributes (GA) [134]	Used CFRS—algorithm uses Course X (GA X Students) matrix	Synthetic data—100 students, 100 courses and 28 attributes (competences) 1–5 values	CFRS
A Score Prediction Approach for Optional Course Recommendation via Cross-User-Domain Collaborative Filtering [133]	Cross-user-domain CF and optional course recommendation based on the predicted scores and the curriculum schedule.	Sun Yat-sen University—1166 students, 8 majors, 2311 course-score records (2013–2014)	CFRS
A Cross-Curriculum Video Recommendation Algorithm Based on a Video-Associated Knowledge Map [135]	Video recommendation for online learning using learner’s learning preferences and the knowledge associated in the video	Xi’an Jiaotong University—224 curriculum videos, 53,491 video learning logs and 140,122 course ware learning logs	CFRS
Collaborative Filtering-based Course Recommender using OWA operators [16]	CF-based approach using re-attempt courses using Ordered Weighted Averaging (OWA)	221 students graduated from Dokuz Eylül University (2013–2017)	CFRS
A model for learning objects adaptation in light of mobile and context-aware computing [136]	Ontology-based learning objects adaptation considering student learning style, preferences, profiles and academic background	Conducted a survey to test the App in a course with 20 learners	KBRS
Goal-based Course Recommendation [88]	Variant of Recurrent Neural Network (RNN) called Long Short-Term Memory for grade prediction	University of California—4.8 million Students (2007–2018)	DM (DL)
Designing for Serendipity in a University Course Recommendation System [140]	Based on BOW and RNN which is used to improve course RS presented in [88,137,138,139]	UC Berkeley 164,196 students with 4.8 million enrolments (2008–2017)	DM (RNN and BOW)
Time Slice Imputation for Personalized Goal-based Recommendation in Higher Education [139]	Goal-based course RS applying DL to make inferences about future outcomes based on past actions and MOOC quiz for MOOCs learners	UC Berkeley 4,164,196 anonymized undergraduates, 4.8 m class enrolments, 10,430 courses (2008–2017)	DM (DL-Goal-based)
Directional and Explainable Serendipity Recommendation [146]	Based on Directional and Explainable Serendipity Recommendation method (DESR)	Not given	DM (DESR)
Scholars Walk: A Markov Chain Framework for Course Recommendation [143]	Random-walk-based approach. First-order Markov chain assuming that the selection of course in next semester only depends on current semester courses	University of Minnesota from 23 majors 40 most frequent courses—10 years data	DM (ML-Markov model)
Learning Course Sequencing for Course Recommendation [141]	Ranking good courses higher than the bad ones in the recommended list using SVD and neural networks	University of Minnesota 40 most frequent courses, 500 students (2002–2017)	DM (ML-SVD)
Mining Academic Data to Support Students’ Advisors: A Preliminary Study [142]	Cluster (K-means) students based on marks and enrolments to observe behavior to support course advisors	Data from 2276 students from German university	DM (ML K-means)
Combating the Filter Bubble: Designing for Serendipity in a University Course Recommendation System [138]	Variant Skip-gram-based method compared with RNN. The authors claimed that Variant Skip-gram-based method performed better than RNN	70 undergraduates	DM (ML-RNN)
Intelligent recommendation system for course selection in smart education [144]	Sparse Linear Method (SLM). Empty cells in (student x course) matrix is filled by using sparseness aggregation strategy	No proper testing is given	DM (ML-SLM)
Connectionist Recommendation in the Wild: On the utility and scrutability of neural networks for personalized course guidance [137]	Combination of skip-gram and RNN used to personalized the course recommendation	University of California 108,033 undergraduates 2.2 M course enrolments, 265 majors (2008–2016)	DM (Neural networks)
Intelligent Course Plan Recommendation for Higher Education: A Framework of Decision Tree [145]	Algorithm based on decision trees and applied in computer major selection	Course scores and employment information of the students Xihua University (2014–2017)	DM (ML-Decision Tree)
Application of Predictive Analytics in Intelligent Course Recommendation [47]	Association rules to record previous student skills. Methodology and testing not presented clearly	Synthetic data 1000 students	DM (ARM)
An Association Rule Model of Course Recommendation in MOOCs: Based on edX Platform [147]	The recommendation mechanism is based on ARM and applied on edX platform	User log files database	DM (ARM)
Personalized Employee Training Course Recommendation with Career Development Awareness [97]	Applied Demand-ware Collaborative Bayesian Variational Network (DCBVN) framework to recommend personalized online training courses for employees	High technology company—China. 714,091 records, 30,662 employees, 8693 courses (2016–2019)	Hybrid (CFRS and ML)
Learning to be Relevant: Evolution of a Course Recommendation [104]	RS for LinkedIn Learning, based on combination of unsupervised and supervised models for online learning	645 million+ members and 200+ courses	Hybrid (CFRS and statistical)
Design of adaptive learning system based on big data [148]	Design of hybrid RS—Domain module is built using ARM followed by ML and not applied to specific scenarios	Not given	Hybrid (CFRS, ARM, ML)
PCRS: Personalized course recommender system based on hybrid approach [33]	N-Gram query expansion and select more related courses from constructed Ontology for Computer science domain	300 Computer Science courses at BSAR crescent university Chennai	Hybrid (CFRS and ontology)
A hybrid course recommendation system by integrating collaborative filtering and artificial immune systems [34]	Two-stage process—initial CF based on student performance and followed by clustering mechanism based on Artificial Immune System	Yuan Ze University, 3455 courses, 22,262 students 2005–2009	Hybrid (CFRS and Artificial Immune system)
Will this Course Increase or Decrease Your GPA? Towards Grade-aware Course Recommendation [105]	Two approaches: Represent learning approaches using SVD, Course2vec and grade prediction using Cumulative Knowledge-based Regression Models	University of Minnesota 23 majors, 50 most frequent courses with at least 500 students from 2002–2017	Hybrid (ML, statistical)
Data-Driven Approach Towards a Personalized Curriculum [151]	Personalized course recommendation graph: Course dependency graph (using Mann-Whitney U-test) and Grade prediction using Collaborative	Saarland University 1700 students’ basic information, grades, 72 courses, lecturer information, 16,090 entries	Hybrid (CFRS and statistical)
Using Ontology for Personalized Course Recommendation Applications [149]	Data integration is performed by building three (3) ontologies for student, course and job. Recommendation is performed by using CBRS and CFRS techniques	Extract information from Universities and Colleges Admissions Service (UCAS) UK	Hybrid (CBR and CF)
Developing a Course Recommender by Combining Clustering and Fuzzy Association Rules [46]	Cluster students on demographic data and previous course scores and association rules are mined from the clusters to do the recommendations	798 undergraduate students University of Tehran between 2004 and 2015	Hybrid (ML and ARM)
Research of Personalized Course Recommended Algorithm based on the Hybrid Recommendation [29]	Uses weighted output from improved New A Priori algorithm-based RS and user-based CF RS to implement hybrid system	Central China Normal University 13,140 students 3 computer courses 2013–2015	Hybrid (CFRS and ARM)
ULEARN: Personalized Course Learning Objects Based on Hybrid Recommendation Approach [32]	Recommend learning resources using CF and content-based, initialized the student profile based on a questionnaire before recommending learning objects to the user	No proper testing is conducted	Hybrid (CBRS and CFRS)
On Recommendation of Learning Objects Using Felder-Silverman Learning Style Model [150]	CBRS and CFRS-based filtering to enhance accuracy of predictions and recommend the most suitable course learning objects (LOs) considering student learning styles, LO profiles and students’ ratings of LOs	Arab Academy for Science and Technology and Maritime Transport (AAST) for the autumn and spring semesters 2016–2018	Hybrid (CFRS and CBRS)
An automated recommender system for course selection [3]	CFRS with K-means to cluster students, KNN to select K-most similar groups, ARM to discover course selection patterns	2000 graduate students with 54 courses including four elective courses	Hybrid (CFRS and DM)
Ontology-Based Personalized Course Recommendation Framework [27]	Hybrid filtering based on both the CBRS and CFRS methods and incorporated three ontologies; course, student and jobs to personalize the recommendation	Scrapes courses from Universities and Colleges Admissions Service, jobs (Indeed.com), 123 students University of Portsmouth	Hybrid (CBRS, CFRS and ontologies)
Knowledge-Based Recommender System for Academia Using Machine Learning: A Case Study on Higher Education Landscape of Pakistan [24]	Proposes two popular topic models: LDA and Author Topic Model (ATM) to discover underlying themes of text to support the recommendation process	Conference proceedings 1999–2000, 420 documents, 192 documents for model validation	Hybrid (Statistical and ML)
Bio-Inspiring Learning Style Chatbot Inventory Using Brain Computing Interface to Increase the Efficiency of E-Learning [50]	Classify E-Learners; Introvert or Extrovert using chatbot based on their personality types (likes and dislikes) and their learning styles by analyzing the Brain wave (EEG) signals and recommend course contents	Students and staffs from SRM Institute of Science and Technology Kattankulathur, India, 118 learners between 18 and 79 years	Other (CRS)
Bounded Off-Policy Evaluation with Missing Data for Course Recommendation and Curriculum Design [109]	Regression estimator penalized variance optimization methods based on the logged data of students	Course grades of 920 anonymized students graduated 2013–2015	Other (Statistical)
Placement Predictor and Course Recommender System [111]	C4.5 algorithm and Naïve Bayes Classifier applied for placement prediction and found C4.5 has better accuracy	772 student records from 5 different schools of Tamil Nadu 2006	Other (Statistical)
Context-Aware Online Learning for Course Recommendation of MOOC Big Data [110]	RHT and DSRHT algorithms for course RS in MOOC big data and analyze the time and space complexity	11352 courses from edX and intermediary website of MOOC	Other (Statistical)
Content-based Course Recommender System for Liberal Arts Education [56]	Estimate course’s topic distribution—LDA generative probabilistic model, Gibbs sampling and Kullback–Leibler, an asymmetric measure to estimate the degree to which a course’s topic distribution corresponds to academic interests of the student	490 courses Maastricht University 2018–2019, 2526 students, 79,245 course enrolments. However, no results for course recommendation	Other (Statistical)

(in Appendix A) in terms of techniques applied in the selected course RS, data source used for evaluation of the proposed system and classification based on the recommendation technique. Next, we present our findings of the systematic review in the areas of cutting-edge approaches applied in selected course RS.

4.1. Findings on Recommendation Techniques

In Section 2, we presented a detail review of the state of the art of methodologies implemented in different RS. The summary results of this systematic review is presented in Table A1 which reveals that number of recommendation techniques have been proposed, applied and evaluated in the selected course RS research studies carried out from year 2016–June 2020. Our review identifies six major categories of course RS in research literature; CBRS, CFRS, KBRS, DM, Hybrid and Other (CRS and statistical). Next, we discuss the findings of systematic review of each course RS recommendation technique listed above, and we will start with CBRS.

4.1.1. Content-Based Recommender System (CBRS)

We introduced and discussed the concept of CBRS techniques in general in Section 2.1. Next, we present the outcomes of systematic review of CBRS technique applied in the context of course RS. The authors in [129] provide a solution to the new user problem in CBRS. To overcome the new user problem, proposed system integrates the studentś LinkedIn profile features to represent students in the vector space. Moreover, the research conducted in [130] applied query expansion techniques to generate revised query from the keywords in the studentś query. Then the revised query will be matched with the existing course contents available in the repository. Then, they ranked the expanded queries based on the contents in the repository to find the best match. Self-reported goal orientation-based course recommendation is presented in [57]. When recommending courses to the students, system categorizes students into Mastery Goal Oriented (MGO) and Performance Goal Oriented (PGO) based on the survey data collected from the students.

Table 1. A summary of CBRS-based course RS.

Paper [Ref]	Drawbacks/Comments
Basavaraj et al. [57]	Categorize students into Mastery Goal Oriented (MGO) and Performance Goal Oriented (PGO) based on survey data. Suitable for recommending courses for existing students.
Lessa et al. [129]	Provides a solution to new user problem using students LinkedIn profile features to represent students in vector space. Main drawback is some students do not have LinkedIn profiles.
Venugopalan et al. [130]	Query expansion is used to increase number of features before matching them with existing course contents, ranking the number of learning objects (closest to the userś query) returned by the revised query to control the output/ scalability.

illustrates the summary of CBRS-based course RS systems identified in the systematic review in terms of drawbacks or comments.

4.1.2. Collaborative Filtering-Based Recommendation System (CFRS)

We introduced and discussed CFRS techniques in general in Section 2.2. Next, we present the outcomes from the systematic review for CFRS applied in the selected course RS. CFRS-based course RS used different techniques to recommend courses to students such as; nearest neighbor and MF user-based CF to predict top-n courses for students [131], topic models to partition users based on demographic profiles, student interests and recommend courses based on the cluster [132] and recommending courses by calculating course score distribution of the most similar senior students [133]. Furthermore, one study results showed that better performance can be obtained from K-nearest neighbor by selecting neighborhood between 10 to 15 students [134].

The main challenge in CFRS is to overcome the cold-start problem. Some research studies overcome this by taking weighted average of most recent grades of the courses taken by the student [16] and using K-Modes algorithm to partition users based on demographic profiles and assign cluster average rating to the users in the same cluster [132].

Table 2. A summary of CFRS-based course RS.

Paper [Ref]	Drawbacks/Comments
Elbadrawy et al. [131]	Nearest neighbor and MF, suitable for existing users. Paper analyzes large grade data by incorporating features-based groups into matrix factorization methods which leads to better grade prediction.
Jing et al. [132]	Address cold-start problem—K-Modes to partition users and assign average rating to the users in the cluster, model user interest on what they accessed instead of courses enrolled.
Bozyigit et al. [16]	Cold-start problem is overcome by taking Weighted Averaging of most recent grades of the courses taken by the students
Bakhshinategh et al. [134]	The results showed that better performance can be obtained by selecting the number of users in the neighborhood for K-nearest neighbor between 10 to 15 students
Huang et al. [133]	Novel cross-user-domain CF by using course-score distribution of the most similar senior students, scalability issues is addressed by selecting the top t optional courses with the highest predicted scores. Extensive testing has been conducted.
Zhu et al. [135]	The experiment is limited to three subjects for online learning and contains 224 curriculum videos, 53,491 video learning logs and 140,122 courseware learning logs

illustrates the summary of CFRS-based course RS systems identified in the systematic review in terms of drawbacks or comments.

4.1.3. Knowledge-Based Recommender System (KBRS)

We presented KBRS techniques in general in Section 2.3 and here we present the outcomes of systematic review of KBRS in the context in course RS. In the selected list, we found only one course RS on pure KBRS. This research proposed ontology-based learning objects adaptation by considering students learning style, preference, profile and background [136].

4.1.4. Data Mining (DM) Approaches

We discussed DM techniques including ML, DL and ARM in general in Section 2.4. In this section, we present the findings from our systematic review for DM techniques. Recent literature demonstrates propensity for applying DM techniques in course RS as it is beneficial when building course RS with large data sets.

DL approaches have been proposed in many course RS in our selected publications. Pardos et al. used RNN and skip-gram model in [137] and skip-gram model variant model in [138] for course RS. A novel approach to goal-based course RS has been proposed in [88]. This is an extension to the goal-based course RS, they adopted variant of RNNs called Long Short-Term Memory (LSTM) framework to make personalized goal-based recommendations. Three prediction validations have also been conducted for; grade prediction, prerequisite prediction, and course selection prediction. Furthermore, the same authors discussed goal-based course recommendation model for MOOCs learners by employing DL to make inferences about future outcomes based on past actions and MOOC quiz [139]. In [140], Pardos et al. proposed BOW and RNN as an improvement for course RS presented in [88,137,138,139]. In our review, a few publications proposed pure ML techniques for course recommendation [141,142]. Some selected studies used Random-walk-based approach [143], SLM [144], Decision tree expression [145], Gaussian Mixture Model and capsule network [146].

Pure ARM has also been applied in course RS research literature [47,147]. In [47], association rules have been used to record student academic records but the methodology and testing process is not clear. In [147], authors introduced ARM-based course recommendation mechanism on edX platform.

Table 3. A summary of Data Mining (DM)-based course RS.

Paper [Ref]	Drawbacks/Comments
Jiang et al. [88]	Variant of RNN framework is adopted. Conducted three (3) prediction validations; grade prediction, prerequisite prediction, course selection prediction.
Jiang et al. [139]	This is an improvement for [88], use DL to make inferences about future outcomes based on past actions and MOOC quiz.
Pardos et al. [137]	Combination of skip-gram and RNN used to personalized the course recommendation based on the course history of the student and their majors.
Pardos et al. [138]	This is an improvement for [137]. Variant Skip-gram-based method performed better than RNN.
Pardos et al. [140]	BOW and RNN used to improve course RS presented in [88,137,138,139].
Upendran et al. [47]	The methodology and testing process is not clear, ARM on student previous academic records.
Tan et al. [147]	ARM on edX platform, reported some defects on the mechanism with associated risks and limitations
Li et al. [146]	Serendipity Recommendation is proposed as a solution to overspecialization.
Chen et al. [145]	Result reveals that IT companies pay attention to 6 groups: Group 1 (Sales and Product Manager), Group 2 (Software Engineer and Technical Management), Group 3 (Software Engineer and Research), Groups 4 and 5 (Algorithm Engineer and Research), Group 6 (Software Engineer), and Group 7 (Technical Management and Other Administrative Post).
Morsy et al. [141]	This system is suitable for students who have taken at least one term with at least one course.
Egbers et al. [142]	Preliminary work is presented here. clustering results reveals that students with good marks study quicker while students with lower marks need longer time to complete and could recommend study part-time to minimize the dropping out rate
Lin et al. [144]	Suitable for recommending courses to existing students as initial course selection of a specific student is used in sparseness aggregation strategy. Evaluations were conducted on small set of data.
Polyzou et al. [143]	Proposed model was compared against other models and result shows that first-order Markov chain perform better than the higher-order models.

illustrates the summary of DM-based course RS identified in the systematic review process in terms of drawbacks or comments.

4.1.5. Hybrid Approach

We introduced hybrid techniques used in RS in general in Section 2.5. In this section, we present the findings from our systematic review for hybrid techniques used in the context of course RS. In the academic context, new student users of the course RS do not have past history or user profile and difficulty to identify similar users with the same academic background in the existing system. Hence, in recent research literature CBRS and CFRS incorporated hybrid techniques to build the user profile and improve the quality of the final recommendation [32,33]. These hybrid recommendation approaches will significantly contribute to improving the performance of the pure content-based or CF course RS approaches by providing solutions to; new user problem, new item problem, sparsity and scalability.

Furthermore, hybrid models were developed by combining unsupervised and supervised DM models with CFRS for recommending courses for online LinkedIn learning have been proposed in [104]. Researchers are also interested in using single unified hybrid techniques, by combining CFRS with ARM in course RS. For example, improved new A Priori association rules and collaborative filtering is applied in [27,29]. Researchers have also considered creating hybrid models, combining DM, statistical, KBS with CFRS, which is more than one RS in course RS [3,104].

Al-Badarenah et al. [3] presents CFRS approach for course selection, which applies K-means clustering to table (student x course_grade) to group students based on their courses and grades. Manhattan distance is used in K-means to cluster students by mapping grades A to F to integers 1 to 6 respectively and $-1$ to indicate that “student did not take the course”. Next, KNN selected K-most similar groups produced by the clustering. Finally, ARM is applied to recommend courses to the selected student groups by identifying course selection patterns. Furthermore, a ratio called matchrule is used to assess the quality of rules in the recommendation. If matchrule value is greater than a threshold, then that rule is assumed to have the highest support, confidence, and the highest match and is used for recommendation. This allows user to select the top-N courses. The main drawbacks of this are, it does not recommend courses to the new students since they do not have courses taken yet (cold-start problem) and rule formation is a complex process as it depends on the student’s previous grades.

DCBVN model is applied in [97] for recommending personalized online courses to employee training and development . Furthermore, some research studies implemented hybrid with DM and statistical techniques [24,105,148]. For example, Morsy and Karypis [105] applied SVD, Course2vec and cumulative Knowledge-based Regression Model to predict sequencing of courses. This grade-aware prediction used the student’s previously taken courses and next-term’s good courses.

Another area of interest of researchers was to overcome shortcoming in the current models due to lack of comprehensive knowledge about the existing courses. Researchers have developed ontologies by using multiple heterogeneous data sources and incorporated ontologies with CFRS or DM techniques to suggest suitable courses to the prospective students [27,149]. For example, in [33] researchers used ontology only for Query expansion. N-Gram query expansion is applied here to convert user query into a list of categories and expand the query to select more related courses from the constructed ontology on computer science domain. The main drawback is system returns large number of recommended courses. Scalability was addressed by filtering and categorizing courses based on the query term frequency. However, course RS also need knowledge about learners and learning materials to generate recommendations to learners. In

Table 4. A summary of hybrid course RS.

Paper [Ref]	Drawbacks/Comments
Ibrahim et al. [149]	Hybrid with CBRS, CFRS and incorporating 3 ontologies, student, course and job to produce personalized course recommendation. However, initial results presented with limited discussion.
Ibrahim et al. [27]	Hybrid with CBRS and CFRS with student and course ontologies to overcome problems in lack of knowledge about the existing courses.
Chang et al. [34]	Hybrid combining CFRS with Artificial Immune System and DM to identify similar student groups. The paper claimed that the proposed RS will give highly accurate outcomes for students with higher mean grades.
Nafea et al. [150]	Hybrid RS combining CBRS and CFRS to enhance the accuracy of the predictions considering student learning styles, LO profiles and students’ ratings of LOs to overcome cold-start problem.
Backenkohler et al. [151]	Hybrid RS combining CFRS with statistical and DM. Mann-Whitney U-test to create course dependency graph in CFRS to enhance performance in prediction
Yang et al. [29]	Hybrid RS with CFRS incorporates improved ARM. Algorithm generates recommendations by calculating target student’s similarity and producing a nearest neighbor set.
Wang et al. [97]	Hybrid RS with CFRS combining with statistical(DCBVN model) techniques to extract skill profiles and to recommend courses. Experiment is conducted on real-world data to exhibit the effectiveness and the interpretation power of the proposed course RS
Rao et al. [104]	Hybrid RS combining DM (unsupervised and supervised models) with statistical, experiment is conducted using custom built A/B testing framework, used various LinkedIn channels to check the usefulness.
Al-Badarenah et al. [3]	Hybrid combining CFRS with K-means, KNN, ARM Effectively reduces scalability by clustering and recommending a popular item using ARM. Main disadvantage is the cold-start problem as new users do not have courses taken and rule formation is complex as it depends on the student’s previous grades.
Morsy et al. [105]	Hybrid RS combining DM (SVD and Course2vec) with statistical methods. Predicting sequence of courses based on previously taken courses and next terms good courses.
Samin et al. [24]	Hybrid RS combining DM and statistical (LDA and Author Topic Model (ATM)) methods in recommending courses for teaching staff based on the staff research profile
Asadi et al. [46]	Effectively reduces scalability by clustering on demographic data and previous course scores and ARM to mine the cluster; however new courses with low number of students will create the limitations
Nafea et al. [32]	Provides a solution to the new user problem by initializing the student profile based on a questionnaire.
Gulzar et al. [33]	WordNet is proposed for Query expansion and then select courses based on the expanded query from the constructed Ontology for Computer science domain. Scalability was addressed by filtering and categorizing courses based on the term frequency cold-start problem is overcome by using an ontology.
Liu et al. [148]	hybrid with ARM and ML. Only the design is proposed not applied to specific scenarios for evaluation.

, we summarize hybrids-based course RS identified in the systematic review process in terms of drawback or comments.

4.1.6. Conversational Recommender System (CRS) and Statistical-Based Approaches

We introduced CRS and statistical techniques as other RS categories in general in Section 2.6. Next, we present the outcomes of systematic review of these techniques applied in the context of course RS. The researchers have used several statistical techniques in course RS. LDA generative probabilistic model and Gibbs sampling algorithm have been applied to estimate the course topic distribution and Kullback–Leibler and an asymmetric measure are used to estimate the degree to which a course topic distribution corresponds to the academic interests of the student [56]. Then, Naive Bayes Classifier and C4.5 algorithms are used to predict the placement results [111] and regression estimator penalized variance optimization methods are used for course schedule recommendation [109]. In [109], authors applied regression estimator penalized variance optimization methods in course RS. Furthermore, researchers in [110] have considered Reformational Hierarchical Trees (RHT) and Distributed Storage Reformational Hierarchical Tree (DSRHT) algorithms for recommending courses for online learning. These algorithms are proposed for recommending courses with huge and changeable data sets.

Only one course RS was in our database which applied CRS. This research proposed a framework to classify E-Learners into two classes; Introvert or Extroverts, before recommending course contents. It is based on student personality types and their learning styles. This classification is done by using a chatbot to obtain their likes and dislikes. Then, examine the learning styles by analyzing the brainwave (EEG) signals [50].

Table 5. A summary of CRS and statistical-based course RS.

Paper [Ref]	Drawbacks/Comments
Rajkumar et al. [50]	Implementation has been explained extensively ,chatbot based on learner likes and dislikes and examine the learning styles by analyzing the Brainwave (EEG). Results shows that it takes less time to classify learners.
Morsomme et al. [56]	For each course, system proposes five suitable preparatory courses. Lasso regularized multiple linear regression model is used for grade prediction.
Hou et al. [110]	Presented algorithms suitable for huge and changeable data sets. Experiments showed that RHT algorithm outperformed traditional Adaptive Clustering Recommendation Algorithm (ACR) and High Confidence Tree algorithm (HCT)
Wadekar et al. [111]	Results showed that C4.5 has better accuracy than Naïve Bayes Classifier
Hoiles et al. [109]	Provided a methodology to include missing data into the regression estimators when domain knowledge is available.

illustrates the summary of CRS and statistical-based course RS systems identified in the systematic review in terms of techniques applied and drawbacks or comments.

4.2. Findings on Data Sources

From the analysis of the third column in Table A1, the dataset used can be categorized into several areas: 76.7% of publications used student enrolment data from university/college/online course, 4.7% of publications used synthetic data, 2.3% of publications used user log file database, LinkedIn profiles and conference proceedings. However, 9.3% of publications did not use data sources to evaluate and publication is limited to the design or no information is provided regarding the dataset. Therefore, most of the publications used student enrolment data to evaluate the proposed course RS; however, some data sources such as LinkedIn profiles, WordNet and survey data, were used to overcome the cold-start problem in selected course RS.

4.3. Summary of the Systematic Review

Anatomy of different course recommendation techniques identified in our systematic review have been summarized in the Figure 5 and the selected publications for the systematic review have been classified within these categories in Table A1. The percentage of the total number of papers categorized under each technique is shown in

Table 6. Course recommendation research papers and techniques.

Technique	Reference(s)	Percentage (%)
CBRS	[57,129,130]	7.0
CFRS	[16,131,132,133,134,135]	14.0
KBRS	[136]	2.3
DM	[47,88,137,138,139,140,141,142,143,144,145,146,147]	30.2
Hybrid	[3,24,27,29,32,33,34,46,97,104,105,148,149,150,151]	34.9
Other (CRS and statistical)	[50,56,109,110,111]	11.6

with references. According to our findings, 34.9% of the selected research papers applied more than one technique (hybrid) and 30.2% applied DM techniques to recommend courses to students. Figure 6 and Figure 7 visualize the evolution of these approaches throughout the years. Figure 6 presents summary of publications in each technique, identified during the systematic review. We can observe that growing popularity in hybrid and DM techniques in recent publications.

Hybrid course RS combine collaborative models [27,149,150] with other techniques such as; nearest neighbor, topic models, CBRS, clustering, Mann-Whitney U-test and ARM [29] for knowledge association (Table 4). Among these approaches Clustering, Neural Networks or Bayesian Classifiers are the common approaches applied to cluster items and compare them with the characteristics in the user profile. Many hybrid course RS applied statistical or ML techniques to make final recommendations [3,24,34,46,97,104,105,151]. Furthermore, among DM techniques applied; K-means, SVD, RNN, decision trees, ARM techniques are the most common techniques implemented in selected papers. Most of the time these techniques are implemented with the statistical methods. It can be seen that ontologies are also incorporated into CFRS-based hybrid course RS to improve final recommendation.

Many studies on CBRS incorporated several techniques to improve the quality of the final recommendation [57,129,130]. In [57] used LinkedIn profile features to represent students in vector space and as a solution to new user problem. N-gram query expansion techniques is used [130] to deduce the studentś academic interests from the key words they entered into the system before matching it with the existing course contents available in the system. Then they ranked the expanded queries based on the contents in the repository to find the best match. To overcome the cold-start issue some hybrid systems [32] initialized the student profile; based on a questionnaire before recommending learning objects to the user; used the studentś LinkedIn profile features to represent students in the vector space [129]; initialized with weighted average of most recent grades of the courses taken by the students; or used K-Modes algorithm to partition users based on demographic profiles and assign cluster average rating to the users in the same cluster.

From Table 6 and Figure 6, it is evident that there is a significant increment in using DM techniques in course RS in papers published from 2016 to June 2020. In addition to that knowledge-based and content-based techniques have been applied in very few course RS studies. This may be due to some challenges in implementing KBRS such as; creating and implementing ontology is time-consuming; need expert knowledge; and knowledge representation should be able to embrace the changing requirements. In general, CBRS is applied in these course RS to generate the recommendation by determining the similarity between the given item with the user profile. However, few CBRS-based course RS were present in the selected publications.

Another prevalent feature is Fuzzy and CRS techniques are less popular in course RS domain among these research papers. Research on CRS for course recommendation domain is still new and it can be the new trend for future implementations. Combining ML/DL and statistical techniques with fuzzy-based and conversational-based techniques has potential in improving the performance of the future course recommendation systems. Fuzzy-based and conversational techniques can be applied to overcome the drawbacks associated with vagueness and imprecision in identifying user requirements. According to our review of course RS, we can categorize selected course RS mainly into two areas; recommending courses (offline) to university students, recommending online courses to new students. First, it can be specific to a university course selection when the number of course offerings increases, students required additional support to choose suitable courses according to their career objectives. The other category is recommending online courses for new students considering their career objectives and learning skills. In this category of course RS used web crawlers/ online user profiles (e.g., LinkedIn profiles) to gather necessary job specifications and online course requirements.

Most of the data sources used in course RS are student enrolment data from university/college/online courses. However, some course RS used data sources such as LinkedIn profiles, WordNet and survey data.

5. Conclusions and Future Directions

Generally, course RS are algorithms suggesting suitable courses to the users. These algorithms consider various factors related to the students and the courses. This study conducted a systematic review of publications from 2016 to June 2020 on course RS to identify the state of the art in methodologies of course RS. Following a search of three academic repositories; ACM, IEEE Xplore and Scholar, 154 peer-reviewed publications were selected, and primary review was conducted. After preliminary filtering based on the abstract, title of the article and inclusion and exclusion criteria 43 papers were selected for final review. This survey has uncovered that there is a growing popularity in hybrid and DM techniques in course RS published from 2016–June 2020. Furthermore, hybrid techniques are applied mainly with CFRS combining with other systems such as; statistical and DM. Results further shows that collaborative models are applied in course recommender domains with other techniques such as; nearest neighbor, topic models, CBRS, clustering, Mann-Whitney U-test and ARM for knowledge association. Another predominant feature is Fuzzy and CRS techniques are applied in a very few research studies on course RS. Future research will require investigation of the possibilities of incorporating fuzzy and conversational techniques into existing course RS to address the ambiguity in user requirements and to integrate dialog system into course RS to build personalized CRS to enhance the quality of the recommendation. Current studies estimate user preferences based on their past behavior or based on the user profiles which is a one directional form of user interaction. By incorporating intelligent dialog agents to course RS, user interactions can be improved by allowing user to question about the recommendations and delivering suitable feedback and it also improves the ability to understand user preferences accurately and deliver personalized recommendations.