Knowledge Tracing Model and Student Profile Based on Clustering-Neural-Network

Abstract

Knowledge tracing models based on deep neural networks are currently widely studied to enhance personalized learning. However, to ensure the practical deployment of DNN-based KT models, prediction accuracy, training efficiency, and interpretability should be greatly improved. In this paper, we observe that the prediction accuracy of KT models can be improved by clustering the features of both students and questions. Based on this observation, a distributed KT scheme is proposed: (1) it classifies both students and questions based on clustering technology to reduce the interaction between different feature data to improve the prediction accuracy; (2) models for different classifications are trained in parallel in this distributed deployment architecture to improve the training efficiency; (3) the combination of a students’ knowledge state matrix and an RPa-LLM model is designed to display the knowledge status of students in the learning process, which can be used to build students’ portraits, thus improving the interpretability of the model. Real educational data are collected to conduct experiments. The results show that the proposed scheme improves both prediction accuracy and training efficiency by 4.08% and 67.28%, respectively, compared to the baseline methods. Furthermore, the proposed method maintains the interpretability of KT models, making it suitable for practical deployment.

1. Introduction

In the age of intelligence, online learning techniques have rapidly developed, facilitated by massive educational data. This data-driven approach enables the provision of intelligent education services. By utilizing vast educational datasets, it is possible for teachers to extract and analyze students’ knowledge status, which provides them with valuable insights into students’ needs. In this case, teachers can offer personalized instruction that is tailored to each student’s individual requirements [1]. Knowledge Tracing (KT) is a technology that captures the state of students’ knowledge by learning their behavior when answering questions. It is widely studied by worldwide researchers to enhance personalized learning.

With massive learning data, KT models tend to become more complex. Recently, the state-of-the-arts (SOATs) are based on deep learning techniques and show significant improvements compared with Bayes Knowledge Tracing (BKT). Previously, KT models based on Deep Neural Networks (DNN) have been widely studied and improved. Some works improve the prediction accuracy of Deep Knowledge Tracing (DKT) through data engineering [2,3,4,5,6]. For example, Sonkar proposes question-level Deep Knowledge Tracing (qDKT), which provides richer information and more accurate prediction by estimating students’ mastery of each question [4]. Sun introduces a Classification and Regression Tree (CART) to filter all the features in the dataset to improve the characteristics of the input layer [6]. Furthermore, KT models based on the attention mechanism have been widely focused upon in recent years, and they can obviously improve the prediction performance of DKT [7,8,9,10,11]. In these works, some approaches improve the model details combined with educational and cognitive theory. For example, Cheng considers adding the “mistake” and “guess” factors to the KT model to improve the prediction accuracy of the model [12]. Liu believes that the relationship of knowledge points has an impact on sequence prediction, so they used an attention mechanism to describe the relationship between knowledge points [13]. As for DNN-based KT models, interpretability is also an important aspect because students and teachers need clear information to guide them to improve deficiencies. Thus, some works focus on improving the interpretability of KT models [14,15]. For example, Yeung uses a KT model to predict problem difficulty and student ability, improving the interpretability of the model. Using Item Response Theory (IRT) to predict student reactions based on problem difficulty and student abilities [14], Su expands each student’s knowledge state vector into a knowledge state matrix that updates over time, capturing the student’s knowledge state and improving the interpretability of the model [15].

Current works rarely consider improving the performance of KT by classifying the students and questions. We consider that different students have different reactions to different kinds of questions. Thus, if we use all students’ data and questions to train one KT model, the accuracy of models would be impacted by the noises between different groups of students and questions. In this work, we conduct an experiment (in Section 1.1) to find that clustering data with different characteristics can improve the prediction accuracy of the overall model. In this case, the most related work is the work of Minn. They believe that different methods for predicting students’ learning abilities are different, so they cluster students’ abilities based on the K-means clustering algorithm, using students’ abilities as additional inputs to improve their prediction ability [16]. However, they only considered the characteristics of the students, not the characteristics of the questions. In this work, we classify both students and questions to improve the accuracy of models. Then, based on our proposed classification structure, a distributed deployment training architecture is designed to improve the timeliness of model training. Additionally, a Latent Linear Model (RPa-LLM) is considered to improve the interoperability of KT. As for experiments, real data are collected, and the results show that the proposed scheme can improve prediction accuracy, training efficiency, and interpretability.

1.1. Motivation

To verify the effect of data characteristics on KT, we designed the following experiment. The dataset is designed as follows: (1) Type A students have a probability of answering Type A questions correctly, but Type A students must answer Type B questions incorrectly; (2) Type B students have a probability of answering Type B questions correctly, but Type B students must not answer Type A questions correctly. The results of the training are shown in

Table 1. Simulation data training results.

	AUC(%)	F1 (%)	Recall (%)	Precision (%)
Non-Clustering	51.42	0	0	0
Question Clustering	84.63	86.06	71.92	84.62
Student & Question Clustering	96.82	85.18	79.03	92.37

The experimental results in Table 1 are trained based on the KT model of the Long Short Term Memory (LSTM) neural network, which is the same model as the DKT model in the experimental part. It can be concluded that clustering can improve the prediction accuracy of the overall model by pre-extracting data features.

.

This paper is organized as follows: Section 1 introduces the research related to KT; Section 2 illustrates the KT framework; Section 3 presents the experimental results; the discussion of the results is in Section 4; and Section 5 concludes this paper.

2. Frame of C-B-KT

This section introduces the preprocessing and training process of the Clustering-Bert-Knowledge Tracing (C-B-KT) model, and the flow chart is shown in Figure 1. In Steps S100–S200, the general service encodes the data; Step S300 clusters the encoded data; Step S400 sends the clustered data to the corresponding proxy server for neural network training; and Step S500 extracts the knowledge state matrix of the corresponding model and constructs and summarizes student portraits.

2.1. Data Encoding and Data Clustering

Question encoding is used for question vectorization to facilitate clustering. One-hot coding has been widely used in KT research [17]. In one-hot coding, each question is usually represented as a manually marked knowledge concept, but the rich information contained in the question text has not been fully mined. In order to mine text information, this paper uses the Bert model for coding. Bert uses Transformer, which is more efficient than Recurrent Neural Networks (RNNs) and can capture longer-distance dependencies [18]. The question vector after Bert encoding contains more hidden information.

Student encoding is used for student vectorization to facilitate clustering. This paper assumes that a student’s ability does not change in a short period of time. The student vector is obtained by observing the student’s response to all questions. Suppose the response sequence $s_i=(e_{i_1},r_{i_1}),(e_{i_2},r_{i_2}),\cdots ,(e_{i_t},r_{i_t})$. Here, $(e_{i_t},r_{i_t})$ means that if student i answers the question $e_{i_t}$ correctly, then $r_{i_t}$ = 1; for a wrong answer, $r_{i_t}$ = 1; for no answer, $r_{i_t}$ = 2. According to the student answer sequence, the student vector $q_i=(r_{i_1},r_{i_2},\cdots ,r_{i_t})$, $q_i$ represents the vector of student i.

Data clustering. In this paper, the K-means clustering algorithm is used to cluster the data. The algorithm has the following features: (1) the unsupervised clustering algorithm can automatically cluster data features; (2) clustering is controllable, and the number of clusters can be controlled by the K-value; (3) it benefits from high performance, meaning that it is less time consuming than hierarchical clustering algorithms; (4) and the clustering effect is the best possible. This paper studies various clustering algorithms based on experimental data (see

Table A1. Cluster Index Results.

	K-Means	Agglomerative	DBSCAN
Time (s)	44.94	3120.61	81.73
Calinski-Harabasz	2113	723.31	197.32
Adjusted Rand index	0.83	1	1

Table uses three indicators, Time, Calinski-Harabasz and Adjusted Rand Index, to represent the performance, data adaptability and stability of the clustering algorithm, respectively, to determine the best clustering algorithm.

).The clustering results are shown in Figure 2.

2.2. Model Training

For all combined data, we use a Bert−KT model (B−KT) with the following structure (Figure 3) to conduct training separately.

Question Embedding. Text information $e_t$ automatically learns the semantic representation code $x_t$ of each question, that is, question embedding. Each word $w_{t_j}$ in the question $e_{t_j}$ is converted into a $d_0$-dimensional word vector $x_{t_j}$, using the weighted average to get the question vector. The equation is

$$x_t=\frac{1}{m}\sum _{j=1}^m{x}_{t_j}$$

(1)

where $x_{t_j}\in R^{d_0}$ represents the word, $w_{t_j}$ is the word vector obtained after Bert processing, m represents the number of words in question $e_t$, and $x_t$ represents the question vector.

Student embedding is the modeling of the different training processes of different students and traces students’ hidden knowledge state in different training processes, that is, student embedding. The KT framework relies on two basic assumptions: (1) the student’s knowledge state is related to the question information and answer results; (2) students usually learn and forget their long-term continuous questions. Based on the above assumptions, this paper uses the LSTM model to embed students. This is the equation for LSTM:

$$\begin{array}{cc}\hfill f_t& =\sigma (W_f·[h_{t-1},x_t]+b_f)\hfill \\ \hfill i_t& =\sigma (W_i·[h_{t-1},x_t]+b_i)\hfill \\ \hfill \widetilde{C_t}& =tanh(W_C·[h_{t-1},x_t]+b_C)\hfill \\ \hfill o_t& =\sigma (W_o·[h_{t-1},x_t]+b_o)\hfill \end{array}$$

(2)

where $W_{*}\in R^{4d_h}$, which is the weight matrix between the input layer and the hidden layer; $b_{*}\in R^{d_h}$, which is the deviation vector between the input layer and the hidden layer; and $d_h$ is the input dimension obtained in the question embedding.

In the process of training, correct or incorrect answers to questions will affect students’ status in different ways, and it is necessary to distinguish their different effects on specific students. Specifically, when training on the t-th question in the sequence, the input of the model should be embedded with question $x_t$ and the corresponding score $r_t$. First, the score $r_t$ is extended to the question embedding $x_t$. With the same dimension $R^{d_0+d_q}$, the feature vector $\mathbf{0}=(0,0,\cdots ,0)$ is obtained, and then the score vector and the question vector are combined to get the input vector $x_t^{+}$, as follows:

$$x_t^{+}=\left\{\begin{array}{cc}[x_t\oplus \mathbf{0}]\hfill & r_t>0\hfill \\ \mathbf{0}\oplus x_t]\hfill & r_t<0\hfill \end{array}\right.$$

(3)

The neurons of the model are divided into upper and lower parts (half of the value of the input vector is 0), where $W_{*}$ can be divided into $W_{*}^{+},W_{*}^{-}\in R^{2d_h}$. This can be understood as using two models to capture the impact of the question $e_t$ and reflect it in real-time during the training process.

2.3. Model Output

This section presents specific strategies for predicting student performance. Student performance depends on students’ state and questions characteristics. For a typical sequence prediction task, it is assumed that the next state depends only on the current state, not on the previous sequence [19]. According to this theory, this paper uses the softmax layer to compress the hidden layer to obtain the dynamic matrix $S_t$, that is, the knowledge status. Inspired by DKVMN, one can predict the answers of the question $e_{t+1}$ according to the knowledge points $Q_{t+1}$ of the question. The equation is as follows:

$$\begin{array}{cc}\hfill y_{t+1}& =RPa\_LLM(Softmax\left(h_t\right),{\mathbf{Q}}_{t+1})\hfill \\ \hfill RPa\_LLM(x_i,y_i)& =\frac{\exp ({\lambda }_{i0}+{\displaystyle \sum _0^K}{\lambda }_{ik}{\omega }_{ik}{x}_{ik}^{*})}{1+\exp ({\lambda }_{i0}+{\displaystyle \sum _0^K}{\lambda }_{ik}{y}_{ik}^{*})}\hfill \\ \hfill {\omega }_{ik}& =1\{x_{ik}^{*}\ge y_{ik}^{*}\mathrm{and}{y}_{ik}^{*}=1\}\hfill \end{array}$$

(4)

2.4. Loss Function

This section introduces the loss function under the C-B-KT framework (the function used to measure the degree of difference between the predicted value and the actual value of the model)—the cross-entropy loss function. Since there is a log function in this function, after the derivation, the gradient is about $y-\widehat{y}$. The linear function of the parameter update amplitude is proportional to the error, and the optimal solution can be found more quickly. The equation is as follows:

$$loss=-alog\left(p\right)-(1-a)log(1-p)$$

(5)

where p is the predicted value of the model, a is the true value of the data, and $loss$ is the loss value.

3. Experiment

To verify the effectiveness of the proposed method, a series of experiments are conducted.

3.1. Setup

Public datasets do not contain the question texts that are needed for the proposed scheme; thus, in this paper, real questions are collected and labeled from a university calculus textbook. In this dataset, a total of 715 questions are labeled by n graduate students of computer science over two months. Labels include question text, knowledge points, the relations between knowledge points, and other information. The real answer data of students come from our real admission collection of nearly 1000 students, and a total of 82,517 behavior data are collected. Then, according to the real data distribution, 3000 students’ answering behaviors are simulated (See Figure A1).

To evaluate the effectiveness of the proposed method, several related works are compared. For fairness, all models are tuned for optimal performance. The details are as follows:

• DKT: The classical DKT model [17], in which the data embedding is based on the one-hot method.
• EERNN: Exercise-Enhanced Sequential Modeling [15]—the improved DKT model, in which the data embedding is based on the bidirectional LSTM method.
• R-DKT: Random Deep Knowledge Tracing—the clustered DKT model, in which the clustering algorithm is based on the random method.
• B-KT: Bert Knowledge Tracing—the improved DKT model, in which the data embedding is based on the Bert method.
• C-DKT, C-EERNN: Clustering Deep Knowledge Tracing and Clustering Exercise-Enhanced Sequential Modeling—the improved DKT models, in which the clustering idea is added to the basic models.
• C-B-KT: The proposed scheme, which uses the idea of clustering based on B-KT.

A qualified model for student performance prediction should have good results from both regression and classification perspectives. The output value is a continuous probability value, and regression indicators can be used to calculate the error of the continuous value. In this work, the sum of squares due to error (SSE) is used to evaluate the density of the cluster. The mean absolute error (MAE), the mean squared error (MSE), the root mean error (R2), and Root Mean Square Error (RMSE) are used to evaluate the error of the proposed methods. They are also used in other related work [15,20]. The Area Under Curve (AUC), Recall, Precision, and F1-score are used to evaluate the accuracy of models. They are also used in other related works [21,22,23,24,25]. The calculation details are as shown in

Table 2. Calculation details.

SSE	F1-Score	Recall	Precision
${\displaystyle \sum _{i=1}^m}{(x_i-y_i)}^2$	$\frac{2TP}{2TP+FP+FN}$	$\frac{TP}{TP+FN}$	$\frac{TP}{TP+FP}$
MAE	MSE	RMSE	R2
$\frac{1}{m}{\displaystyle \sum _{i=1}^m}|h\left(x_i\right)-y_i|$	$\frac{1}{m}{\displaystyle \sum _{i=1}^m}{(h\left(x_i\right)-y_i)}^2$	$\sqrt{\frac{1}{m}{\displaystyle \sum _{i=1}^m}{(h\left(x_i\right)-y_i)}^2}$	$1-\frac{{\displaystyle \sum _{i=1}^m}{(x_i-y_i)}^2}{{\displaystyle \sum _{i=1}^m}{(\overline{x}-y_i)}^2}$

TP: Positive samples predicted as positive by the model; TN: Negative samples predicted as negative by the model; FP: Negative samples predicted as positive by the model; FN: Positive samples predicted as negative by the model. $x_i$ represents the predicted value; $y_i$ represents the true value; $\overline{x}$ represents the average of the predicted values.

.

The configuration of the proposed models is set as follows:

Question embedding supplements. The words of the mathematical questions embedded in the question are different from the traditional ones. To preserve the mathematical semantics, a formula tool in the Spotlight mechanism is used to convert each formula into its TEX code feature [26].

Frame settings. The dimension of each embedded problem, $d_0$, is set to 768; the dimension of the hidden state in the student embedding, $d_h$, is set to 100; and the dimension of the dynamic matrix in the output layer, $d_K$, is set to K, which is equal to the number of knowledge points.

Training settings. Initialize all parameters of C-B-KT, with the range of $0.5\pm 0.1$. In addition, the $batch$ size is set to 32 and the dropout is set to $0.5$ to prevent overfitting.

Determine the optimal number of clusters. As for K-means, the number of clusters should be determined manually. This article uses the elbow method (an algorithm for calculating the optimal clustering number k) to determine the optimal number of clusters, as shown in Figure 4.

In the results, when the number of clusters k increases, the samples are divided more finely and the degree of aggregation of each cluster increases, so SSE decreases. When k is less than the optimal number of clusters, an increase of k will significantly increase the degree of aggregation of each cluster and therefore decrease the SSE significantly. When k equals the optimal number of clusters, the degree of aggregation obtained by increasing k will rapidly decrease, so the downward trend of SSE will decrease and then tend to level off as the value of k continues to increase. This means that the figure composed of $SSE$ and k is the shape of the elbow. When the curvature is maximum, the k value is the optimal number of clusters for the data. The curvature of the Figure 4a,b is greatest when $k=3$. Therefore, the student data and question data are divided into three classes.

3.2. Results

3.2.1. Prediction Performance

The core of this experiment is to observe the impact of data clustering and data coding on the accuracy of the model prediction.

The prediction results are presented in

Table 3. Model result analysis.

Model	Classification Indicators (%)				Regression Indicators
	AUC	F1	Recall	Precision	MAE	RMSE	MSE	R2
DKT	81.10	72.89	74.29	71.54	0.36	0.42	0.18	0.29
R-DKT	81.21	72.22	70.69	73.81	0.34	0.42	0.18	0.29
EERNN	85.73	80.27	79.36	81.20	0.25	0.36	0.13	0.49
B-KT	89.94	81.89	81.26	82.52	0.26	0.36	0.13	0.49
C-DKT	84.55	75.70	74.52	76.92	0.33	0.40	0.16	0.36
C-EERNN	88.49	83.41	80.90	86.07	0.24	0.35	0.12	0.50
C-B-KT	91.01	83.15	81.26	85.14	0.24	0.35	0.12	0.52

. Several conclusions can be gained (see Figure A2 for a more intuitive R-P curve). Firstly, the proposed C-B-KT model shows the best in all metrics. This result proves that the proposed classification idea of the C-B-KT model can indeed improve the prediction accuracy of the KT models. Secondly, the C-DKT, C-ERNN, and C-B-KT models produce better results than DKT, EERNN, and B-KT models on all indicators. This indicates that the clustered data can be more accurately captured by neural networks. This improves the prediction accuracy of KT. Then, the B-KT model is superior to the classical DKT model and the same type of EERRN model. This indicates that the Bert model has a better ability to extract question features than the traditional one-hot model and bidirectional LSTM model. The addition of Bert to the B-KT model further reduces the information loss caused by feature-based or specific knowledge representation in existing methods.

3.2.2. Training Efficiency

This experiment aims to verify the performance improvement of the clustering-based neural network model.

Based on the results in

Table 4. Performance Analysis.

	Knowledge Tracing							K-Means
model	DKT	C-DKT	R-DKT	EERNN	C-EERNN	B-KT	C-B-KT	MyData
Time (s)	7039	2986	1721	17051	4134	6955	2190	44.94

, several conclusions can be gained. Firstly, compared with the traditional B-KT, EERNN, and DKT models, the C-B-KT, C-EERNN, and C-DKT models have an average time reduction of 67.28%. The EERNN model has the highest performance improvement, up to 75.75%. Although the R-DKT model has the highest training efficiency, its prediction accuracy did not improve compared to the DKT model. This indicates that the clustering-based neural network model can improve the training efficiency of the model through parallel training. Then, the training efficiency of the C-DKT model is higher than that of the DKT model but lower than that of the R-DKT model (random clustering makes the data amount more average). The DKT model has not undergone the processing embedded in this article and still has the same experimental results. This shows that the improvement of the model performance is mainly related to the data volume and not to the text information embedding.

After applying the clustering idea to all models, it performs very well in terms of both prediction accuracy and training efficiency, with good versatility.

3.2.3. Visualization Analysis

The core purpose of this paper is to construct student portraits. The dynamic matrix, $S_t$, in Figure 3 is the students’ knowledge state matrix. The data information is extracted and visualized as shown in Figure 5.

3.2.4. Clustering Model Analysis

To verify the idea that clustering can reduce the noise between different groups and improve the prediction accuracy, this section shows the prediction results of each model after clustering as presented in Figure 6.

Considering the classification indicator, AUC, except for the “31” model, the prediction accuracy of other models has been significantly improved. This indicates that after clustering data features, training data that have the same features can improve the prediction accuracy of the model, and the overall prediction accuracy after clustering is higher than that “Non-Clustering”, which indicates the high feasibility and effectiveness of clustering.

Considering the regression indicator, MAE, the regression index of the independent model has decreased. This indicates that the clustering-based model is closer to the real data than the “Non-Clustering” based model, which further proves the high effectiveness of clustering.

Observing the time indicator, the training-time of all models based on clustering is significantly shorter than that of the “Non-clustering” model, and the overall training efficiency is improved by 57.57%. It shows that the training efficiency of the model is inversely proportional to the amount of data.

In summary, although the amount of data for the model decreased, the accuracy of model prediction did not decrease, and the efficiency of model training was greatly improved.

4. Discussion

In this paper, we study the impact of data features on KT models and use the relationships to propose a KT model based on clustering neural networks.

In this study, we show that the input data features would impact the performance of KT models. Firstly, the performance of the model is significantly improved after clustering. This result shows that data clustering impacts KT models by partitioning data features. Secondly, data clustering provides a solution for distributed deployment and improves training efficiency. Finally, the student’s knowledge state matrix $S_t$ is added to optimize the KT model and enhance interpretability.

This paper uses the control variable method to conduct experiments and analyze the data and finds that random clustering does not affect the accuracy of KT, which further shows that our method is effective and improves the credibility and accuracy of research.

In this paper, a clustering algorithm is used to train the data separately, which improves the prediction accuracy of the model and provides a distributed training scheme for the KT model. This paper uses the diagnostic classification model RPa-LLM, an educational theory, to display and trace students’ knowledge status, which further improves the interpretability of the model.

5. Conclusions

This paper makes a comprehensive study of the problem of student performance prediction. Firstly, an enhanced neural network framework, B-KT, is proposed to explore the effect of question text on KT. B-KT can effectively predict students’ performance in future questions. Secondly, the B-KT framework is extended to the C-B-KT model. The clustering algorithm can help the KT model to discover the hidden rules and structures in the dataset so that the neural network can better understand the data and improve the model’s prediction accuracy. Thirdly, the addition of the clustering module enables parallel training, which improves training efficiency. Then, to verify the effects of clustering and question embedding, the C-B-KT is compared with C-DKT and C-EERNN models. The C-B-KT can cluster and trace the historical focus state for prediction, which is superior to all other models. Finally, the classification discussion and the addition of the student’s knowledge state matrix $S_t$ brought by clustering improve the interpretability of KT.

In this study, we only investigated the effect of K-means clustering on KT models, without considering the effect of other potential factors. In the future, different clustering algorithms can be used to explore the impact of data characteristics on KT from the perspective of data distribution changes. The deep integration of diagnostic classification models and neural networks can be further explored.