Grade Prediction Modeling in Hybrid Learning Environments for Sustainable Engineering Education

Abstract

Since mid-March 2020, due to the COVID-19 pandemic, higher education has been facing a very uncertain situation, despite the hasty implementation of information and communication technologies for distance and online learning. Hybrid learning, i.e., the mixing of distance and face-to-face learning, seems to be the rule in most universities today. In order to build a post-COVID-19 university education, i.e., one that is increasingly digital and sustainable, it is essential to learn from these years of health crisis. In this context, this paper aims to identify and quantify the main factors affecting mechanical engineering student performance in order to build a generalized linear autoregressive (GLAR) model. This model, which is distinguished by its simplicity and ease of implementation, is responsible for predicting student grades in online learning situations in hybrid environments. The thirty or so variables identified by a previously tested model in 2020–2021, in which distance learning was the exclusive mode of learning, were evaluated in blended learning spaces. Given the low predictive power of the original model, about ten new factors, specific to blended learning, were then identified and tested. The refined version of the GLAR model predicts student grades to within ±1 with a success rate of 63.70%, making it 28.08% more accurate than the model originally created in 2020–2021. Special attention was also given to students whose grade predictions were underestimated and who failed. The methodology presented is applicable to all aspects of the academic process, including students, instructors, and decisionmakers.

1. Introduction

Traditional university education systems have been totally disrupted due to the COVID-19 health crisis that has been affecting the world since mid-March 2020. The immediate and often precipitous transformation of face-to-face learning environments into virtual online spaces has prompted trainers, instructors, and teachers to integrate numerous technology tools and platforms into the educational process [1,2]. Learning management systems (LMS) and learning platforms have been combined with social networking sites (SNS) to transmit as many synchronized courses as possible, share new knowledge, and facilitate academic communication on task support [3,4,5,6].

In the first year of this health crisis, when online education became the exclusive mode of learning, researchers became aware of an unexpected challenge: the transformation of the educational schema through the incorporation of technological tools that previously remained inert or undiscovered by students and instructors. Several aspects of these new learning environments have been investigated, revealing new parameters that affect the academic performance of students in exclusively distance learning environments, while demonstrating the effectiveness of the technological tools used, as well as their sustainability [6,7,8,9]. Virtual learning environments (VLE) could also make a contribution in this still uncertain global health context, especially since distance learning and asynchronous support for knowledge transmission have already been tested in strictly distance learning environments [7]. Blended learning environments could also benefit from the above findings.

Although the COVID-19 pandemic was neither predictable nor completely controllable at the beginning of the 2021–2022 academic year [8], policymakers decided to reopen universities so that students would lose as little of the close connection with faculty as possible. The educational models that were used prior to this reopening of the universities’ doors were able to expand their use, but this time there was no “exclusivity” in the modes of instruction. Indeed, at some universities, instructors were teaching lab modules and theory courses in physical learning spaces, but at the same time, they were delivering their courses synchronously, using the learning platforms applied during the first phase of the pandemic. Once again, new learning environments were created, mixed or virtual, with technological features being the headliners of the educational procedure.

The first semester of the 2021–2022 academic year ushered in a new period of exciting opportunity: the era of technological and sustainable transformation of the learning process. Blended learning environments have been created to meet the needs of all learners, regardless of their background. First-year students can take courses in face-to-face learning environments on university campuses, working students can take online modules, and instructors can deliver courses synchronously from their university laboratories.

The objective of this work is to study in depth the factors affecting the academic performance of students in hybrid learning environments, i.e., coupling face-to-face and distance learning. Obviously, we will use the results already obtained in situations exclusively carried out at a distance, especially those imposed during the most critical period of the health crisis. This work aims to contribute to the construction of sustainable higher education in an era of digital transformation of educational systems [9], which we refer to here as meta-COVID-19. The term “meta”, which is used as a prefix, means “beyond” in ancient Greek. It implies that, although much remains to be done to achieve a transformation of both pedagogical tools and practices accepted by all actors in the academic world, this is part of a promising future ahead.

Like any crisis situation, the still unstabilized global health context continues to push all actors in the academic world to review their ways of doing things, and in particular to analyze in detail the impact of information and communication technologies in blended learning situations. This is the topic of this article, which aims especially at identifying the most important factors that contribute to effective knowledge sharing through hybrid and sustainable teaching processes.

The three research questions that led to this study are:

• What factors are common, and what factors might emerge, that affect student performance in online and blended learning environments?
• Which factors that were significant in online learning environments carry less weight in blended learning sessions?
• Can a grade prediction model performed in online circumstances have the same success rate in hybrid environments?

Answers to the above research questions should contribute to the creation of sustainable learning environments, linked or not to physical, distance, and blended learning workspaces.

The outline of this article is as follows. A literature review is provided in Section 2 to highlight the contributions of this article in relation to what is already available to the educational community. In Section 3, the proposed methodology includes data collection from surveys, communication platform reports, and student attendance reports. In Section 4, the analysis of the obtained results relies on a statistical analysis (Spearman correlation) to filter the most significant predictors. A thorough revision of a GLAR-type (generalized linear auto-regression) statistical model, which has already proven successful in distance-only learning modes, tests the performance of embedded predictors during the online spaces of blended learning situations. Section 5 finally proposes a discussion based on the presented results.

2. Related Works

The notion of sustainable learning [10,11] has been structured around a wide variety of motivations, targets, and goals for finding solutions to the problems faced, based on the challenges presented.

The renewed agenda for higher education policy development in the European Union (EU) countries [12,13] has identified four key objectives. These include the contribution to innovation of higher education institutions, as well as support for effective learning systems. Blended or mixed learning environments with similar online and hybrid learning frameworks could be incorporated into the learning process, as they have been tested and found to be adequate to benefit the university community.

From the beginning of the health crisis, a plethora of digital tools were implemented to support synchronous and asynchronous learning [2,14,15]. Learning management systems, learning platforms, social media channels, and mobile learning platforms were applied and evaluated by performing statistical analysis using electronic data obtained from online surveys and learning platform activity reports [14,15,16].

Blended learning can be defined as a combination of practices applied in face-to-face and online learning environments [17,18]. Colleges and universities around the world are the leading knowledge-sharing institutions that aim to provide scalable information in a modern, lifelong educational system [19]. Comparing physical learning environments to online ones has always been a challenge for researchers [20]; it has been shown that online enrolments have increased drastically during the period 2002–2007, due to the introduction of information and communication technologies (ICT), inviting students to participate in module instruction from the comfort of their own homes. The authors of [21] defined hybrid learning as a learning mode where parts of the physical educational procedures have been replaced by online modes of knowledge transmission. The hybrid learning model is an Internet-based technology model that can be considered a “sustainable” model [22]. It can offer alternative possibilities for user-learners and create flexible online learning environments [23,24].

In [25], the development of critical thinking and problem solving skills has always attracted educational societies, and examples can be found in mechanical engineering. For example, the authors of [2] developed an innovative methodology for teaching computer-aided design in a first-year mechanical engineering module. The learning objectives were to design the meaning of drawings in object views, as well as the ability of students to establish the relationships of tasks assigned to their future work as mechanical engineers. In [26], the authors evaluated the design of geometric object projection in vertical and horizontal planes. In [27], the authors used a survey to obtain the feedback needed to redesign the teaching process of a computer-aided design module. All of these examples show that policymakers need to accelerate reforms in higher education whilst taking into account that skills and knowledge are interrelated [25].

As the limitations of social distance have become less restrictive, institutions around the world have returned to face-to-face learning environments, and instructors have been faced with a dilemma regarding which existing components of emergency remote teaching environments (ERTE) [28,29] will be maintained during this early period of meta-COVID-19 and which will be discontinued [30].

A recent study on students’ perception of the educational challenges induced by the COVID-19 pandemic [31] showed that learners prefer combined modes of instruction. In [32], the authors created a multiple linear regression model for grade prediction in a first-year mechanical engineering CAD module in exclusively distance learning environments. A total of 146 first-year students participated in this research and 35 variables were filtered from several sources (surveys, learning platform attendance reports, and university registration files). This model predicted students’ grades on their final exams at a threshold of ±1 with a pass rate of 67%. A generalized linear auto-regression model (GLAR) [32,33] was constructed and its variances were used as an additional predictor for an artificial neural network. This hybrid model was trained and yielded a coefficient of determination equal to 1. In [34], the authors used machine learning and feature analysis methods to assess student behaviors. In [35], the authors processed behavioral log data of blended learning modes from multiple learning platforms and introduced a multiple classification model to predict learners’ educational performance. In [36], the authors investigated the importance of predicting the next term’s grades to ensure students’ retention in college until graduation.

Given this comprehensive review of the literature in response to the research questions listed above, a GLAR model was constructed to predict grades in blended learning environments. This GLAR model was preferred over a multiple regression model due to the fact that the variables implemented are nominal, ordinal, and continuous. This model must be simple enough to be easily implemented and used, while ensuring a satisfactory level of prediction.

The main contributions of this work can be communicated to all members of the academic community:

• Students can be warned in advance of a potential risk of failure and take appropriate steps to overcome their deficiencies and better prepare for their final exams [36].
• Instructors can more appropriately manage course flow and provide individualized support to students at high risk of failure.
• Decisionmakers can reevaluate the teaching strategies applied in the first meta-COVID-19 period after the end of each semester, reshape them, and develop new viable learning modes for continuous and sustainable graduate engineering education.

3. Materials and Methods

3.1. General Presentation of the Proposed Approach

The work presented here took place at the Department of Mechanical Engineering, School of Engineering, University of West Attica, Greece. Two periods were selected: the first during the first semester of the 2020–2021 academic year and the second during the first semester of the 2021–2022 academic year. During the first period, students worked exclusively in distance learning environments. In the second period, mixed modes of instruction were used. Regardless of the period and in order to ensure the relevance of the proposed analysis, the investigations focused on a computer-based mechanical design module entitled “Mechanical Design CAD I”. This is a first semester laboratory module aimed at introducing students to the concepts of 3-Dimensional drawings as well as their representation using computer-aided design tools. The main results obtained have already been analyzed in the literature [32,37].

As health conditions gradually improved around the world, the same learning approach used previously was applied in a hybrid learning environment as a parallel asynchronous support mechanism.

As shown in Figure 1, the study itself was structured around the following two stages.

3.1.1. First Stage

The first stage (see Figure 1) implements and tests the relevance of a linear model dedicated to grade prediction in exclusively distance learning environments. For this, we rely on data from 2020–2021. To go further and test the robustness of the model, we enriched the current database with a new population of students, namely that of the first semester of the 2021–2022 academic year. Thus, the GLAR model—having learned from the 2021–2021 data—would be able to predict students’ grades in a face-to-face, online, blended, virtual, and combined learning environment.

It should be noted that, for this first test stage, we made the assumption that the significant statistical variables used on the model created during online spaces would keep their significance in hybrid learning spaces. It turned out that was not the case; however, this step was imperative in order to proceed to the following stage.

3.1.2. Second Stage

The second stage (see Figure 1) evaluates the success rate of the previous model and proposes new methods for predicting grades in current blended learning modes. The goal is to adapt the existing model to determine the percentage of prediction in the current blended learning spaces. Thus, a second entirely new GLAR model is proposed to fit the newly acquired data, i.e., 2021–2022 students who benefited from a blended learning environment.

The construction procedure of the GLAR model is currently being published by the authors of [32]. Nevertheless, we recall here its main foundations. As shown in (1), the variables selected by the filtering process are isolated to construct a function to relate student grades ($y$) to the most significant dependent variables in the database. More specifically, $x_1,x_2, x_3, \dots , x_n$ are the predictors that relate to the variables that were found to have no statistical association with the students’ grades.

$$y=f\left(x_1,x_2,x_3, \dots ,x_n\right)$$

(1)

The objective of the GLAR model is to generalize the characteristics of a linear regression model. Using parameters such as the mean response, the response variable can follow a regular, binomial, Poisson, gamma, or inverse Gaussian distribution. Thus, as shown in (2), the relationship between a dependent variable and the linear combination of altered predictors is defined by a set of link functions that transform each independent variable into a generalized linear regression model. In (2), $y$ is the response variable; $x_1,x_2, \dots , x_n$ are the predictor variables; ${\beta }_0$ is the $y$-intercept, ${\beta }_1,{\beta }_2, \dots , {\beta }_n$ are the regression coefficients; and the intercept represents the residual error.

$$y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+\cdots +{\beta }_n{x}_n+\mathrm{intercept}$$

(2)

The results of the work presented in [32], which s conducted in the first semester of the 2020–2021 academic year, identified several factors that had a significant impact on students’ academic performance during the pandemic. As the identified factors do not all have the same weight, the presented methodology was applied to the first semester of the 2021–2022 academic year in order to test the impact of each variable and thereby reach more accurate conclusions. The general diagram of the research methodology is schematized in Figure 2.

3.2. Data Sources

The data used in the modeling comes from online surveys conducted in 2020–2021 to detect failing students. Compared to the work already available in [32], new questions, mathematically corresponding to new variables for statistical analysis, were incorporated into the existing survey. The objective was to interpret the behavior of students, especially in terms of how they attend laboratory activities. Attendance lists were analyzed to distinguish between students who were physically present at the lab activities and those who took the course online. For each group of students in the course module, we identified the total number of courses taught during the semester (over 12 weeks), the number of courses taken face-to-face, and the number of courses taken online.

The pedagogical approach initiated by the authors of [2,5] was applied to students in the first semester of the 2021–2022 academic year, i.e., in a face-to-face, blended, or virtual learning environment. The objective is to evaluate the sustainability of the methodology suggested by previous experiments in an early meta-COVID-19 period or in future blended learning environments. To do so, the evaluation methods used exactly the same tools and in particular the same communication platform, while taking into account the changing guidelines imposed by the health situation [38].

In addition to the different categories of data sources, ex-final test results were added to make a comparison between the predicted and actual score. The threshold of success for the model was set to ±1. Finally, the difference between the predicted and the actual grade induces a new statistical variable that we call the error. When the value of the error is greater than +1, the student’s grade is overestimated by the model. When this error is less than −1, the model underestimates the student’s grade [32]. The grade considered for analysis is 70% of the final grade obtained after taking into account the students’ average grades on the assignments. Students with grades above 4.5 are considered to have passed their exams.

3.3. Applied Blended Learning Modes and Assessment of Student Performance

The “Mechanical Design CAD I” module in the first semester of the 2021–2022 academic year was conducted in a blended/combined teaching mode. The module was delivered over 12 weeks, in accordance with the university’s academic calendar. As shown in Figure 3, for the first four weeks, the educational process took place in a laboratory at the University of West Attica, without the possibility of online participation. This specific strategy was implemented to facilitate the familiarization of the first semester students with the facilities and equipment of the university [36,39].

In order to achieve similar conditions for instructors and students, the same scoring rubric was applied. Each instructor was asked to evaluate each student’s work based on the following criteria: completeness and accuracy of the three CAD views of the object, including a front view, a top view, and a cross-sectional view; correct use of layers and line types; grading; and regular student participation during the semester. The same rubric was applied in the online learning environments. The final exams took place during the 12th class in the university lab, face-to-face for all students.

3.4. Participant Demographics

The “Mechanical Design CAD I” module was delivered to 212 students from all levels of education (first year, second, third, and final semester students). In total 82.7% were male and the remaining 17.3% were female. Their average age range was 18–21 years (87.2%). Among the information from the forms used to collect the responses to the online survey was information about the students’ family status: 65.5% lived with their families and only 16.2% lived alone. The remaining 18.3% lived with siblings or friends.

The sample was processed and—once the survey responses of the final year students had been eliminated in order to be as close as possible to the population that participated in the 2020–2021 academic year—culminated in 146 valid responses.

Beginning in the fifth lab session, students were given the option to choose the mode of participation, i.e., face-to-face or online, with lessons delivered synchronously via the MS Teams platform. Instructors encouraged virtual learning modes as a health precaution, given the increase in pandemic cases at the time, to avoid concentration and close social contact among students. Students were asked to express their reasons for choosing their mode of participation. A total of 185 of 212 students responded to this survey and the main results are presented in Figure 4 and Figure 5. As shown in Figure 4, the top two reasons for taking the module remotely were distance from home (about 49% of the responses obtained) and comfort taking the course (about 38% of the responses obtained). Restrictions imposed by the health crisis came in third place with 20% of the responses obtained. As shown in Figure 4, the reasons for choosing to take the course face-to-face are more disparate. Students feel that they are better able to assimilate the course being taught (about 55% of the responses obtained). The reasons are also social, to keep in touch with the teachers (about 49% of the responses obtained), but also other students (about 30% of the responses obtained). Students also chose this face-to-face course because they had other face-to-face courses during the same period (about 43% of the responses obtained). Finally, the motivations were also of a practical nature, with the possibility of asking questions directly when certain points of the course needed clarification (about 32% of the responses obtained).

4. Main Results

In this article, strictly based on the approach implemented in 2020–2021, we chose to evaluate the relevance of the GLAR model applied when teaching the 2021–2022 “Mechanical Design CAD I” module, which was delivered online in hybrid learning environments. The main results obtained are analyzed in this section.

4.1. Relevance of Grade Prediction Modeling for Online Learning Spaces in Hybrid Systems

Before applying the GLAR model to the 2021–2022 data, it is worth recalling its relevance to the 2020–2021 data, i.e., in exclusively online teaching environments. Under these conditions, the prediction reached 67% success [32] and the fitting errors of the GLAR model were used as additional predictors. A hybrid model was thus created, leading to a coefficient of determination ($R^2$) equal to unity. It should be noted that the technique used was chosen taking into account that at the time the experiment took place, there was no prior population that could be included in the sample. Therefore, the dataset was divided into three subsets, where 70% of the population represented the training set, 15% the test set and the remaining 15% the validation set [32,33]. The data fitting process was applied to the training set, and the prediction success rate depended on the prediction of student scores in the other subsets with minimum mean square error.

If we now apply this GLAR model as-is to predict student grades in 2021–2022 in blended learning environments, the results are 70.65% pass, 44.44% fail, and 35.62% correct prediction in the range of ±1. The model underestimated 54.79% of students’ actual final grades and overestimated 45.21%. Figure 6 shows the error in grade estimation and it can be seen that the majority of the errors are in the range of −0.47 to +2.33.

The results we obtained are not fully satisfactory, and adjustments to the existing GLAR model are therefore required to achieve the same levels of accuracy as those displayed in terms of the 2020–2021 population.

4.2. Statistical Analysis Applied to the 2021–2022 Population

The 2021–2022 sample consisted of 146 students. In order to identify the variables that have a significant relationship with the students’ final exam grade, a Spearman correlation analysis was performed. The purpose of this analysis is to establish some degree of quantification of the correlation of all independent variables with the dependent variable (grade). The statistical analysis proposed here was performed in MATLAB R2020a.

All categorical variables were transformed into a series of dummy variables, corresponding to the categories contained in a categorical variable. Each dummy variable was assigned the value 1, if an observation satisfied the following criterion “Is the $i$-category true?” and the value 0, otherwise. This was done for all observations.

If there are $L$ categories related to a categorical variable, then $L-1$ dummy variables are created. The dummy variables that were transformed using this method are “class group”, “instructor ID”, and “gender”. Specifically, the “class group” variable expresses the number of 11 three-hour lab groups taught by one of the four instructors. After transformation, ten dummy variables were created (11 −1 = 10). The remaining variables did not need to be transformed, as they were ordinal or continuous variables.

4.2.1. Analysis of the Spearman Correlation Coefficient (Rho)

In order to analyze the significant correlations between 68 variables, a correlation matrix was created, aiming to measure the Spearman correlation coefficients (Rho), where values close to 1 imply a positive correlation and close to −1, a negative correlation [40]. Values near zero are considered negligible. Students’ ability to complete their tasks after completing three months of laboratory instruction, compared to their final grade, is presented in

Table 1. Variables positively and negatively correlated with students’ ability to complete assignments (Spearman’s Rho correlation coefficients).

Correlated Variable	Rho	Correlated Variable	Rho
Usefulness of videos demonstrating the sketching methodology	−0.24	Did you fully conceive the objects assessed?	0.30
Study time per week for CAD I *	−0.22	Enjoyable versus other Labs *	0.32
Sustainability of the learning process *	0.18	Enjoyable CAD I compared to other theories	0.34
Assignments related to future tasks as a Mechanical Engineer *	0.19	Attend the four first lectures face-to-face	0.35
Comfortable for final exams of all modules	0.21	Have you noticed any weaknesses during CAD I Lectures? *	0.37
Theory contributes on understanding the topics	0.23	Number of lectures performed	0.38
CAD I preferred to other modules	0.24	Did you understand the concept of planes? *	0.38
Helpful the presence of Cutting planes in Blue in 3-D views *	0.27	Classroom fatigue *	0.39
Are your questions answered during lectures?	0.27	Rate CAD I out of 10 *	0.44
Receive help on assignments?	0.27	Insecurity *	0.52
Computer skills	0.28	Managing knowledge gaps *	0.55
Evaluation versus other modules *	0.28	Comfortable for CAD I final exams *	0.58
Evaluate workload	0.28	Likely to succeed in a similar future task *	0.59

* Variables that were statistically significant in 2020–2021 [32], i.e., in exclusively online environments.

. We recall that in statistics, the Spearman correlation is an interesting non-parametric alternative to the Pearson correlation. The Spearman’s Rho parameter is a very good asset here, especially when the data to be processed are ordinal, as is the case here. In Table 1, the prefix * distinguishes variables that were statistically significant in 2020–2021 [32], i.e., in exclusively online environments.

As shown in Figure 6, of the 68 variables that were tested by correlation analysis, calculation of Spearman’s coefficient (Rho) indicated significant new positive and negative correlations when these variables were correlated with students’ final exam score.

In a Spearman correlation analysis, looking only at the values of Rho between −1 and 1 is not enough. When Rho is positive, when one variable increases, the other also seems to increase. When Rho is negative, either the variables are seemingly inversely related, or when one variable seemingly increases, the other decreases. In what follows, we will examine another variable called the p-value, which represents the statistical significance of a Rho coefficient resulting from the comparison of two variables. In particular, we will take the 5% threshold value classically used in statistics. In particular, when the p-value is less than or equal to 5%, the result is statistically significant, allowing us to affirm that the difference is not only the result of chance. The significant variables with p-values ≤ 5% are shown in Figure 7.

4.2.2. Analysis of Spearman’s Correlation p-Values

The objective here is to determine which variables have the lowest p-values (i.e., less than or equal to 5%), when compared to the students’ academic results; results expressed by their final exam grade. Specifically, variables with p-values less than or equal to 5% [41] reject the null hypothesis [40], which in our case is “the variables are not related to the final exam score”.

Table 2. Variables with p-values less than or equal to 5%.

Variable	p-Value	Variable	p-Value	Variable	p-Value	Variable	p-Value
Class group 03	0.05%	Class group 05	0.13%	Class group 09	0.00%	Age	1.00%
Able to complete assignments after 3 months	0.08%	Assignments related to future tasks *	0.40%	Enjoyable versus other Labs *	0.20%	Comfortable for final exams *	0.19%
Assimilation of the concept of planes *	3.50%	Pleasant compared to other theories	0.03%	Likely to sucked in a similar future task *	0.30%	Assignments relevant to future work *	1.90%
Managing knowledge gaps *	1.50%	Instructor number 3	0.09%

* Variables that were statistically significant in 2020–2021 [32], i.e., in exclusively online environments.

shows the variables in favor of the alternative hypothesis.

As shown in Table 2, fourteen variables were selected as predictors with the value of less than or equal to 5% as a criterion for filtering the data. Of these fourteen selected variables, seven of them contributed to the development and implementation of the GLAR model used in 2020–2021 to predict student grades in exclusively online learning environments.

4.3. Validity of the Data Collected

In this section, we propose to perform a statistical test to verify the validity of the students’ responses to the surveys. Cronbach’s alpha is an excellent tool to determine the consistency of the set of survey responses. This coefficient, ranging from 0 to 1, is used to determine the degree of reliability of the variables from the correlation analysis [42]. For this test, ordinal variables were considered. These variables were expressed on a 5-Likert scale, referring to the survey questions that were ordinally formed and applying the same scale to measure the objective. Although the questionnaire was originally designed to study several constructs involving module teaching and student behaviors, the test presented below focuses specifically on the teaching methodology construct [43]. According to a relatively broad consensus, the minimum acceptable value for Cronbach’s alpha is 0.70. A lower value means that the test questions do not measure what they are supposed to measure. The maximum expected value is 0.90. However, a score above this value does not necessarily mean that internal consistency is high, but rather that there is redundancy or duplication of items. In concrete terms, this means that several questions measure exactly the same element of a construct.

Initially, all 146 participant responses were included in the test, with no exclusions. A total of 23 ordinal variables related to the success of the methodology in hybrid learning environments were more specifically isolated from the questionnaire, and the Cronbach’s alpha statistics of each of these 23 variables is presented in

Table 3. Cronbach’s alpha statistics of the 23 ordinal variables isolated from the questionnaires.

	Variables	Scale Mean If Item Deleted	Scale Variance If Item Deleted	Corrected Item-Total Correlation	Squared Multiple Correlation	Cronbach’s Alpha If Item Deleted
1	Comfortable for final exams of all modules *	90.34	62.556	0.36	0.35	0.826
2	Able to perform assignments after 3 months	89.05	60.032	0.61	0.56	0.815
3	Assessing the workload *	89.21	65.489	0.27	0.20	0.829
4	Theory contributes to the understanding of subjects *	89.32	63.020	0.30	0.21	0.829
5	Clarity of image and sound in videos *	89.03	65.302	0.20	0.19	0.832
6	Answers to the questions asked during the lectures *	88.54	64.940	0.29	0.17	0.828
7	Computer skills *	89.08	60.705	0.45	0.50	0.821
8	Social media application skills *	88.54	64.540	0.30	0.37	0.828
9	Assignments related to future tasks as a mechanical engineer *	89.21	62.968	0.36	0.32	0.825
10	User-friendly compared to other laboratory modules *	88.58	63.086	0.50	0.51	0.821
11	Familiarization with MS Teams *	89.07	64.575	0.22	0.21	0.832
12	Insecurity *	89.29	60.623	0.48	0.45	0.820
13	Comfortable for CAD I module final exams *	89.50	59.838	0.59	0.57	0.815
14	Quality of task evaluation on MS Teams *	88.95	65.625	0.25	0.35	0.829
15	Quizzes help to assimilate the theory *	88.80	63.788	0.30	0.27	0.828
16	Assimilation of the conception of planes *	88.87	60.859	0.56	0.42	0.817
17	Usefulness of the presence of cutting planes in blue *	88.74	63.132	0.38	0.24	0.825
18	User-friendly compared to other theoretical modules *	88.53	64.058	0.41	0.54	0.824
19	Identify weaknesses *	88.55	61.973	0.35	0.35	0.827
20	Basic software skills *	89.84	60.731	0.38	0.42	0.826
21	Likely to succeed at a similar task in the future *	89.31	60.242	0.61	0.59	0.815
22	Sustainability of the learning process *	89.55	63.807	0.28	0.26	0.829
23	CAD I preferred to the other modules *	88.66	63.714	0.44	0.51	0.823

* Variables that were statistically significant in 2020–2021 [32], i.e., in exclusively online environments.

. In the end, the Cronbach’s alpha test revealed a coefficient in the range of 0.831, proving that these 23 questionnaire variables are statistically consistent.

Different ranges of qualitative descriptions of Cronbach’s alpha can be found in the related literature, but the value of 0.831 is considered good, or “robust” [44]. Some specific variables did not discriminate understanding of deep structure as expected, decreasing the reliability statistic of the test. They were therefore discarded, as suggested in [45].

4.4. GLAR Modeling for Grade Prediction in Hybrid Learning Environments

To refine the existing GLAR model, predictors were selected to exclude all variables whose Spearman correlation coefficients (using student grade as the dependent variable) had p-values less than or equal to 5%. Based on this criterion, 14 variables were finally selected as statistically significant.

The refined GLAR model thus obtained has the following main metrics: a mean absolute error (MAE) equal to 0.79 and a mean absolute error percentage (MAP) equal to 18.50%. Figure 8 shows the histogram of errors. The majority of errors (61 of 146 cases) are in the range of scores from $-$0.37 to $+$0.63. Thirty-four cases are in the range of $-$1.37 to $-$0.37, thirty cases are in the range of $+$0.63 to $+$1.63, and, finally, fifteen cases are in the range of $-$2.37 to $-$1.37.

4.5. Comparative Analysis between the Two Models

As can be seen in

Table 4. Success criteria for the prediction of the two models.

	GLAR Model— 2020–2021 Release		GLAR Model— 2021–2022 Release
	Number of Students	%	Number of Students	%
Actual pass	92	63.01%	92	63.01%
Actual fail	54	36.99%	54	36.99%
Prediction “pass” correct	65	70.65%	82	89.13%
Prediction “fail” correct	24	44.44%	27	50.00%
Prediction ±1 correct	52	35.62%	93	63.70%
Overestimated	66	45.21%	71	48.63%
Underestimated	80	54.79%	75	51.37%
Overestimated and failed	30	20.55%	26	17.80%
Underestimated and passed	26	17.81%	10	6.85%

, the participation of the seven new variables and the reduction of the previous 35 to 14 increased the success rate of grade prediction in hybrid learning conditions.

For grade prediction to be relevant, it is important to detect learners who are on the verge of failure early enough to provide appropriate instructional support [34,35,36]. As shown in Figure 9, all of the 26 students who were overestimated and failed were present at the final exam.

5. Discussion

5.1. Statistical Analysis of Factors Affecting Students’ Grades

The results analyzed above show that some variables, from a statistical point of view, have little or no impact on student performance. In particular, when comparing the significant parameters affecting students’ final grades following their participation in exclusively online learning (2020–2021) and blended learning (2021–2022), we find, for example, student engagement as expressed in communication platform usage reports. In the exclusively online learning mode, this variable is highly correlated with the final grade [26]. In blended learning modes, its statistical significance is reduced, because unlike online learning where student activities can be assessed via reports from communication platforms, student engagement could not be measured in face-to-face learning environments. The number of lectures each student attended online or onsite were also other examples of variables that did not show statistical significance.

From Figure 7, seven variables have a prefix * indicating that they are the same variables identified in 2020–2021, when the learning modes were exclusively online and virtual [32]. Student activity on MS Teams, identifiable from the report downloadable via this communication platform, has a low correlation in 2021–2022 (Rho parameter equal to +0.12) with student grades, unlike in 2020–2021. In addition, further analysis of Figure 7 shows that the following six variables with the prefix * have a significant positive relationship with student grades:

• The feeling that students are comfortable taking final exams (Rho parameter of +0.25);
• Ability of students to perform assignments related to their future duties as mechanical engineers (Rho parameter of +0.23);
• Their sense of being able to cope with knowledge deficits (Rho parameter of +0.20);
• The probability of success in a similar future task (Rho parameter of +0.19);
• The enjoyment felt by the students compared to other more theoretical courses (Rho parameter of +0.19);
• Relevant assignments to learners for future work (Rho parameter of +0.19).

New variables identified in 2021–2022 include the ability to perform assignments after the last three months of teaching (Rho parameter of +0.27) and the direct impact of the instructors. On this last point, instructor 03 was positively correlated (Rho parameter of +0.27) with the dependent variable (grade). The same instructor taught the module in class group 09 (Rho parameter of +0.39), which had the highest correlation with student grades.

A close analysis of Figure 6 also shows that three variables have a significant negative relationship with students’ grades:

• The first is the course group, particularly groups 3 and 5. Course groups 03 (Rho parameter of −0.28) and 05 (Rho parameter of −0.26) seem to have a negative correlation with students’ final grades. Of the 11 class groups, 3 were morning classes (09:00–12:00), 6 were lunch classes (12:00–18:00), and the remaining 2 were evening classes (18:00–21:00). Class group 3 is a lunch class, while class group 5 is an evening class.
• The second is the age of the students (Rho parameter of −0.21), which is negatively related, which is explained by the fact that younger students are always more familiar with the technological features.
• The third is the instructor themself. The two groups of classes listed above were taught by the same instructor (number 4) and the number of students enrolled was the highest (24).

The variables that were originally included in the 2020–2021 prediction model (i.e., in exclusively online environments), and that in the new 2021–2022 model do not meet the criterion of a p-value of 5% or less, are:

• Technical difficulties (p-value of 94%) do not appear to be an issue in blended learning spaces. This is likely due to the fact that, after a year and a half of exclusively online learning, students have taken steps to obtain new electronic devices and faster Internet connections.
• Student activity on MS Teams, obtained from the communication platform reports (p-value of 13%), is no longer a representative indicator of learner engagement.

In hybrid learning environments, learners have the option to attend classes either face-to-face or remotely [11,14,16]. Since each student’s attendance was categorized by mode of presence (i.e., physical or remote), this characteristic deserved to be studied to assess its importance. Unexpectedly, the new metrics related to hybrid learning modes (not included in Table 2) did not meet the criterion of p-value less than or equal to 5%. Indeed, for the number of courses taught online and face-to-face in the first semester of 2021–2022, the p-values (on the order of 24%) were greater than 5%. The same is true for distance from home (p-value of 13%), the distinction between morning, mid-day, and evening classes (p-value of 10%), and online attendance due to the health crisis (p-value of 94%).

In the error histogram for the GLAR model created in hybrid learning environments (see Figure 7), the majority of errors (i.e., about 42%) fall in the score range of −0.37 to +0.63; about 23% of the cases fall in the range of −1.37 to −0.37; about 21% of the errors fall in the range of +0.63 to +1.63; and finally, slightly more than 10% of the errors fall above the range of ±2.

As shown in Table 4, the prediction of failing students increased from 44.44% to 50.00%. The prediction of grades in the ±1 range also increased from 35.62% to 63.70%. The number of students whose grades were underestimated and who ultimately passed was limited to 6.85% instead of 17.81%. The number of students whose grades were overestimated compared to the prediction and who failed the exam was 26 instead of 30. The percentage of students predicted to pass increased from 70.65% to 89.13%.

5.2. Qualitative Analysis of the Failing Class

As shown in Table 4, 26 students (17.80%) failed the exams and the model overestimated their scores. It is essential to explain the reasons for this overestimation here. To do this, all student records for the period in question had to be analyzed in depth, including information that was not initially considered. We carefully accessed each of the 10 students’ archives, retrieving weekly homework scores that were previously been discarded because they would have confirmed the obvious assumption that students who scored well on weekly homework tasks will be well-prepared and pass the final exam. Despite this, the initial matrix of 68 variables was refined by adding the variable representing students’ weekly homework grades. Special attention should be paid to students who were diligent and performed well during the semester but failed the final exam, despite the positive predictions of the model.

In the end, out of the 26 students, 16 showed poor performance during the semester, either by receiving very low grades or by not uploading their work at all. These 16 cases were therefore eliminated for non-diligence. The remaining 10 cases are studied here (6.85% of the population), corresponding to diligent students, i.e., those who received more than 75% of the grade through their weekly assignments.

Because all 146 participants provided their student numbers when they completed the questionnaire, their responses allowed us to diagnose the reasons for their failure. The fact that each participant’s ID number is known individually allows us to validate each student’s response based on actual results and not simulated experiments.

After reviewing the initial data matrix, it became apparent that several important aspects of the learning method were not designed by the specific group of students. Other parameters were revealed, but most of them were related to validating the significance of the 14 variables listed in Table 2. Analysis of the students’ survey responses revealed that the performance of 2 students’ was affected by learning disabilities. It should be noted that of the 146 participants, 22 were formally diagnosed with learning disabilities; 13 of these students passed and 9 failed, and 2 are considered part of the study group.

In particular, Figure 10 shows the model validation and external factors applied to failing students.

In Figure 10, the lowest values, i.e., those close to 1 on the 5-Likert scale and represented by a shorter bar on the histogram, are considered to have negatively affected students’ performance. Three students reported having high computer skills, while three others felt they had average skills. In terms of basic software skills, only two students felt they were highly proficient. On the other hand, all of them reported having high skills in social media applications. Nevertheless, the vast majority did not feel capable of succeeding at similar tasks in the future, and did not feel comfortable taking final exams. Three students were honest enough to admit that they had never studied the theory notes attached to each course. These three students had an average prediction error of −1.50, which is considered the highest error.

All participants in the group studied in this section were supervised by different instructors, implying that the variable “instructor ID” had no impact on their failure.

Table 5. Non-show cases in relation to the size of the class.

Class Size	NON-SHOW	Error	Instructor’s ID	Student’s ID
23	1	−1.39	3	XX392057
23	1	−1.12	3	XX392071
21	3 *	−1.48	1	XX392093
21	3 *	−1.70	1	XX392110
21	3 *	−1.58	1	XX392117
21	5 *	−0.85	2	XX392141
18	5 *	−1.57	4	XX392068
18	5 *	−1.01	4	XX392195
18	5 *	−1.88	2	XX392177
20	6 *	−0.76	4	XX392003

* Non-diligent students not included.

shows that student non-shows (regular attendance, non-shows on exams) are higher in classes supervised by instructor 4. Although larger classes do not have an effect on absences, the “instructor” variable appears to be related. Instructor 1 (author team member and module organizer) is more severe with promising students who do not perform well on the final exam.

After personally interviewing the 10 participants about determining other causes of failure, student XX392068 introduced a new parameter: software version. It seems that different versions of software with alternative workspaces can lead to student confusion, and thus failure.

5.3. Contribution on the Learning Methodology

Finally, regarding more operational aspects of the teaching team, Figure 11 illustrates the steps of the learning process and their contribution to making teaching more sustainable. These steps were initiated by one of the co-authors of this article and coordinator of the 11 class groups. After twenty years of experience in teaching computer-aided mechanical design in higher education, and based on the empirical data collected in this study, the instructor observed over the years that students who have difficulties in conceiving geometric properties of objects and layout of planes (vertical, horizontal, front, back) have the greatest difficulties in drawing cross-sectional views starting from the seventh class. Eventually, some of them dropped out of the module and others failed the final exam. Students’ attitudes and preferences, as expressed by the surveys, are related to the achievement of the learning objectives of the CAD module itself. In particular, Figure 11 shows that from the level of achievement of the learning objectives, students are expected to design the shape of the evaluated object and the meaning of the planes in the top and side views of the studied objects. It is assumed that the students work on Autodesks Inventor Pro 2020, educational version. Since the view representations become more complicated with the introduction of sectional views, the sectional planes were highlighted in blue in order to facilitate the design of the cut-out volumes and projected edges. Finally, the students consider that they can succeed in a similar task in the future, and are able to relate their assignments to future tasks as mechanical engineers, which proves the sustainability of the teaching strategy.

To conclude, by creating innovative and transformative hybrid learning structures and combining physical and virtual learning spaces, the authors have responded to one of UNESCO’s recommendations on open educational resources (OER). Indeed, the development of methods, such as those described here, facilitates the use and adaptation of OER while supporting quality, inclusion, and equitable access in higher education and lifelong learning [12,46].

6. Conclusions

Traditional educational systems have been completely challenged by the COVID-19 pandemic that began in mid-March 2020. University education has had to undergo major transformations from a face-to-face environment to an online learning space. Despite a still unpredictable global health situation, the doors of universities around the world were able to reopen. Since a return to normalcy has not yet been possible, new learning environments, both blended and virtual, have been created by massively integrating information and communication technology platforms into the educational process.

The purpose of this research was to better understand the factors affecting students’ early meta-COVID-19 performance in blended learning environments when sustainable learning modes were imposed due to health crisis restrictions. Specifically, the study was conducted in a first-year university mechanical engineering course. The module selected for this experiment was a “minds on” lab designed to teach computer-aided mechanical design.

A generalized linear autoregressive (GLAR) model for grade prediction—created in 2020–2021 during the early phases of the pandemic, i.e., when online learning was exclusive—was implemented, and we recalled here its key metrics, including a success rate of 67% [32]. This model was used as-is to predict student grades in online learning included in hybrid educational systems. To do this, the existing model was augmented with data from 146 students in the 2021–2022 academic year. The 35 variables in the model were used to predict student grades within a ±1 range. Under these conditions, the pass rate (35.62%) was found to be inadequate.

In order to refine the existing GLAR model further for full utilization of online education included in hybrid educational systems, new factors affecting student academic outcomes were identified. A Spearman statistical analysis was conducted on the 2021–2022 data. Fourteen variables were thus filtered: seven of them were from the 2020–2021 period and the other seven emerged as a result of the hybrid learning environments implemented in 2021–2022.

Paradoxically, whether students had the opportunity to take the courses online or face-to-face, the number of courses each learner attended in both learning modes, distance from home and pandemic conditions were not significantly correlated with the students’ final grade. However, the sense of enjoyment from participating in the module, as well as variables related to 3-Dimensional drawing design, engineering education skills, and students’ ability to relate assigned tasks to their future professional lives as mechanical engineers exhibited the lowest Spearman’s correlation coefficient p values.

The refined version of the GLAR model predicts student grades to within ±1 with a pass rate of 63.70%, making it 28.08% more accurate than the model originally created in 2020–2021. The percentage of students predicting success increased from 70.65% to 89.13%, an improvement of 18.48%. Underestimated scores that were borderline passing the module were limited to 6.85% instead of 17.81%, a reduction of 10.96%. Finally, the $R^2$ coefficient of the GLAR model was 44.32%, its mean absolute error equal to 0.78, and the mean absolute percentage error equal to 18.38%. Special attention was given to students whose grade predictions were underestimated and who failed. Not having studied the theoretical part of the module accentuated their lack of knowledge, increasing their sense of insecurity and discomfort in the final exam.

Synthesizing the above, in order to achieve the goals of sustainable engineering education environments, new parameters must be revealed depending on the mode of learning, i.e., distance or hybrid.

Overall, the advantages of the model are its simplicity and its speed of creation through the use of essential statistical filtering tools (correlation analysis), while maintaining an appropriate level of accuracy. The contributions of this work may therefore be of major interest to all members of the academic community:

• Measuring students’ academic performance through grade prediction modeling can serve as an “alarm” for learners likely to fail final exams, as well as a self-assessment of the level of knowledge acquired.
• As for the variables validated by the GLAR model in the context of online and hybrid learning spaces, they respond to current needs in terms of skills acquisition; they are closely linked to the professional requirements of tomorrow.
• In terms of policy incentives, university administrations could consider the results of this work as a good way to reevaluate the teaching strategies applied in higher engineering education [47] and thus contribute to the sustainability of university systems.

The work presented here is the result of a transnational collaboration between two European universities, the University of West Attica in Greece and the University of Tours in France. As mentioned in the recent European Commission proposal on building bridges for effective cooperation in European higher education [46], the higher education sector can benefit from such cooperation while promoting the quality and diversity of academic learning and teaching through digital transformation and ultimately build a sustainable future for European universities.

Although the simplicity and ease of use of the model is undeniable, there were practical limitations in applying the methodology described in this article. Indeed, because the data processing took time (collecting survey responses, identifying and excluding invalid responses, as well as those of students in higher semesters), only 10 days were left to alert students who were predicted to fail. In future experiments, the survey will be launched a week earlier so that the instructor team has more time to generate new results and alert potentially failing students. Moreover, there were other limitations due to the health situation itself. Although all participants (i.e., students and instructors) had prior experience with online teaching spaces, the application of hybrid environments quickly gained momentum, with no real consensus on the methods and tools to be used.

The methodology defined and deployed in this work is therefore related to the social parameters and restrictions that were applied throughout the research period. It is important to note that this study is directly related to society [48]: the participants were real students, all “teaching experiments” were conducted in real learning environments, based on trust between learners and their instructors. All students who participated declared their consent to take part in the research. Student identities were used to qualify the results for statistical analysis. In this paper, all student identification records and personal data were properly anonymized to avoid any potential misconduct between science and society.

Currently, online learning environments are no longer allowed in higher education in Greece. Hybrid learning spaces were initially created in response to the increase in pandemic cases among students, but have been welcomed by the majority of participants, as they offer a high level of freedom in terms of physical presence. The question is, will future first-year engineering students who take mixed-mode modules, without having previously experienced online modes, be as proficient as the participants in this study? Or can we hope that future generations, unaffected by the pandemic, will be more competent than the current population? These questions can be answered in future studies that validate or adjust the variables involved.

Some research perspectives can be drawn from this work. First, the variables involved in hybrid learning environments should be further examined, with the aim of adding more variables to improve the robustness of the modeling and thus further strengthen the conclusions presented above. The proposed model can also be optimized, especially by inserting the fitting errors as an additional predictor in an artificial neural network [49,50,51]. This could allow the creation of a hybrid model to improve the coefficient of determination. This research work can therefore be extended by testing other prediction models, such as deep learning [52], hybrid neural networks, decision trees, support vector machines. Finally, it should be noted that the use of holograms and artificial intelligence will soon be possible via 5G and the future 6G. Faced with the big data—including sounds, words, images, and video—that will have to be taken into account, it will be essential to use efficient methods of data compression, at the level of the acquisition itself, of the compressed sensing type [53,54].