Comprehensive Assessment of Distance Learning Modules by Fuzzy AHP-TOPSIS Method

Abstract

This survey is focussed on distance learning studies, where there can be met a lot of technical obstacles, which creates complications in decision making. To get an ideal solution for these kinds of problems, the Fuzzy TOPSIS (Technique for Order Preference by Similarities to Ideal Solution) is one of the best solutions. Therefore, this paper presents the distance learning quality assessment surveys when the Fuzzy AHP (Analytic Hierarchy Process) and TOPSIS methods are used. Research results describe the application of the Fuzzy AHP—TOPSIS hybrid method. MCDM (Multi-Criteria Decision Making) programs with MATLAB (R2020b) mathematical package were written to calculate the evaluation results for three distance learning courses. In the practical implementation of the proposed distance learning module evaluation methodology, the experts’ evaluation method was applied. Thirty-four judges were chosen with specific knowledge and skills and with very different competencies to assess three alternatives by fourteen criteria. Following the experts’ evaluation, a statistical analysis method was used to process the data. After applying the complex evaluation, the comparative analysis method was used to summarize the obtained results. This work further provides useful guidelines for the development of an easily understandable hierarchy of criteria model that reflects the main goal of study quality assessment.

1. Introduction

Due to the COVID-19 pandemic, quarantine has forced institutions to reorient and reorganize the entire education system in a short period of time. The biggest part of the Military Academy of Lithuania (MAL)’s education process (subject, social, even physical) was transferred from the direct from into the form of distance education. This sudden change caused by unforeseen conditions affected the interactions, involvement, and roles of the participants in the educational process, both teachers and cadets, in the educational process, the educational environment, the content, and the nature. This situation forced the MAL to remember and apply the already well-known experience of distance learning and to look for criteria for improving the quality of distance learning courses.

As is well known, distance learning is generally perceived as a way of learning where the learner’s geographical location differs from that of the geographical location of the education institution [1]. The development of distance learning has witnessed gradual progress. Typically, new technologies for knowledge transfer have penetrated different distance learning systems and their development has promoted the emergence of modern distance learning courses. Many researchers classify the history of distance learning modules (DLMs) in terms of so-called generations of distance learning [2,3].

First-generation distance learning courses mainly refer to handwritten or printed handouts that were sent to learners via postal mail. This form of DLM has been expanding in all developed countries since the mid-19th century. With the advent of radio in the 1920s and television in the 1950s, new information and communication technologies came into use, including DLMs in the form of developing and broadcasting radio and television training programmes. The year when the British Open University was established, i.e., 1969, is considered to be the beginning of the second generation of the DLM. It was then that multimedia entered into service. The British Open University was famous for producing high-quality learning materials specially designed for distance learning. The third generation is characterised by the use of interactive, electronic and other computer-related technologies. Computer systems provide a two-way communication channel, both synchronous (simultaneous communication via video/audio conferencing) or asynchronous (communication at different times using e-mail, discussion forums). Despite the fact that computerised distance learning was implemented in several educational institutions in 1980, it gained popularity only with the advent of the World Wide Web. The spread of the Internet contributed to the popularity of DLM due to the ability to communicate quickly and flexibly and to present learning materials interactively using Web technologies.

The Dictionary of Contemporary Lithuanian defines “a course” as “a one-year study unit in a higher education institution or special secondary school or the delivery of a subject in a higher education institution” [4]. The form of distance learning is applied in both higher education and non-formal adult education. This paper analyses distance courses delivered in higher education institutions. Thus, a distance course can be defined as a course unit delivered remotely using information technologies. According to [2], “studies” mean personal learning, when someone who has finished at least his education in the secondary school continues his studies in a higher-level education institution in a particular study programme.

Analysis of e-learning and distance learning has a prominent place in Lithuanian research. Many Lithuanian researchers have devoted a considerable amount of work to the area of learning community and second-generation web (Web 2.0) technologies [1,5,6]. Subsequently, analysis of Web 3.0-based individualisation of learning objects in simulated learning–teaching environments has been carried out [7]. Lithuanian researchers have proposed and used in their works quality control methods for training software [1] and quality evaluation for learning scenarios [7]. The issue of software localisation has also been addressed in a number of research studies [5]. In addition, digital learning objects have been analysed, extending their metadata model [8]. The use of reusable learning objects has been explored, seeking methods to separate e-learning materials from display rules, thus improving the use of reusable learning objects [9]. A number of researchers have published their findings on the importance of virtual learning environments in e-learning [10,11,12,13].

Advantages and disadvantages of e-learning and blended learning related to the use of the virtual learning environment Moodle for blended learning have been analysed by Stonkienė [8]. A number of research studies have also been carried out to explore the importance of e-learning and visualisation of user actions as well as learners’ activity and analysis of their preferences [14,15]. Researchers have further explored the architecture of the virtual learning environment to increase options for individualisation and investigated actions taken by software agents in a virtual learning architecture based on Q-learning algorithms, with the virtual learning environment playing the curating role. A number of studies have been carried out on the integration of smart modules into the capabilities of the virtual learning environment Moodle and the implementation of personalised learning depending on certain learning styles [16,17]. In addition, many research studies have been conducted to explore the choice of learning scenarios and to analyse the components of e-services in virtual learning environments [18]. Researchers have developed and integrated into the Moodle platform a model of grades transfer from one scale to another [19].

The aim of this research is to identify the cause-and-effect relationship between the quality of the content of a distance learning module and its technical feasibility using the expert group’s aggregated decision. Group decision making (GDM) is more multifaceted compared with individual decision making, since it includes a number of challenging aspects such as incompatible personal purposes, ineffective learning, validity of communication, individual incentive, and private opinions. The GDM problem lies in combining the opinions of multiple experts into a decision of a certain single group. In this research, each group’s decisions were pooled in the group’s aggregated decision. Given that expert evaluation was independent, each expert produced his/her individual set of evaluation criteria selected for the research independently of other experts. These individual judgements were subsequently combined and eventually presented as a set of evaluation parameters of a separate expert group. On the basis of this set, a multi-attribute decision-making approach was applied to express the opinion of and decision made by a certain group [20]. In order to facilitate the systematisation of the methodology used for distance learning quality evaluation, the research employed a scheme presented in Figure 1. The research scheme illustrates in detail the structure of the research process, which was followed to analyse the quality of three distance learning modules and to identify key criteria.

The conducted study’s purpose and results are represented in seven sections. The relevance of the problem and a short overview of the history of distance learning according to our researches investigations is presented in the introduction. The representation of the problem of distance study quality assessment in detail is discussed by a literature review in Section 2. Section 3 describes the methodology that was chosen to solve the multi-criteria problem for decision making by Fuzzy Technique used to Order Preference by Similarities to Ideal Solution (FTOPSIS) procedure when the Triangular Fuzzy Numbers are applied to give proper weightage to criteria by the Fuzzy Analytic Hierarchy Process (FAHP). The problem of qualitative data in the TOPSIS model is discussed in Section 4. The numerical examples showing how the applications of the use of fuzzy set theory with the MCDM technique help decision-making to deal with indistinctness and ambiguity are explained in Section 5. Lastly, the discussions and some concluding explanations are made in the last two sections, Section 6 and Section 7.

2. Materials and Methods

The quality of e-learning can be linked to all learning and educational processes, outcomes, and services provided through information and communication systems [18,19,20,21]. The quality of distance learning can be improved during subject design through the purposive selection of IT tools for the implementation of the distance learning process and during subject delivery through efforts to increase the efficiency of the virtual learning environment. A number of significant research studies have been conducted in the area of quality assurance in distance learning, focusing on various aspects of such education. Many researchers have analysed the impact of technological change on the quality of distance courses, as the main focus in developing a distance course module and providing training is on the development of learning material. This is an important moment in the whole learning process [22,23]. Therefore, researchers have analysed in detail and presented a specific model for designing distance learning curricula based on quality assessment factors of distance learning content [24]. One of the factors for assessing the quality of distance learning curricula design identified by the authors is the development of a student assessment strategy. In addition, peculiarities of quality assessment of distance learning content were analysed with the use of expert interview data [25].

With regard to the regularity analysis of planning and organising distance (further vocational) learning content, researchers have emphasised that adequate training of teachers/lecturers, supervisors, administrators, managers, and consultants at pre-learning, learning and post-learning stages are factors that play an important role. In this context, research suggests that achievement of learning content quality requires a regular analysis of learners’ preferences [26], setting learning objectives, application of appropriate learning organisation methods, proper planning of the assessment of learning outcomes, and selection of appropriate curriculum-specific technological means [27].

To assess the quality of distance learning, researchers also used Quality Function Deployment (QFD), consisting of interactions between customer preferences and quality criteria of alternatives [28]. The Quality Function Deployment was proposed by a Japanese scholar in 1996 [29]. Another method explored and proposed by researchers is a method for selecting and evaluating e-learning products, which involves evaluation and ranking of learning products and alternatives with respect to target product characteristic values [30] in evaluating the quality of services in e-learning [31].

Evaluation of teaching employs various models. Some researchers applied a neural network model based on the optimisation approach [32], while others conducted research based on the Fuzzy Analytic Hierarchy Process (FAHP) [33] to evaluate teaching quality in the classroom. In evaluations of distance learning quality, researchers emphasised the learner’s perspective as the most important factor, since the quality of the delivered module directly depends on the learner’s expectations [34]. This approach generated several tools for Student’s Evaluation of Teaching Effectiveness (SETE), including Student Instructional Rating System (SIRS), Instructor and Course Evaluation System (ICES), Student Description of Teaching (SDT), Students’ Evaluations of Educational Quality (SEEQ), and Instructional Development and Effectiveness Assessment (IDEA) [35].

The literature review shows that effective teaching using web-based remote systems depends on many factors, also known as criteria [36,37,38,39,40]. Therefore, it is not a simple task to develop an easily perceived hierarchy of criteria—a model that reflects the main purpose of the task. After research, certain criteria selection and grouping methods and model-making principles are proposed that could facilitate modelling. Drawing on the research [41] analysing the principles of the structure of models (criteria) for evaluating the quality of multi-criteria decision-making analysis, evaluation criteria can be selected taking into account the following important aspects:

• Linking concepts with their goals (value relevance): decision-makers should be able to link the key concepts of expert evaluation in the appropriate area with the purpose of the evaluation.
• Equal understandability: all decision-makers should have a shared understanding of the evaluation criteria.
• Measurability: the criteria need to be realistically measurable in practice; they should be explained by sub-criteria and quality compliance criterion levels.
• Non-redundancy: the same aspect (factor) should not be measured by several different criteria.
• Judgement independence: decision-makers should evaluate alternatives separately with respect to each criterion but keep in mind that there are links between the criteria; i.e., criteria form a framework.
• Balancing completeness and conciseness: there should be neither too many nor too few criteria, and they should not be too large or too detailed; i.e., they should capture all the main aspects of the alternative without going into too much detail. In other words, the model should not force decision-makers to evaluate the quality of alternatives blindly according to formal sub-criteria. The decision-makers should understand the requirements of the criterion as a whole and judge accordingly.
• Operationality: the model should be applicable in practice without taking an unreasonable amount of time.
• Balancing simplicity and complexity: notwithstanding the complexity of the problem of quality assessment, the modeller should provide decision-makers with a simple and clear criteria tree [41].
• The Consistent Fuzzy Preference Relations (CFPR) method could be used for ranking and selecting the most important criteria for study quality assessment model (i.e., establishing consistent fuzzy preference relations). The CFPR method evaluates the effectiveness of the criteria for effective distance learning and reduces the number of questions in the questionnaire and the number of similar criteria and prevents inconsistencies [42].

Drawing on the researchers’ insights emphasising that all researchers focus on quality assessment methods while not giving due attention to the quality assessment model, the validity and reliability of research findings should be therefore sought with a focus on constructing the Multi-Criteria Decision Making (MCDM) model itself. For this reason, researchers began to construct integrated models using the aspect investigation and Decision Making Trial and Evaluation Laboratory (DEMATEL) method. The DEMATEL technique has received reasonable interest, as it allows identifying the relationships between the criteria and determining the main criterion, which is the principal one to represent the effectiveness of the group of criteria [43,44].

To justify the choice of the MCDA method, we would like to turn the reader’s attention to, e.g., the effectiveness of methods for determining the significance of standards in supportable transport difficulties; see e.g., [44,45].

Highlighting criteria sensitivity analysis, Thomas L. Saaty for different criteria evaluation proposed the Fuzzy Analytical Hierarchy Process (AHP) [46], which is one of the more popular MCDM methods, because a set of factors can be shaped and logically assessed taking into account their importance. Alternatives can be evaluated based on these factors and obtain a final mark that replicates this importance rank.

There is no agreement on how to regulate the sensitivity investigations, i.e., the precision of a decision technique and the consistency of the outcomes. The sensitivity investigation can be defined as stability or performance of the solution to minor variations in preferences, which happen throughout the determination process or to minor variations in the values taken for measures; this is what some research studies call effectiveness multicriteria decision analysis, e.g., Pamučar and Ćirović [45]. Notwithstanding a large number of established and new methods, the challenge of multicriteria choice is still not trivial. Subsequent to the outcomes achieved, the assessment of alternatives according to the criteria and the choice of the criterion for ranking alternatives using dissimilar ranks have a reflective effect on the concluding choice.

The Fuzzy Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) is the other analysis that is similarly widespread. In TOPSIS analysis, criteria of the alternatives are associated with that of an ideal. TOPSIS replicates a normal human thought process where people judge things not by certain criteria but by an ideal example of the same type [47,48,49].

Because the quality of the remote module depends on many factors such as clearly presented and interestingly arranged material, well-organized teaching process, appropriate IT tools used, relevance of module material, student motivation and teacher/lecturer qualification and professionalism, all of these criteria were included in the study and all chosen criteria were evaluated by experts in the relevant field. Moreover, the Fuzzy Analytic Hierarchy Process (FAHP) and Fuzzy Technique for Order of Preference by Similarity to Ideal Solution (FTOPSIS) is used to identify the best distance learning module (DLM). The AHP was chosen to identify the crispified value of factors [46]. Inaccurate linguistic terms used by the experts were transformed to Triangular Fuzzy Numbers. These investigations’ idea was to rank the distance learning modules using fuzzy numbers for criteria weights and by using FAHP to FTOPSIS ranking.

It is common knowledge that the AHP method has a problem with the rank reversal phenomenon. To prevent, it the authors used the methodology presented by Wang and Elhag in [39]. They explained the origin of rank problem phenomenon and how that rank exchange is caused by variation of local significances before and after an alternative is added or deleted. It is recommended to consider the local priorities unchanged to obviate the rank exchange phenomenon.

A wide range of multicriteria decision methods have been proposed by researchers, but all these methods primarily use the same evaluation table, and only then their analysis background can vary according to the additional information they are looking for. The PROMETHEE analysis methods necessitate very clear supplementary data that are found simply and understood by both decision-makers and consultants.

In the PROMETHEE computer software PROMCALC and DECISION LAB, the analyzer can easily express the relative importance of the criteria, because it allows one to present agreed-upon values for the criteria weights. These values are then divided by their quantity so that the weights are normed routinely. Measuring weights to the criteria is not simple. This process involves the priorities and observations of the decision-maker. PROMCALC and DECISION LAB take in numerous sensitivity tools, which can be involved in a different set of weights in order to help to fix them. Additional information can be found in Goumas and Lygerou’s paper [40].

When solving the optimization problem while evaluating the distance learning courses, the values of the criteria weights are determined by experts and are not changed. Therefore, it became necessary to use multi-criteria methods such as SAW, TOPSIS, COPRAS, MOORA, and PROMETHEE, which use constant expert estimates in the calculations.

The use of this priority function does not highlight the advantages of one alternative, as the values from q to s increase linearly. PROMETHEE, like supplementary multi-criteria analysis, supports the principles of the SAW method, uses the values of specially selected priority functions instead of the normalized values of the criteria [20,33,50,51].

The study methodology and methods are presented in detail in Section 3.

3. Research Methodology

It is a characteristic of many tasks that there are insufficient numerical data to solve them or that the research object is impossible to measure. In such cases, experts are relied upon to complement data with their judgements. Valid decisions must be based on experience, knowledge, and intuition, since the degree of quality should be determined on the basis of expert judgements by professionals qualified in the field at issue. Such people are required to have specific knowledge and skills in particular fields. In the process of research, the problem of selecting experts is one of the most difficult, both theoretically and practically, because persons evaluating one or another factor may differ in their competences and values. Therefore, for the purpose of this research, experts were selected on the basis of their characteristics related to their professional competence: work experience, length of service, academic degree and research activity, and the ability to solve specific problems in the respective field [21]. Interviews with selected experts employed different methods of interviewing: implicit expert method in terms of interrelationships between experts, one-off interviews in terms of judgement synchronisation procedure, and one-to-one interviews in terms of the number of experts. Research-based principles were applied to define groups of criteria for the quality of distance learning courses [41].

3.1. The Construction of a Decision-Making Matrix

Responding to the individual or group decision problematic with a multi-criteria background, the analysis starts with the creation of a decision-making matrix or a few matrices. These matrices represent the values of the criteria for assessment alternatives chosen. Criteria values, according to researchers’ choice, can be real, interval numbers, fuzzy numbers, or qualitative labels. Accordingly, criteria values can designate a decision-maker’s expert judgement set of values D = {1, 2,…, K}. The multi-criteria values can be transformed into a k matrix:

	C1	C2	$\dots$	Cn
A1	$x_{11}^k$	$x_{12}^k$	$\dots$	$x_{1n}^k$
A2	$x_{21}^k$	$x_{22}^k$	$\dots$	$x_{2n}^k$
$\dots$	$\dots$	$\dots$	$\dots$	$\dots$
Am	$x_{m1}^k$	$x_{m2}^k$	$\dots$	$x_{mn}^k$

where:

-

$A1,A2,\dots Am$ are feasible alternatives that decision-makers have to select from,

-

$C1,C2,\dots Cm$ are the criteria for which the alternative feature is judged,

-

$x_{ij}^k$ is the decision-maker $k$’s score of alternatives $A_i$ with connection to the criterion $C_j$ ($x_{ij}^k$ is a numerical, interval data, or fuzzy number). According to this rule for $m$ alternatives and $n$ criteria, the matrix $X^k=\left(x_{ij}^k\right)$ can be drawn, where $x_{ij}^k$ is the value of alternative $i$ taking into account the criterion $j$ for the decision-maker k, $j=1,2,\dots ,n,k=1,2,\dots ,K$.

The comparative rank of each criterion can be calculated by a set of weights that are normalized to sum to one. The weight vector can be represented by $W^k=[W_1^k,W_2^k,\dots ,W_n^k]$ for decision-maker k, where $W_n^k\in \Re$ is the decision-maker k’s weight of criterion $C_j$ and $W_1^k,W_2^k+\cdots +W_n^k=1$.

Finally, it can be pointed out that multi-criteria methods principally emphasise three types of decision difficulties: to make a choice when the most appropriate (best) alternative must be selected, to make a ranking to explain the order of the alternatives from the best to the worst, and to make the categorization when the best k alternatives must be from a list.

3.2. The Construction of Fuzzy AHP

Researchers advise a wide range of different methods for criteria weights assessment [52], but the most used is the AHP (Analytic Hierarchy Process) developed by Saaty [46]. The fuzzy theory Buckley [53] combined with AHP creates the fuzzy AHP (FAHP). The method for finding fuzzy weights, based on a direct fuzzification, was designed by Saaty and is described in his work [46]. The fuzzification procedure of FAHP [46] can be conducted by few steps.

Step 1: To present fuzziness, the pair-wise numeric has to be changed into a matrix using Triangular Fuzzy Number (TFN).

To determine the importance of each criteria preference, natural language or numeric values can be used. In this case, each of the two elements first have to be compared using a scale of importance. For these investigations, first, a linguistic scale from one to nine was determined (see

Table 1. The TFN to make pair-wise comparison matrix for the main criteria.

FN 1	Linguistic Variable	MF 1
1	Equally important (EI)	(1,1,3)
2	Equally moderate important (EMI)	(1,2,4)
3	Weakly important (WI)	(1,3,5)
4	Moderate important (MI)	(2,4,6)
5	Moderately strong important (MSI)	(3,5,7)
6	Strongly important (SI)	(4,6,8)
7	Very strongly important (VSI)	(5,7,9)
8	Very strongly extreme important (VSEI)	(6,8,9)
9	Absolutely important (AI)	(7,9,9)

¹ Notes: FN = fuzzy number; MF = membership function.

), and the TFN was defined by the membership function presented by Chen [47].

Step 2: Next, fuzzy pair-wise comparison matrices have to be designed.

According to analysis methodology, the decision-makers individually make the pair-wise comparison among all criteria and give the linguistic term represented by a triangular fuzzy number.

Mathematically, this can be represented as $\tilde{P}=\left[{\tilde{a}}_{ij}\right]$ being an n × n matrix, where ${\tilde{a}}_{ij}$ is the rank of criterion $C_i$ in account to criterion $C_j$, according to the fuzzy preference scale shown in the table:

$$\tilde{P}=\left[\begin{array}{ccc}\left(1,1,1\right)& \cdots & {\tilde{a}}_{1n}\\ ⋮& \ddots & ⋮\\ {\tilde{a}}_{n1}& \cdots & \left(1,1,1\right)\end{array}\right] =\left[\begin{array}{ccc}\left(1,1,1\right)& \cdots & {\tilde{a}}_{1n}\\ ⋮& \ddots & ⋮\\ \left(1,1,1\right)/{\tilde{a}}_{1n}& \cdots & \left(1,1,1\right)\end{array}\right].$$

(1)

Step 3: This step is about how to compute the normalization of the fuzzy weights.

The $C_i$ criterion fuzzy weight represented as ${\tilde{w}}_i$ can be found by the equation:

$${\tilde{w}}_i={\tilde{r}}_i\times {({\tilde{r}}_1+{\tilde{r}}_2+\cdots +{\tilde{r}}_n)}^{-1},$$

(2)

where ${\tilde{r}}_i={\left[{\tilde{a}}_{i1}\times {\tilde{a}}_{i2}\times \cdots \times {\tilde{a}}_{in}\right]}^{1/n}$.

3.3. The Fuzzy TOPSIS Method

As was mentioned above, for this study, the Fuzzy TOPSIS was chosen. Overall, the procedure for the TOPSIS can be started by creating the decision matrix illustrating the determined cost of each criterion according to each alternative. Subsequently, the direct matrix is normalized by a chosen method, and the criteria values are multiplied by the identified criteria weights. Next, the positive-ideal and negative-ideal outcomes are calculated, and the distance of each alternative to these outcomes is calculated with a distance value. Lastly, all alternatives are ordered based on their comparative closeness to the ideal measure.

In this section, the detailed TOPSIS procedure for group decision-making based on Shih, Shyur, and Lee’s suggestion [49] is presented as a procedure.

Step 1: Create a decision matrix.

To assess the distance learning courses, fourteen criteria and three alternatives (distance learning courses) were chosen that were ranked based on the Fuzzy TOPSIS method. The table below presents the type of criterion and weight assigned to each criterion as fuzzy numbers, linguistic terms and triangular scale (see

Table 2. Fuzzy scale description.

FN 1	Linguistic Terms	Triangular Scale
		L 1	M 1	U 1
1	Very low (VL)	1	1	3
3	Low (L)	1	3	5
5	Medium (M)	3	5	7
7	High (H)	5	7	9
9	Very high (VH)	7	9	9

¹ Notes: FN = fuzzy number; L = the lower limit; M = the most likely value; U = the upper limit.

).

Step 2: Create the normalized decision matrix.

Based on the positive and negative ideal solutions, a normalized decision matrix can be calculated by the following relation:

$${\tilde{r}}_{ij}=\left(\frac{a_{ij}}{c_j^{\ast }},\frac{b_{ij}}{c_j^{\ast }},\frac{c_{ij}}{c_j^{\ast }}\right);\hspace{1em}{c}_j^{\ast }={\max }_i c_{ij}; \mathrm{Positive} \mathrm{ideal} \mathrm{solution}$$

(3)

$${\tilde{r}}_{ij}=\left(\frac{a_j^{-}}{c_{ij}},\frac{a_j^{-}}{b_{ij}},\frac{a_j^{-}}{a_{ij}}\right);\hspace{1em}{a}_j^{-}={\min }_i a_{ij}; \mathrm{Negative} \mathrm{ideal} \mathrm{solution}$$

(4)

The normalized decision matrixes were calculated for future step 3 analysis.

Step 3: Design the weighted normalized decision matrix.

Considering the dissimilar weights of separate criteria, the weighted normalized decision matrix can be calculated by multiplying the weight of individual criteria in the normalized fuzzy decision matrix, according to the following formula:

$${\tilde{v}}_{ij}={\tilde{r}}_{ij}.{\tilde{w}}_{ij}$$

(5)

where ${\tilde{w}}_{ij}$ represents the weight of criterion $c_j$ The weighted normalized decision matrixes are presented in the results section below (see in Section 5).

Step 4: Determine the fuzzy positive ideal solution (FPIS, A*) and the fuzzy nega-tive ideal solution ($\mathrm{FNIS},A^{-}$).

As mathematical formulas, the FPIS and FNIS of the alternatives can be defined by equations:

$$A^{\ast }=\left\{{\tilde{v}}_1^{\ast },{\tilde{v}}_2^{\ast },\dots ,{\tilde{v}}_n^{\ast }\right\}=\left\{\left(\underset{j}{\max } v_{ij}|i\in B\right),\left(\underset{j}{\min } v_{ij}|i\in C\right)\right\},\hspace{1em}\left(\mathrm{benefit} \mathrm{criteria}\right)$$

(6)

$$A^{-}=\left\{{\tilde{v}}_1^{-},{\tilde{v}}_2^{-},\dots ,{\tilde{v}}_n^{-}\right\}=\left\{\left(\underset{j}{\min } v_{ij}|i\in B\right),\left(\underset{j}{\max } v_{ij}|i\in C\right)\right\},\hspace{1em}\left(\mathrm{cost} \mathrm{criteria}\right)$$

(7)

where ${\tilde{v}}_i^{\ast }$ represents the max measure of $i$ of all assessed alternatives and ${\tilde{v}}_1^{-}$ is the min measure of $i$ for all the alternatives. B and C explain the positive and negative ideal explanations, respectively. The positive and negative ideal solutions are presented below in the results Section 5.

Step 5: Calculate the distance between each alternative and the fuzzy positive ideal outcome $A^{\ast }$ and the distance between each alternative and the fuzzy negative ideal outcome $A^{-}$.

The gap between each alternative and FPIS and the gap between each alternative and FNIS are calculated, respectively, as follows:

$$S_i^{\ast }={{\displaystyle \sum }}_{j=1}^{n}d({\tilde{v}}_{ij},{\tilde{v}}_j^{\ast })\hspace{1em}i=1,2,\dots ,\mathrm{m};$$

(8)

$$S_i^{-}={{\displaystyle \sum }}_{j=1}^{n}d({\tilde{v}}_{ij},{\tilde{v}}_j^{-})\hspace{1em}i=1,2,\dots ,\mathrm{m};$$

(9)

where d is the gap between two fuzzy numbers, when given two triangular fuzzy numbers $a_1,b_1,c_1$ and $a_2,b_2,c_2$ and a measure between the two criteria can be calculated by the equation:

$$d_v\left({\tilde{M}}_1,{\tilde{M}}_2\right)=\sqrt{\frac{1}{3}\left[{\left(a_1-a_2\right)}^2+{\left(b_1-b_2\right)}^2+{\left(c_1-c_2\right)}^2\right]}.$$

(10)

It can be noted that $d\left({\tilde{v}}_{ij},{\tilde{v}}_j^{\ast }\right)$ and $d\left({\tilde{v}}_{ij},{\tilde{v}}_j^{-}\right)$ are crisp numbers.

The calculated distances from positive and negative ideal solutions are presented below in results Section 5.

Step 6: Identify the closeness coefficient and rank the alternatives.

The closeness coefficient of each alternative can be identified by the equation:

$$C{C}_i=\frac{S_i^{-}}{S_i^{+}+S_i^{-}}$$

(11)

The higher evaluated alternative is always closest to the FPIS, and the lower-ranked to the FNIS. The closeness coefficient of individual alternatives and the ranking are presented below in the results Section 5.

4. Practical Case Study

4.1. The Subject of a Case Study

This study used three distance learning courses: Research Methodology and Statistical Analysis (RMSA), Social Statistics (SOST), and Political Science Research Methodology (PSRM). Each module begins with the most important information about retaking exams and student surveys. The first section of the module is designed to provide information about the course. It also contains introductory lectures, a list of topics to be studied throughout the whole module, and a brief description of the module. In addition, there is detailed information provided by teachers on the passing procedure, including preliminary deadlines and assessment. Each course differs slightly in the arrangement of the course material, but the presentation of the course material of the module generally follows a unified style and there is a clear general structure for each module.

4.1.1. Distance Learning Module: Social Statistics

The Social Statistics (SOST) module is available for the first study cycle in Management of Defence Technologies at the Military Academy of Lithuania (MAL). The scope of the distance module is 5.0 credits (5.00 ECTS credits). The module is available in the Lithuanian language. Teaching methods include lectures, practicals (26 h per semester), consultations (2 h per semester), exam (2 h per semester), and self-learning (95 h per semester). The aim of the module is to provide basic knowledge about social statistics methods, formulate research interests, and develop quantitative research method application skills.

The course in social analysis is designed to develop research skills necessary for conducting applied research and obtaining reliable results. The students are introduced to empirical research as one of the facets of knowledge in objective reality. The course encompasses the analysis of methods of quantitative research, generic problem-based research aspects, research purpose, objectives and tasks, and preconditions for their implementation, as well as procedural and methodological principles of the research process. The subject programme pays special attention to the development of practical skills. After completing the course and programme requirements, the students should have knowledge of the methodological provisions of applied research and be able to choose appropriate research methods, conduct research, interpret data, formulate conclusions, and draw up and present research reports. The students will also be able to apply the SPSS software package to perform data analysis, interpret the findings of the performed analysis, and draw conclusions based on statistical analysis. The study module is developed in the MAL’s virtual learning environment Moodle and is available at https://lka2019.vma.liedm.lt/course/view.php?id=837 (accessed on 6 January 2021).

4.1.2. Distance Learning Module: Research Methodology and Statistical Analysis

The Research Methodology and Statistical Analysis (RMSA) module is delivered to the first study cycle for students in Management of Defence Technologies at the MAL. The scope of the distance module is 5.0 credits (5.00 ECTS credits). The module is in the Lithuanian language. Teaching methods include lectures and practicals (45 h per semester), consultations (2 h per semester), exam (2 h per semester), and self-learning (89 h per semester). The aim of the module is to provide the students with knowledge of the methods, methodology, and techniques used in research conducted in the area of social sciences and develop skills necessary to conduct research independently and process data using the statistical analysis software (SPSS). The acquired knowledge is necessary to develop the ability of cadets to conduct applied research requiring statistics to process collected information, draw conclusions, and formulate new hypotheses based on the results of the research. Therefore, the subject programme is based on the explanation of the methods of modern statistics as a science of information collection, systematisation, analysis, and interpretation, emphasising their practical application in empirical research. The course covers the following topics: Research based on defence and technology management; working with statistical packages; primary data and descriptive statistics with SPSS; research statistics; linear Gaussian models and the use of SPSS for research; questionnaire reliability assessment; factor analysis; case study; selection of statistical methods for research; research methodology and statistical analysis. The study module is developed in the MAL’s virtual learning environment Moodle and is available at https://lka.vma.liedm.lt/course/view.php?id=98 (accessed on 17 January 2021).

4.1.3. Distance Learning Module: Political Science Research Methodology

The Political Science Research Methodology (PSRM) module is available for the first study cycle in Management of Defence technologies at the MAL. The scope of the distance module is 10.0 credits (10.00 ECTS credits). The module is developed in the Lithuanian language. Teaching methods include lectures, practicals (64 h per semester), consultations (4 h per semester), exam (2 h per semester), and self-learning (180 h per semester). The aim of the module is to introduce to the students the principles of academic research in the field of social sciences, with methods of academic evidence-based research, with academic sources of information and academic databases, and with the principles of their interpretation, to acquaint students with the standards of presentation of the academic research results and language culture in research papers.

The course introduces research approaches, assumptions, and principles; application of quantitative and qualitative research in the analysis of domestic and foreign policy and international relations; and public surveys, expert interviews, text analysis, and other research methods. The course builds knowledge of the interrelationship between the research methods applied and the research results obtained. Students learn how to use social statistics (SPSS software package), develop the ability to perform graphical analysis of quantitative and qualitative data, learn to properly present (structure) and critically analyse academic texts, get to know the basics of academic writing (writing reports, essays, bachelor’s theses, scientific articles), and develop academic language skills. The study module is developed in the MAL’s virtual learning environment Moodle and is available at https://lka2019.vma.liedm.lt/course/view.php?id=412 (accessed on 23 December 2020).

4.2. Applications of the Proposed Method

The development of a distance learning module is a long process involving professionals in several fields. Distance teaching has its own specifics features: it is not limited to the preparation of the teaching/learning material, but also involves uploading of the module into the virtual learning environment and organisation of the whole study process. The development of a distance learning course has two main stages: the development of teaching/learning materials and the use of information techniques to deliver the course remotely. The development takes place in successive stages. Each stage ends with expert evaluation. If the evaluation is negative, the course has to be improved.

The basis of any training material is its content. The content of the material prepared by the teaching staff in higher education institutions should be in line with the aim and objectives of the study programme to be delivered. Study programmes are carefully designed taking into account market demand, scientific innovation, and other factors. The quality of study programmes is verified by experts of the Centre for Quality Assessment in Higher Education. Suitable programmes are accredited; i.e., a higher education institution is authorised to conduct studies in accordance with the developed programme. Therefore, the first stage in the assessment of a distance learning course is its content analysis.

4.2.1. Study Data Collection and Analysis

There are three dimensions and sixteen criteria to determine the quality of distance learning courses as were measured by other researches [9]. The use of these dimensions and fourteen criteria has been accepted and applied for practical application in real life. The dimensions are the assessment of the content of the distance learning course (A), evaluation of the use of information tools (B), and opinion on the quality of the courses (C).

The criteria for the assessment of the distance learning course content (A) include distance learning course structure (A1), compliance of the material chosen for the implementation of the distance learning module with the study programme (A2), novelty of the material chosen for the course (A3), selection of tools for testing module participants’ knowledge (A4), and clarity of the material provided by the distance learning module (A5). For the evaluation of the use of information tools (B), criteria include knowledge testing and assessment scoring tools (B1), material readability and accessibility (B2), learning process personalisation capacity (B3), support available to students (B4), and synchronous and asynchronous tools provided for communication (B5). For the opinion on the quality of the course (C), criteria include teacher’s professionalism (C1), support for students in the process of training organisation (detailed explanation of how to participate in online lectures) (C2), organisation of self-learning for students of the distance learning module, including possibilities for feedback (C3), and practical benefits of the course (assessment of acquired knowledge, practical skills and competences (C4).

The research was carried out at the Military Academy of Lithuania in 2020. For the purpose of the research, a group of 34 experts was interviewed, of which three experts were from the Administration Section, ten were teachers with expertise in the development of distance learning modules, three were IT experts from the Study Support Section, three were experts from the Study Planning Section, and fifteen were students who had completed one of the modules at issue. Each of the experts made an independent decision on the quality assessment of the distance learning modules, and in this way the quality of the courses was evaluated by independent experts. The fuzzy AHP method was used to identify the significance of the assessment criteria.

In the context of information available for assessment, expert (teachers’) judgements were expressed in linguistic terms and numbers (grades). IT professionals rated the course uploaded to the virtual learning environment on a five-point scale from Very Low to Very High. In part, they assessed first-stage factors such as course structure and aims. Unlike the first two stages of the assessment, the third stage involved cadets’ opinion on the quality of the course expressed using a linguistic variable scale [54]—1 = Very Low (VL), 2 = Low (L), 3 = Average (A), 4 = High (H), 5 = Very High (VH)—taking into account the professionalism of the teacher and the entire study procedure. Given the importance of the opinion of MAL’s cadets, a survey on teachers and course subjects was conducted at the Academy after the completion of each module. Study results are presented below in Section 5.

5. Study Results

In this study, fuzzy AHP was used because the fuzzy analysis takes into account the uncertainty and vagueness of the experts’ conclusion. The FAHP methodology, like AHP, was based on the pairwise comparison method, applied to recognize the relative importance of criteria (see

Table A1. The aggregated initial criteria importance weight scores presented by the expert groups.

1 CR	Course Structure		1 CR	Information Tools Usage		1 CR	Military Students’ Opinion
	2 FAHP	Rank		2 FAHP	Rank		2 FAHP	Rank
A1	0.0439	5	B1	0.5473	1	C1	0.5989	1
A2	0.6325	1	B2	0.5212	2	C2	0.4152	2
A3	0.1668	3	B3	0.0513	5	C3	0.0421	4
A4	0.1365	4	B4	0.1183	3	C4	0.1396	3
A5	0.2488	2	B5	0.1152	4

^1,2 Notes: A1 = distance learning course structure; A2 = compliance of the material with the study program; A3 = the novelty of the material chosen for the DLM; A4 = types of the knowledge testing of the participants of the module; A5 = clarity of the presented material; B1 = assessment calculation tools for knowledge testing; B2 = material scanning and availability; B3 = the possibility of personalizing the learning process; B4 = assistance that can be given to the student; B5 = resources of synchronous and asynchronous communication; C1 = the professionalism of the lecturer; C2 = assistance to students in the learning processes; C3 = organization of individual learning with feedback; C4 = practical benefits of the course for the student; FAHP = pairwise comparison weights calculated for the criteria by Fuzzy Analytic Hierarchy Process.

, Appendix A). Before calculating the final ranking of distance learning courses, the weights for each criterion were first defined. Accordingly, the linguistic terms used by the experts were converted into Triangular Fuzzy Numbers (TFNs), and then the combined decision matrix was calculated. This research used criteria fuzzy weights that connected the experts answers with a membership function characteristic presented in Table 1 and additionally in

Table A2. The aggregated importance of weight for each criterion given by experts’ groups for each distance learning module.

1 CR	2 DLM	Aggregated Decision Matrix FTOPSIS		3 FAHP
		Linguistic Terms	4 Fuzzy Number
A1	DLM1	Strongly important	(3, 5.667, 9)	(0.0168, 0.0333, 0.0816)
	DLM 2	Very strongly important	(5, 7, 9)
	DLM 3	Strongly important	(3, 7, 9)
A2	DLM1	Very strongly important	(5, 7.667, 9)	(0.2291, 0.5229, 1.1453)
	DLM 2	Strongly important	(3, 6.333, 9)
	DLM 3	Strongly important	(3, 6.333, 9)
A3	DLM1	Very strongly important	(5, 7, 9)	(0.0610, 0.1339, 0.3055)
	DLM 2	Strongly important	(3, 5.667, 9)
	DLM 3	Strongly important	(3, 5.667, 9)
A4	DLM1	Strongly important	(3, 6.333, 9)	(0.0507, 0.1121, 0.2466)
	DLM 2	Strongly important	(3, 7, 9)
	DLM 3	Strongly important	(3, 5.667, 9)
A5	DLM1	Strongly important	(3, 6.333, 9)	(0.0892, 0.1977, 0.4597)
	DLM 2	Very strongly extreme important	(5, 8.333, 9)
	DLM 3	Strongly important	(3, 7, 9)
B1	DLM1	Very strongly important	(5,7,9)	(0.1647, 0.3928, 1.0843)
	DLM 2	Very strongly important	(5,7,9)
	DLM 3	Very strongly extreme important	(7,9,9)
B2	DLM1	Very strongly extreme important	(5,8.333,9)	(0.1264, 0.3858, 1.0514)
	DLM 2	Very strongly important	(5,7,9)
	DLM 3	Very strongly important	(5,7.667,9)
B3	DLM1	Strongly important	(3,5.667,9)	(0.0169, 0.0380, 0.0989)
	DLM 2	Strongly important	(3,5.667,9)
	DLM 3	Strongly important	(3,5.667,9)
B4	DLM1	Strongly important	(3,5.667,9)	(0.0276, 0.0983, 0.2290)
	DLM 2	Very strongly important	(3,7.667,9)
	DLM 3	Very strongly extreme important	(5,8.333,9)
B5	DLM1	Moderately strong important	(1,5,9)	(0.0364, 0.0850, 0.2242)
	DLM 2	Moderately strong important	(1,5,9)
	DLM3	Strongly important	(3,6.333,9)
C1	DLM1	Very strongly important	(5, 7, 9)	(0.2446, 0.4472, 1.1049)
	DLM 2	Strongly important	(3, 6.3333, 9)
	DLM 3	Absolutely important	(7, 9, 9)
C2	DLM1	Moderately strong important	(3, 5, 7)	(0.1422, 0.4178, 0.6855)
	DLM 2	Strongly important	(3, 6.3333, 9)
	DLM 3	Very strongly important	(5, 7, 9)
C3	DLM1	Moderately strong important	(3, 5, 7)	(0.0194, 0.0335, 0.0735)
	DLM 2	Strongly important	(3, 6.3333, 9)
	DLM 3	Moderately strong important	(3, 5, 7)
C4	DLM1	Moderately strong important	(3, 5, 7)	(0.0668, 0.1014, 0.2506)
	DLM 2	Very strongly extreme important	(5, 8.3333, 9)
	DLM 3	Very strongly extreme important	(5, 8.3333, 9)

^1,2,3,4 Notes: ¹ CR- judged criteria: A4 = types of the knowledge testing of the participants of the module; A5 = clarity of the presented material; B1 = assessment calculation tools for knowledge testing; B2 = material scanning and availability; B3 = the possibility of personalizing the learning process; B4 = assistance that can be given to the student; B5 = resources of synchronous and asynchronous communication; C1 = the professionalism of the lecturer; C2 = assistance to students in the learning processes; C3 = organization of individual learning with feedback; C4 = practical benefits of the course for the student; ² DLM1-Research methodology and statistical analysis, DLM2—Social Statistics and DLM3—Political Science Research Methodology; ³ FAHP- aggregated pairwise comparison fuzzy weights for each of criteria; ⁴ Fuzzy TOPSIS number of combined decision-makers judgement.

, Appendix A. The calculated FAHP weights for the main criteria are presented in the

Table 3. The importance FAHP weights of the main criteria calculated after the experts made a pairwise comparison.

CR 1	CCA 1 Fuzzy Weights	CR 1	ITE 1 Fuzzy Weights	CR 1	MSO 1 Fuzzy Weights
A1	(0.0168, 0.0333, 0.0816)	B1	(0.1647, 0.3928, 1.0843)	C1	(0.2446, 0.4472, 1.1049)
A2	(0.2291, 0.5229, 1.1453)	B2	(0.1264, 0.3858, 1.0514)	C2	(0.1422, 0.4178, 0.6855)
A3	(0.0610, 0.1339, 0.3055)	B3	(0.0169, 0.0380, 0.0989)	C3	(0.0194, 0.0335, 0.0735)
A4	(0.0507, 0.1121, 0.2466)	B4	(0.0276, 0.0983, 0.2290)	C4	(0.0668, 0.1014, 0.2506)
A5	(0.0892, 0.1977, 0.4597)	B5	(0.0364, 0.0850, 0.2242)

¹ Notes: CR = Criteria code; CCA = Course content assessment; ITE = Information tools evaluation; MSO = military students’ opinion.

.

The calculated fuzzy AHP criteria weights then were used for distance learning courses ranking using fuzzy TOPSIS. In this study, the criterion A1 “Distance learning course structure” for the content of the distance learning course part assessment, B3 “Possibility of personalization of the learning process“ for the use of information tools part evaluation, and C3 “Self-learning organization for the participants of the distance module” for the military students’ opinion of the quality of the courses part were chosen as non-beneficial criteria, while the other criteria in this study were used as beneficial criteria. The steps of numerical FTOPSIS have been executed below for three expert groups.

Step 1: The distance learning courses in terms of fourteen criteria were evaluated. The multiple experts (34 experts) participated in the evaluation; accordingly the matrix below represents the arithmetic mean of all expert groups (see in

Table 4. Combined decision matrix for three distance learning course structure assessment by five criteria.

DLM 1	A1	A2	A3	A4	A5
DLM1	(3, 5.667, 9)	(5, 7.667, 9)	(5, 7, 9)	(3, 6.333, 9)	(3, 6.333, 9)
DLM2	(5, 7, 9)	(3, 6.333, 9)	(3, 5.667, 9)	(3, 7, 9)	(5, 8.333, 9)
DLM3	(3, 7, 9)	(3, 6.333, 9)	(3, 5.667, 9)	(3, 5.667, 9)	(3, 7, 9)

¹ Notes: DLM = distance learning course; DLM1—Research methodology and statistical analysis, DLM2—Social Statistics and DLM3—Political Science Research Methodology.

,

Table 5. Combined decision matrix for three distance learning course information tools usage assessment by five criteria.

DLM 1	B1	B2	B3	B4	B5
DLM1	(5, 7, 9)	(5, 8.333, 9)	(3, 5.667, 9)	(3, 5.667, 9)	(1, 5, 9)
DLM2	(5, 7, 9)	(5, 7, 9)	(3, 5.667, 9)	(3, 7.667, 9)	(1, 5, 9)
DLM3	(7, 9, 9)	(5, 7.667, 9)	(3, 5.667, 9)	(5, 8.333, 9)	(3, 6.333, 9)

¹ Notes: DLM = distance learning course; DLM1—Research methodology and statistical analysis, DLM2—Social Statistics and DLM3—Political Science Research Methodology.

and

Table 6. Combined decision matrix for three distance learning course assessment according to the military students’ opinion on the quality of these courses.

DLM 1	C1	C2	C3	C4
DLM1	(5, 7, 9)	(3, 5, 7)	(3, 5, 7)	(3, 5, 7)
DLM2	(3, 6.3333, 9)	(3, 6.3333, 9)	(3, 6.3333, 9)	(5, 8.3333, 9)
DLM3	(7, 9, 9)	(5, 7, 9)	(3, 5, 7)	(5, 8.3333, 9)

¹ Notes: DLM1 = distance learning course; DLM1—Research methodology and statistical analysis, DLM2—Social Statistics and DLM3—Political Science Research Methodology.

).

Step 2: The fuzzy decision matrixes were normalized using Equations (3) and (4).

Step 3: Equation (5) was used to compute the weighted normalized matrixes for three experts’ groups are presented in three tables (see

Table 7. Weighted normalized fuzzy decision matrix for three distance learning course structure assessment by five criteria.

DLM 1	A1	A2	A3	A4	A5
DLM1	(0.009, 0.017, 0.027)	(0.127, 0.446, 1.145)	(0.034, 0.104, 0.3060)	(0.017, 0.079, 0.247)	(0.030, 0.139, 0.460)
DLM2	(0.006, 0.014, 0.027)	(0.076, 0.368, 1.145)	(0.020, 0.084, 0.3060)	(0.017, 0.087, 0.247)	(0.049, 0.183, 0.460)
DLM3	(0.006, 0.014, 0.027)	(0.076, 0.368, 1.145)	(0.020, 0.084, 0.3060)	(0.017, 0.071, 0.247)	(0.030, 0.154, 0.460)

¹ Notes: DLM = distance learning course; DLM1—Research methodology and statistical analysis, DLM2—Social Statistics and DLM3—Political Science Research Methodology.

,

Table 8. Weighted normalized fuzzy decision matrix for three distance learning course information tools usage assessment by five criteria.

DLM 1	B1	B2	B3	B4	B5
DLM1	(0.092, 0.306, 1.084)	(0.070, 0.357, 1.051)	(0.006, 0.020, 0.0990)	(0.009, 0.062, 0.229)	(0.004, 0.047, 0.224)
DLM2	(0.092, 0.306, 1.084)	(0.070, 0.300, 1.051)	(0.006, 0.020, 0.0990)	(0.009, 0.084, 0.229)	(0.004, 0.047, 0.224)
DLM3	(0.128, 0.393, 1.084)	(0.070, 0.329, 1.051)	(0.006, 0.020, 0.0990)	(0.015, 0.091, 0.229)	(0.012, 0.060, 0.224)

¹ Notes: DLM = distance learning course; DLM1—Research methodology and statistical analysis, DLM2—Social Statistics and DLM3—Political Science Research Methodology.

and

Table 9. Weighted normalized fuzzy decision matrix for three distance learning course assessment according to the military students’ opinion of the quality of these courses.

DLM 1	C1	C2	C3	C4
DLM1	(0.136, 0.348, 1.105)	(0.047, 0.232, 0.5328)	(0.008, 0.020, 0.074)	(0.022, 0.056, 0.195)
DLM2	(0.082, 0.315, 1.105)	(0.047, 0.294, 0.685)	(0.006, 0.016, 0.074)	(0.037, 0.094, 0.251)
DLM3	(0.191, 0.447, 1.105)	(0.079, 0.325, 0.685)	(0.008, 0.020, 0.074)	(0.037, 0.094, 0.251)

¹ Notes: DLM = distance learning course; DLM1—Research methodology and statistical analysis, DLM2—Social Statistics and DLM3—Political Science Research Methodology.

).

Step 4: The fuzzy positive ideal solution (FPIS, A*) and the fuzzy negative ideal solution ( $\mathrm{FNIS},A^{-}$) were determined. The FPIS represents the maximum value of ${\tilde{v}}_1^{\ast }$ for all the distance learning courses, and ${\tilde{v}}_1^{-}$ is the min value (FNIS). The Fuzzy Positive Ideal Solution (FPIS) and Fuzzy Negative Ideal Solution (FNIS) are calculated using Equations (6) and (7) and are presented by three tables according to the investigation stages by expert groups’ assessment (see

Table 10. Fuzzy Positive Ideal Solution (FPIS) and Fuzzy Negative Ideal Solution (FNIS) for the distance learning course structure.

Solution 1	A1	A2	A3	A4	A5
FPIS, A*	(0.009,0.017,0.027)	(0.034,0.104,0.306)	(0.049,0.183,0.460)	(0.076,0.368,1.145)	(0.017,0.071,0.247)
$\mathrm{FNIS},A^{-}$	(0.127,0.446,1.145)	(0.017,0.087,0.247)	(0.006,0.014,0.027)	(0.020,0.084,0.306)	(0.030,0.139,0.460)

¹ Notes: FPIS, A* = Fuzzy Positive Ideal Solution; $\mathrm{FNIS},A^{-}$ = Fuzzy Negative Ideal Solution.

,

Table 11. Fuzzy Positive Ideal Solution (FPIS) and Fuzzy Negative Ideal Solution (FNIS) for information tools usage assessment by five criteria.

Solution 1	B1	B2	B3	B4	B5
FPIS, A*	(0.128, 0.393, 1.084)	(0.070, 0.357, 1.051)	(0.006, 0.020, 0.099)	(0.015, 0.091, 0.229)	(0.012, 0.060, 0.224)
$\mathrm{FNIS},A^{-}$	(0.092, 0.306, 1.084)	(0.070, 0.300, 1.051)	(0.006, 0.020, 0.099)	(0.009, 0.062, 0.229)	(0.004, 0.047, 0.224)

¹ Notes: FPIS, A* = Fuzzy Positive Ideal Solution; $\mathrm{FNIS},A^{-}$ = Fuzzy Negative Ideal Solution.

and

Table 12. Fuzzy Positive Ideal Solution (FPIS) and Fuzzy Negative Ideal Solution (FNIS) for the military students’ opinion of the quality of these courses.

Solution 1	C1	C2	C3	C4
FPIS, A*	(0.136, 0.348, 1.105)	(0.079, 0.325, 0.685)	(0.008, 0.020, 0.074)	(0.037, 0.094, 0.251)
$\mathrm{FNIS},A^{-}$	(0.082, 0.315, 1.105)	(0.047, 0.232, 0.533)	(0.006, 0.016, 0.074)	(0.022, 0.056, 0.195)

¹ Notes: FPIS, A* = Fuzzy Positive Ideal Solution; $\mathrm{FNIS},A^{-}$ = Fuzzy Negative Ideal Solution.

).

The table below shows the distance from positive and negative ideal solutions for distance learning courses (see

Table 13. Distance from positive and negative ideal solutions.

Solution 1		$S_i^{+}$	$S_i^{-}$	1${CC}_i=\frac{S_i^{-}}{S_i^{+}+S_i^{-}}$	Rank
	DLM1	0.033	0.075	0.6965	1
A	DLM2	0.070	0.037	0.3480	2
	DLM3	0.100	0.008	0.0780	3
	DLM1	0.081	0.033	0.2908	2
B	DLM2	0.102	0.013	0.1102	3
	DLM3	0.017	0.097	0.8546	1
	DLM1	0.144	0.039	0.2149	3
C	DLM2	0.065	0.135	0.6743	2
	DLM3	0.065	0.246	0.7899	1

¹ Notes: A = experts’ assessment according to A criteria; B = experts’ assessment according to B criteria; C = experts’ assessment according to C criteria; DLM1 = Research Methodology and Statistical Analysis; DLM2 = Social Statistics; DLM3 = Political Science Research Methodology; $S_i^{+}$ = gap between each alternative and FPIS; $S_i^{-}$ = gap between each alternative and FNIS; ${CC}_i$ = closeness coefficient of each alternative.

).

Notably, the best alternative is nearby to the FPIS and furthest to the FNIS. The distance learning modules (DLM1, DLM2, and DLM3) were evaluated by closeness coefficient, which helped to establish the ranking order for modules by three assessment stages (A—course structure; B—information tools usage; C—military students’ opinion) (see Table 13). Additionally, the study result is presented by a diagram where the closeness coefficient for each distance learning module is used to clarify how experts’ groups rated each of modules by fourteen criteria and three evaluation stages: first, course structure (A); second, information tools usage (B); third, military students’ opinion (C) (see Figure 1). Notable, the larger value for closeness coefficient (${CC}_i$) indicates the most preferred alternatives; accordingly, the distance learning modules ranking can be represented by (${CC}_i$). Therefore, from Figure 2, it can be identified that the largest value of course structure (A) in this study appeared for DLM1 and the lowest value for DLM3. Furthermore, when experts ranked the information tools usage (B), a different situation than they evaluated the course structure (A) was identified, because the DLM3 was evaluated as best and the lowest value was identified for DLM2. Moreover, when military students’ opinions were used for three distance learning modules evaluating, the best was identified as the DLM3 and the lowest value was given for DLM1. The presented distance learning modules’ assessment was identified according to the completed data analysis.

Additionally, the investigation analysis was conducted to assess the impact of criteria weights for different distance learning modules. The students’ opinions on the quality of the three courses are presented in Figure 3.

After evaluating the quality of three distance learning modules DLM1, DLM2, and DLM3 by four criteria C1, C2, C3, and C4, according to expressed military students’ opinion, it was identified that the distance learning module quality ranking highest was due to to the professionalism of the lecturer (C1) and lecturer assistance to students in the learning process (C2).

After analyzing three distance learning modules evaluation by experts’ ranking according to course structure (A), it was realized that the query of importance by five criteria appears as A2 > A5 > A3 > A4 > A1 for each distance learning module (see Figure A1, Appendix A). Therefore, the distance learning module quality (evaluation stage A) was highest due to the content of lecturer’s study material offered, which has to be in high compliance with the purposes of the study program (A2) and novelty and clarity of the material presented (A5).

Additionally, the study results of the second evaluation stage when the information tools usage (B) was judged must be discussed. From the calculation, it can be concluded that the ranking results of the criteria of the B stage can be ordered as B1 > B2 > B4 > B5 > B3 (see Figure A2, Appendix A). Moreover, the most important criteria were identified as knowledge testing assessment calculation tools (B1) and material scanning and availability (B2). This result led us to conclude that the three distance learning modules’ quality was correspondingly ranked in comparison with the evaluation processes of stage A (five criteria), B (five criteria), and stage C (four criteria).

This distance learning quality sensitivity analysis shows that the three stages of different expert groups’ weightings for different criteria fit the differences in the module’s grades. The points found through comprehensive analysis evidently show the sensitive background of the ideal ranking constructed on distance learning modules design. A change in the study quality criteria and quality of experts’ groups leads to changes in the distance learning course’s possible rankings.

6. Discussion

Taking into account the COVID-19 pandemic situation, the Lithuanian government has adopted an order that higher education institutions move to a virtual learning environment. Due to the pandemic situation, the higher study institutions have been forced to reorient and reorganize the entire education system process to distance learning courses in a short period of time. Therefore, like most higher education institutions, MALs are intensively developing and improving distance learning modules in order to improve the quality of studies. However, this process is not straightforward, as all universities face complex challenges that need to be addressed urgently in order to develop more student-focused studies.

Newly emerging information and communication tools make it possible to improve traditional studies, make them more acceptable, but change the main principle of study organization. Notable, the development of information technology affects all areas of human activity, including science and studies. Most of the advantages of information and communication technologies are likely associated with distance learning, which are rapidly gaining popularity due to flexibility and the ability to study at a convenient time and in a convenient location. However, the diversity of information and communication technologies and their application to distance learning does not in itself determine the efficiency of the study process. Therefore, special attention is paid to the quality of distance learning. As a result, there are increasing requirements for teachers’ knowledge of IT tools. The study results of the judged information tools’ usage (B) showed that three modules were ranked by quality in the order DLM3> DLM1> DLM2 by criteria that were ranked as B1 > B2 > B4 > B5 > B3. The results showed that all distance learning modules have must improve the quality of the possibility of the personalization of the learning process (B3).

This result is linked to other research [18] that emphasized that the learning environment should be tailored to individual needs. Applying a specific environment to teaching provides an opportunity to recommend relevant material to a particular learner according to his or her specific needs, such as learning style, modality, cognitive style, and competencies. The survey helped to identify the need for further improvements in synchronous and asynchronous communication capabilities (B5), which highlights the relevance of the need for improvement as it can be an opportunity to improve the quality of distance learning and teaching.

The same investigation result was proven by other researchers [54,55] who showed that, to improve learning conditions and quality, appropriate technological tools must be selected for the learning content, as well as IT procedures such as software agents that support students’ self-directed learning and facilitate the work of teachers. This means that each distance learning module lecturer requires the ability to select the most appropriate resources for organizing distance learning, to evaluate the possibilities of their use, to know alternative resources, and to have a clear plan for distance learning. Moreover, lecturers have to be ready to solve a number of additional difficulties related to the quality of distance learning.

Consequently, the third stage of the conducted survey analysis, which was done in order to achieve the quality of learning content and the needs of students [56,57], showed that the criteria by their importance according to students’ opinion were ranked as C1 > C2 > C4 > C3 (see Figure 2). Notably, an experienced lecturer (C1) presents learning material in an interesting way, strives to provide assistance to students in the process of organizing teaching, uses the appropriate IT tools (C2), and raises students’ motivation by choosing relevant course material to maximize the practical benefits of the course, because this develops students’ competencies and skills (C4). Strong student willingness was also expressed about the importance of the impact of feedback on the distance study results, but according to the results of this study, the students were extremely lacking (C3) (see Figure 2). The study results according to the opinion expressed by military students are similar to what was clarified in other surveys [56,58].

Moreover, after the evaluation of three distance learning courses by the FTOPSIS method by virtual learning environment and the use of IT tools, the DLM3 was rated as the best and the DLM2 as the worst. However, based on the results of the third assessment stage, when the quality of the three distance learning modules was compared by the students’ opinion, the DLM3 was rated as the best and the DLM1 as the worst. We believe that the dissimilarity of modules ranking is due to the qualification and experience of experts who participated in this survey. As was mentioned above in Section 4.2.1, this survey evaluating distance learning courses included experts who were highly interested in the quality of distance studies, but in fact were from various fields of study-related activities. These were the lecturers, cadets, IT professionals, and decision-makers from the administration department of the MAL. This multilayered approach with the replication of various interests of the participants of the evaluated distance learning courses helped to investigate different viewpoints reflecting the diverse judges’ interests. The obtained distance modules evaluations allowed us to identify the directions of the improvement of the distance courses, taking into account the evaluators’ assessments and comments. Correspondingly, it can be pointed out that this survey contributes to the development of an easily understandable hierarchy of criteria model that reflects the main goal of the study quality assessment. This research used FAHP criteria selection, grouping methods, and evaluation principles that facilitated the quality assessment of distance learning modules [59,60,61,62]. This survey had few limitations. First, the thirty-four judges were chosen with specific knowledge and skills in the specific field and with very different competencies. Second, for the distance learning module quality measurement, a different number of criteria were chosen. Third, when the composition of expert groups changed, for example, with the decreasing or increasing the number of experts, the assessment results could be different. Fourth, the source of uncertainty is the subjectivity of expert opinions. Taking into account all these limitations, the presented conclusions must be interpreted carefully. For better clarification of the quality assessment process, the additional analysis of experts’ qualification has to be done, and the additional criteria could be included.

The survey conducted was subject to a comprehensive assessment that took into account the uncertainty of the expert data; in future investigations, a comparable comprehensive course quality assessment methodology could be used to assess the quality of other similar services or courses provided. This evaluation methodology could be successfully applied for the assessment of distance learning courses that are provided by any company or institution, because the evaluation of the quality of distance learning became important for the secondary schools due to the COVID-19 pandemic situation, such that the distance education process has to be organized. Such studies could provide further information about student complaints.

7. Conclusions

This used Fuzzy-TOPSIS-based analysis to identify the value for distance learning courses’ quality. The basis of the analysis used was fuzzy logic and TOPSIS multi-criteria technique. The used analysis’ mixture was demonstrated to be a good alternative with excellent interaction, which can make a comprehensive assessment of three courses by fourteen criteria.

The DLM evaluation was done on three levels: the first stage was the evaluation of distance learning course structure; second, the evaluation of the quality of information tools usage; and third, the military students’ opinion of the quality of distance learning courses regarding the problem of adaptability to distance course user requirements. The survey results showed the fuzzy TOPSIS technique to be a practical method that provides a very satisfactory value analysis and ranking. The conducted survey showed that the backgrounds and ways of assessing the distance learning modules’ quality are appropriate for future usage.

The MCDM techniques FAHP and FTOPSIS were used to select the best distance learning course and to identify the criteria ranks. The study used fourteen criteria for the ranking of the three DLMs. Moreover, the results of the evaluation of the three courses and the application of the AHP Fuzzy method, which allowed us to determine the importance of the criteria weights, helped identify the shortcomings of the distance course modules. However, in summarizing the results of the three-stage assessment in this study, it was not taken into account that the importance of the assessment stages differed due to this approach: the scope of studies differs, as does the time required to prepare and evaluate the DLM.

The result of FTOPSIS and sensitivity analysis of criteria let us identify the best ways to increase distance learning quality; accordingly, the lower-ranked criteria were the distance learning structure (A1), the possibility of the personalization of the learning process (B3), and the strategy of self-learning for participants of the distance module. It can also be concluded that the possible explanation for the best ranking of DLM3 belongs to the enclosed survey results, where the majority of the criteria assessed by three expert groups in this study module were evaluated to be high.

The modular approach resolves the problem of the importance of distance course assessment in determining the specific weights of criteria for which the military academy administration department is responsible. Moreover, it was identified that the individualization of the teaching process can be an important lecturer decision to improve the quality of education, which requires that many factors be taken into account, including learner profiles, learning materials, and learning strategies.

For future investigations, a group of experts can be included for the assessment by other MCDM techniques such as Preference Ranking Organization Method for Enrichment Evaluation (PROMETHEE). In addition, the PROMETHEE method can be applied with fuzzy logic, interval number, or hesitant fuzzy sets.