The Transportability of a Game-Based Learning Approach to Undergraduate Mechanical Engineering Education: Effects on Student Conceptual Understanding, Engagement, and Experience

Abstract

Many game-based instructional designs have demonstrated effectiveness for a variety of educational outcomes, although typically in limited contexts. In this article, we report the results of a four-year study testing the extent to which a game-based learning approach to undergraduate engineering education demonstrating promising results in a university course was transportable to other engineering courses and universities. We evaluated students’ conceptual understanding, emotional engagement (with the Experience Sampling Method), and experience when using the video game, Spumone, for their coursework compared to a textbook-based control condition. Multilevel models and other quantitative analyses showed that the effect of the experimental condition (i.e., game-based) on conceptual understanding and student engagement was not significant. Based on a content analysis of students’ feedback, however, the students reported a positive experience with game-based learning for their assignments overall. Areas of need towards successful implementation of the game-based learning intervention were also examined. This study has important implications for the salience of implementation issues including adequate training and continuing teacher professional development, and ongoing supports for instructors and students to aid in the learning of concepts that the game was intended to teach.

1. Introduction

There has been growing interest in the potential effectiveness of educational video-game based approaches for teaching and learning, especially in STEM subjects (i.e., science, technology, engineering, and mathematics) [1,2,3,4]. Many game-based instructional designs have demonstrated effectiveness for a variety of educational outcomes, although typically in limited contexts [1,4]. For game-based educational innovations to obtain greater reach, an important concern is their transportability to additional contexts beyond those establishing initial success. In this paper, we report the results of a four-year study testing the extent to which a video game-based learning approach to undergraduate engineering education demonstrating promising results in a university course was transportable to other engineering courses and universities.

Students in undergraduate engineering courses typically work through challenging concepts by completing textbook problems in homework and labs. However, those problems are frequently narrow in scope and antithetical to what attracted many such students to engineering in the first place: the desire to design, build, fix, and create vehicles or structures that “work.” Thus, scholars have recognized the need to introduce more carefully structured, open-ended challenges encouraging design thinking early in the curriculum. Modern video games are uniquely poised to help provide such open-ended and scaffolded learning experiences [5,6]. Initial challenges are often simple, but players are quickly advanced to higher levels and confronted with more complex challenges. The goals of the game are clear, and feedback as to whether they are achieving the goals is immediate, abundant, and unambiguous.

In this study, we examined the transportability of a game-based learning approach to undergraduate engineering based on the video game, Spumone, which was created and designed to address the teaching and learning challenges inherent to engineering education described above. In Spumone, students pilot a vehicle named the “spuCraft” through a subterranean, simulated world to reach a goal state without crashing. To be successful, students need to devise and apply strategies learned in an undergraduate engineering dynamics course, and express them mathematically through an equation parser, to solve the game-based challenges that would otherwise be almost impossible to complete by hand (see Figure 1) [7]. Research studies documented greater gains in terms of learning and emotional engagement among students who took the game-based course, as taught by the same engineering professor who designed Spumone (fourth author), compared to a control group [7,8].

This work was anchored in overarching goals to promote sustainability, especially the UN’s Sustainable Development Goals addressing quality education (Goal 4); Work and Economic Growth (Goal 8); Industry, Innovation, and Infrastructure (Goal 9); and Responsibly Consumption and Production (Goal 12) [9,10]. The game-based approach was intended to improve the quality of the teaching and learning in undergraduate engineering courses focusing on the safe and responsible design of dynamic systems important for a sustainable future (e.g., by reducing accidents) [11]. While it is estimated that the COVID-19 pandemic impacted the schooling of 91% of students worldwide [9], a game-based learning approach is a step towards providing high quality education in an online format that is accessible remotely. The use of new innovations such as educational video games and simulations for early career training also reduces both the human and material risks and costs associated with testing of designs for dynamic systems in mechanical form [12].

We investigated the following research questions:

(1)

When adopting instructors used Spumone in their teaching, did they experience similar gains in students’ conceptual understanding that the inventing professor experienced?

(2)

When adopting instructors used Spumone in their teaching, did students experience similar gains in emotional engagement compared to those of the inventing professor?

(3)

How did the students in the adopting courses experience Spumone as an educational tool? Specifically, we focused on students’ attitudes: (a) Did they like using Spumone for their coursework? (b) What aspects of the Spumone-based program did they like and dislike? (c) What suggestions did they have for improvement? (d) Did they characterize their experience overall positively or negatively?

(4)

What areas of need can be identified to help adopting instructors to adopt, adapt, or reinvent Spumone to make it a regular part of their teaching practice?

1.1. Video Games and Flow in Engineering Education

For decades, education scholars have been studying video game-based approaches to teach [13,14,15,16,17,18]. What they have found is that the most successful games “teach” their players how to solve complex problems. The problems within a game typically start off rather easy and then progressively get more difficult as players’ skills develop. Players are motivated to learn within video games because it is clear that knowledge equals power within the game. The learning is situated, and occurs through a process of hypothesizing, probing, and reflecting upon the simulated world within the game. The goals are clear. Games provide players immediate and unambiguous feedback on how well they are progressing. Information becomes available to players at just the time they will be able to make sense of it and use it.

In this study, we tested the transportability of a game-based learning approach to undergraduate mechanical engineering education. For the purposes of this study, “game-based learning” refers to learning with the support of an educational video game, where the educational video game is intended to and designed for specific learning goals. To define game-based learning, we rely on Shute and Ke’s [19] definition of video games as used in Tokac, Novak, and Thompson’s [3] meta-analysis of the effects of game-based learning on mathematics achievement. According to this definition, “good games must have interactive problem solving, specific goals/rules, adaptive challenges, control, ongoing feedback, uncertainty that evokes suspense and player engagement, and sensory stimuli (a combination of graphics, sounds, and/or storyline used to excite the senses). These gaming characteristics are essential for creating an effective learning environment that enhances student engagement and facilitates knowledge and skill acquisition” ([3], p. 408) . We distinguish educational video games from simulations, which can model systems, procedures, and/or phenomena, but do not necessarily include learning objectives [3,20]. Educational video games also create a game-like experience through a sense of competition, and rewards for completing challenges including points and/or advancement to more challenging levels, but they are distinct from all forms of “gamification,” which also provide these game-like elements but do not always represent immersive educational experiences with specific learning goals [21,22].

Within the highly engaging techniques that game designers employ to get players to “learn” the game, one finds the enactment of modern principles and theories of learning such as constructionism, inquiry-based learning, problem-based learning, and anchored instruction [23,24]. Much of the emerging scholarship on video game design is explicitly grounded in scholarship on cognition, including concepts such as Vygotsky’s zone of proximal development [25,26,27,28]. Theories of what make video games fun focus on learning and problem solving [29,30]. According to Koster [30], a game becomes fun when it requires players to gain new skills at a deep level that get “chunked” and absorbed into the subconscious mind, and then requires players to apply the skills/knowledge toward some goal. Furthermore, it remains fun if it requires players to gain new skills/knowledge, or transfer their skills to new problems within the game. Ideally, this is the type of “fun” we would like engineering education to be.

When playing a video game, individuals are said to “plunge into it” and to be “immersed” or “enveloped” by it [31]. The theoretical perspective undergirding this study was based on the theory of “flow” [32,33,34]. Flow is a state of deep absorption in an activity that is intrinsically enjoyable. Individuals in this state perceive their performance to be pleasurable and successful, and the activity is perceived as worth doing for its own sake, even if no further goal is reached [34]. Flow occurs when individuals stretch the limits of their abilities to meet challenges, such that skills are neither overmatched nor underutilized, and can take on greater challenges as their skills improve. Another primary condition for flow experiences is having clear goals, and then receiving immediate feedback on one’s performance towards meeting those goals [34,35]. Perhaps not surprisingly, flow theory has been the natural theoretical base for exploring the implications of learning through immersion or “being enveloped” by a virtual learning environment [36,37,38].

Hamari et al. [39] investigated the impact of flow (operationalized as heightened challenge and skill), engagement, and immersion on learning in two video game-based learning environments: Spumone and Quantum Spectre. The sample consisted of 140 undergraduates using Spumone for assignments in their undergraduate mechanical engineering classes, and 134 high school students in 11 classrooms across the US who played the game, Quantum Spectre, as part of their physics unit on optics. In support of flow theory, findings from the study suggested that increasing levels of challenge and skills through educational video games can increase immersion and engagement to promote learning.

In previous studies drawing on flow theory and methodology to study student engagement, students were found to be significantly more engaged when playing Spumone to complete their mechanical engineering assignments compared to when using only a textbook to do so [8,40]. Prior to the introduction of the game, students tagged most of their experiences in the course as “like work.” After the game was introduced in the same course by the same instructor (in a subsequent semester with a new student roster), the majority of experiences flipped to being labeled as “like both work and play.” Experiencing activities as “like work and play” is a marker of flow that undergirds creative achievement [41]. Students using the game-based approach also made considerably higher learning gains, as demonstrated by their performance on tests of course materials, almost a full standard deviation higher on exams measuring competency in the course than the control group [42]. These studies demonstrated that, by engaging undergraduate students in problem-based thinking within engineering communities of practice [43], a game-based learning approach offered a variety of benefit to the future careers of budding engineers. In keeping with Sustainable Development Goal 8, this approach helped to forge early career paths towards high quality jobs, contributing to economic growth and a higher standard of living [9,44].

1.2. The Implementation and Transportability of Instructional Technologies

In the present study, we examined the transportability of a game-based learning approach to teach undergraduates the mechanical engineering course, Engineering Mechanics: Dynamics, based in Spumone game challenges. Dynamic systems can be easily found in numerous engineering applications. In the Dynamics course, students learn mechanics principles by applying them to real-world problems. Examples include the design and control of dynamic systems ranging from motors to industrial manipulators to airplanes.

In asking what is needed for Dynamics instructors to adopt, adapt, or reinvent Spumone to make it a regular part of their teaching practice, our study focuses on issues of educational technology implementation. Such issues can be beneficially viewed from the perspective of implementation science and design-based implementation research [45,46,47,48,49,50,51]. Implementation has been defined as “the constellation of processes intended to get an intervention into use within an organization” [46], p. 3.

Executing the intervention with fidelity, or as it was intended, is a critical aspect of implementation [46,52,53]. The adopting instructors’ plans for implementation, and the extent that they are aligned with the intended plans of the designers, can provide a measure of fidelity. Another related factor is the adequacy of teacher training or education for the implementation to succeed [45]. The frequency, duration, and sustainability of content coverage (i.e., “dosage”) has also been identified as critical features in effective implementation [54,55].

Other factors influencing the implementation of interventions include characteristics of (a) the intervention, (b) the inner and outer setting, (c) the individuals involved, and (d) the process by which the intervention occurs [56]. Characteristics of the intervention include its source and evidence, strength, and quality. Two other important aspects of the intervention predicting the success of implementation are its complexity and adaptability. Characteristics of the setting have been conceptualized by a distinction between the inner and outer setting. The outer setting includes the economic, political, and social context in which an organization resides, and the inner setting includes structural, political, and cultural contexts through which the implementation proceeds [57]. Relevant to the inner setting is the implementation climate. This includes receptivity of involved individuals. It also includes compatibility, or the extent to which the intervention fits with existing norms, values, workflows, and systems [49,58]. Characteristics of the individual include knowledge and beliefs about the intervention, necessary competencies and background, self-efficacy, and the individual’s stage in the change process [46]. Characteristics of the process include planning, engaging in, executing, and reflecting on the intervention [46].

1.3. Study Expectations

Due to the success of Spumone upon initial testing, our team was cautiously hopeful of finding similar gains in conceptual understanding and student engagement as had been observed by the inventing professor. As to whether this would be possible despite the many barriers to successful implementation substantiated by the research literature was very difficult to predict, so we ventured no firm hypothesis. We were also hopeful that that students’ experience would be positive, and cautiously expected positive comments from students along with constructive feedback from which we could improve implementation in the future. We expected that specific areas for improvement would be related to implementation factors identified in the literature (e.g., fidelity, adequacy, intervention quality, complexity, adaptability, climate, and compatibility), but we did not speculate in greater detail beyond this.

2. Materials and Methods

2.1. Participants

2.1.1. Recruitment of Adopting Instructors

Two male faculty members who teach undergraduate mechanical engineering were recruited, one from the same institution as the male professor who originally designed the game (Institution A), and one from another institution (Institution B). Both understood the potential benefits of Spumone, expressed a positive attitude about it, and volunteered to adopt and integrate it into their courses.

2.1.2. Student Participants

The student sample (N = 243) consisted of the students who took the course in Institution A (n = 140) and Institution B (n = 103), and those who took the course in 2015 (control; n = 64), 2016 (experimental group 1; n = 103), and 2017 (experimental group 2; n = 76). As the study occurred in the context of ongoing for-credit college courses, assurance of participant anonymity was a paramount concern. Therefore, personal background information was not collected. In general, the participant sample was predominately male and ethnically diverse, consistent with the student population of many Schools of Engineering in the US.

2.2. Procedure

2.2.1. Baseline and Experimental Data Collection

In the spring 2015 semester (control), the adopting instructors taught their engineering dynamics courses as normal, without the game, in order to gather baseline data for comparison purposes. Data gathered in the control year was compared to two experimental years in which Spumone was used for assignments; spring 2016 and spring 2017. The pretest was administered before the beginning of each semester, and the posttest was administered at the end of each semester.

2.2.2. Training in Spumone

During the summer of 2015, faculty testers participated in a training course on Spumone, working through all of the game challenges and familiarizing themselves with online instructions. In the training, the creator of Spumone demonstrated how to complete some of the game challenges and answered questions regarding the game-based course.

2.2.3. Game-Based Learning Intervention

In the game-based learning intervention, students played Spumone to pilot a vehicle called the spuCraft in a two-dimensional subterranean world with a game controller. For example, to aid in the understanding of Newton’s second law relating to force and motion, students are asked to complete a “Drop” challenge. In the novice level of the challenge, the spuCraft is dropped into a subterranean cavern and students are able to control the thrusters of the vehicle so it can slow down or turn. Since it is almost impossible to reach the final goal state while avoiding deadly drifting flowers by hand, students need to write mathematical rules that will precisely control the thrusters to match the speed of the protective light moving on a vertical belt (see Figure 1). Writing the correct mathematical rules to complete this challenge requires a full understanding of Newton’s second law (for more information about Spumone, its challenges, and relation to engineering dynamics, see http://www.spumone.org/).

Students were asked to solve two or three such challenges every two weeks on average. Assignments also included exercises such as drawing a free body diagram, decomposing a vector equation, and providing explanations in answer to questions aimed at supporting students’ thinking processes.

2.3. Measures

2.3.1. Measure of Conceptual Understanding

As in the original studies of the Spumone intervention, the Dynamics Conceptual Inventory (DCI) [59] was used to measure conceptual understanding. As a pretest measure of student understanding of basic physics concepts, we used items from the Mechanics Baseline Test [60] and the Force Concept Inventory [61]. Both are reliable and valid instruments [60,61].

2.3.2. Measure of Student Engagement

As in the previous studies, the emotional state of the students while engaged in the learning process was measured using the Experience Sampling Method (ESM). The ESM is a valid and reliable methodology for measuring emotions and the quality of experiences “in the moment” of engagement and therefore does not rely on memory to reconstruct engagement from past experiences [62,63]. In the ESM, subjects are prompted to take short surveys at predetermined moments as they are engaged in an activity. ESM items were embedded in the Spumone game and were administered by pencil and paper for all non-game learning experiences (i.e., homework). After Spumone game challenges, students were asked nine items related to their engagement with the prompt, “Just before this survey …” (e.g., “did you feel immersed in what you were doing?” Primary measures were student engagement (three Likert-scale items: interest, enjoyment, concentration; α = 0.85), flow (three items: immersion, challenge, and perceived skills; α = 0.84), meaningful learning (two items: perceived importance and perceived learning; α = 0.84), and serious play (one categorical item: felt like play and work instead of feeling like play only, work only, or neither).

2.3.3. Student Experience with Spumone

Student participants were asked to submit their feedback, comments, or suggestions about their experience with Spumone in writing to the adopting instructors at the end of each course.

2.3.4. Areas of Need for Future Improvement

Two data sources were used to identify areas of improvement for future transportability and implementation. The first was students’ written feedback about their experience with Spumone at the end of each semester. The second was adopting instructors’ observations, as documented in teacher notebooks and as solicited by a semi-structured interview conducted by one of the investigators who was not an adopting instructor.

2.4. Data Processing and Analyses

For research question 1, we sought to analyze gains in conceptual understanding among students who received the Spumone intervention compared to the control group. For determining if there is a significant pretest-posttest gain in an experimental group compared to a control group, a highly recommended approach is an ANCOVA (analysis of covariance) model predicting the posttest score with the experimental condition as the predictor variable of interest and pretest score as covariate. We used this model for determining gains made in conceptual understanding. We also compared the mean pretest-posttest gains by year with ANOVA (analysis of variance), and examined the means and standard deviations by site and year.

In total, 6675 self-reports of emotional engagement were nested within the 243 student participants who completed them. Students responded to questions about their subjective experience at various points when playing Spumone during assignment (experimental students only). These same students also completed homework without the game, creating a control condition among the same group of students. For research question 2, the preferred test of gains in emotional engagement was a multilevel, within-person comparison of engagement when using the game compared to homework without the game in the two experimental years only [64]. We conducted separate multilevel models for each scale and item, each of which had a single predictor for whether they were playing Spumone or not using Spumone when doing homework for the class. This analysis overcomes the potential influence of person-level factors when comparing different groups of individuals; nevertheless, we also compared student mean engagement variables between control and experimental students.

For research questions 3 and 4, we analyzed student comments and adopted instructors’ field notes in the two experimental years from Institution A only (data were not available from Institution B). Analyses focused on the identification areas of need for Spumone transportability and adaptation to be successful. To do so, we used content analysis to code concepts embedded within the texts [65,66]. We first divided up each student’s comments into meaning units, and then formulated codes for each based on patterns in the comments. Thirty codes were generated from 343 meaning units coming from a total of 61 students in the two experimental years. Each meaning unit was coded with one of these codes. We then counted the number of students providing each code to generate the frequency of students who had addressed a given topic. We also counted the number of times each code was assigned to a meaning unit to produce the total frequency of each code. Finally, based on the results of the content analysis, we rated each student’s overall experience while using Spumone in the course on a five-point scale from 1 (very negative) to 5 (very positive) based on their comments as a whole.

3. Results

3.1. Adopting Courses Student Gains in Conceptual Understanding

Some items were common between the pretest and posttest and some were not. We analyzed as dependent variable scores on both the full posttest and posttest items in common with the pretest only. We first report results predicting the full posttest. Results of an ANCOVA model predicting full posttest scores in conceptual understanding with the game condition and pretest as covariate are presented in

Table 1. a. ANCOVA model predicting posttest scores in conceptual understanding with the game condition and pretest as covariate. b. Pretest and posttest total means, standard deviations, and mean pretest-posttest gains in conceptual understanding.

a
Source	Type III Sum of Squares	Mean Square	df	F	Sig.	Partial Eta2
Corrected Model	2066.322 a	2	1033.161	25.695	0.000	0.192
Intercept	4042.413	1	4042.413	100.534	0.000	0.318
Pretest Total	2023.103	1	2023.103	50.314	0.000	0.189
Game Condition	45.546	1	45.546	1.133	0.288	0.005
Error	8685.221	216	40.209
Total	127,063.000	219
Corrected Total	10,751.543	218
b
	Pretest		Posttest		Pre-Posttest Gain
	M	SD	M	SD	M
Control Year (2015)	18.16	5.85	23.72	6.77	5.56
Exp. Year 1 (2016)	18.35	6.03	24.25	6.87	5.90
Exp. Year 2 (2017)	18.00	5.85	20.29	6.89	2.29

Note. ^a R² = 0.192 (Adjusted R² = 0.185). Partial Eta² indicates the effect size.

a. The effect of the experimental (i.e., game-based) condition on the posttest for conceptual understanding was not significant (F = 1.13, n.s.).

We also examined the means and standard deviations by year and compared the mean pretest-posttest gains by year with ANOVA. Results are provided in Table 1b and mean differences by year are illustrated in Figure 2. In the control year (2015), the mean gain from pretest (M = 18.16, SD = 5.85) to posttest (M = 23.72, SD = 6.77) was 5.56. In the first game-based year (2016), the mean gain from pretest (M = 18.35, SD = 6.03) to posttest (M = 24.25, SD = 6.87) was 5.90. In the second game-based year (2017), the mean gain from pretest (M = 18.00, SD = 5.85) to posttest (M = 20.29, SD = 6.89) was 2.29. The ANOVA of mean pretest-posttest gain by year was significant (F = 5.09, p < 0.01). A post-hoc Tukey test revealed that the mean gain in the second game-based year was significantly lower than that of the control year and first experimental year, but that the gains in the control and first experimental year were in the same statistical category.

To understand these somewhat unexpected results better, we examined the mean scores by year and institution. In Institution A, the pretest-posttest difference was between 6 and 7 in all three years. The average gain in Institution B was 2.4 in the control year, 5.1 in experimental year 1, and −3.1 in experimental year 2. Thus, in the second experimental year in Institution B, students appeared to perform worse at the end of the course than at the beginning.

We also examined the results for the common pre- and posttest questions only. The overall ANCOVA test presented in

Table 2. a. ANCOVA model predicting posttest common questions scores with the game condition and pretest as covariate. b. Pretest and posttest common questions means, standard deviations, and mean pretest-posttest gains.

a
Source	Type III Sum of Squares	df	Mean Square	F	Sig.	Partial Eta2
Corrected Model	495.624 a	2	247.812	69.976	0.000	0.392
Intercept	171.136	1	171.136	48.324	0.000	0.182
Pretest Same Qs	450.994	1	450.994	127.349	0.000	0.370
Game Condition	1.040	1	1.040	0.294	0.589	0.001
Error	768.486	217	3.541
Total	6410.000	220
Corrected Total	1264.109	219
b
	Pretest		Posttest		Pre-Posttest Gain
	M	SD	M	SD	M
Control Year (2015)	4.59	2.17	5.50	2.44	0.91
Exp. Year 1 (2016)	3.40	1.86	4.36	2.41	0.96
Exp. Year 2 (2017)	3.56	1.89	4.59	2.26	1.03

Note. ^a R² = 0.392 (Adjusted R² = 0.386). Partial Eta² indicates the effect size.

a was not significant (F = 0.294, n.s.).

Means, standard deviations, and pretest-posttest gain scores on the common questions by year are presented in Table 2b. The mean pretest-posttest gain was 0.91 in the control year, 0.96 in the first game-based year, and 1.03 in the second game-based year. As confirmed by an ANOVA comparing mean gain scores, there was very little difference in conceptual learning gains between the control and experimental years as measured by the common questions only. For institution B, mean gain was -0.11 in the control year, 0.17 in experimental year 1, and 0.69 in experimental year 2. Thus, institution B results did not appear to have the same problem regarding a learning loss from pretest to posttest in 2017 when using the common question measure. This suggests that there may have been substantial errors in the reliability of the full score measure in 2017 for Institution B.

Overall, evidence that there were significant conceptual learning gains in the game condition compared to the control condition was not compelling.

3.2. Adopting Courses Student Gains in Student’s Emotional Engagement

To measure emotional engagement, students responded to questions about their subjective experience at various points when playing Spumone during assignments (experimental students only). This was compared to engagement when doing homework without the game (control condition). Results are provided in

Table 3. Multilevel within-person comparison of engagement measures in Spumone vs. homework.

Scale or Item	Game vs. Homework Coefficient (SE)	R2
Scales
Engagement	0.01 (0.05)	0
Flow	−0.18 (0.05) ***	0.004
Meaningful learning	−0.50 (0.06) ***	0.03
Full scale	−0.30 (0.05) ***	0.01
Items
Enjoy	0.32 (0.07) ***	0.01
Interesting	0.13 (0.06) *	0.002
Concentrating	−0.42 (0.06) ***	0.02
Immersed	0.08 (0.06)	0.001
Challenging	−0.05 (0.06)	0
Skills	−0.56 (0.06) ***	0.03
Important	−0.34 (0.06) ***	0.01
Learning	−0.65 (0.06) ***	0.04
Like work and play	4.13 (0.74) a ***	NA

Note. Scales refer to full scales, and items refer to individual items composing the scales. * p < 0.05, *** p < 0.001. ^a Odds ratio from a logistic regression model.

. Coefficients for the game/no-game comparison are shown from each of the multilevel models (4 scales and 9 items for a total of 13 multilevel models). There was no significant difference in student engagement when playing Spumone vs. doing homework (coefficient = 0.01, n.s.). Students reported significantly more flow and meaningful learning when completing assignments without Spumone compared to completing them with Spumone (coefficients = −0.18 and −0.50, respectively, p < 0.001). From examining the individual ESM items, students reported more enjoyment (coefficient = 0.32, p < 0.001), interest (coefficient = 0.13, p < 0.05), and serious play (i.e., activity felt like work and play, odds ratio = 4.12, p < 0.001) when playing Spumone during homework compared to completing homework without it. However, they indicated significantly more concentration, perceived skills, perceived importance, and perceived learning when completing homework without Spumone compared to completing assignments with it. Multilevel models with site, year, site * game_condition, and year * game_condition interactions were also analyzed. Few of these effects were significant, however, so Table 3 results are derived from models without these covariates.

We also made a between-person comparison of average emotional engagement of students in the control year (homework only) compared to experimental students (when playing Spumone only). The directionality of the differences was similar to the within-person analyses. However, there were no significant differences in any of the scales. Control students reported significantly more concentration (t = −2.53, p < 0.05) and perceived skills (t = −4.83, p < 0.001) doing homework than did experimental students playing Spumone. No other results were significant.

Overall, the results were surprising. Based on the original studies of Spumone effectiveness, we expected that students would report greater conceptual understanding, engagement, flow, and perceived learning when completing course work in the game. Although we did expect greater interest and enjoyment when playing Spumone, we also expected greater overall engagement, including higher levels of concentration, immersion, and perceived learning.

3.3. Experimental Students’ Experiences Playing Spumone for Coursework

Students’ perspectives of their experience with Spumone were drawn from an analysis of their written comments, as shown in

Table 4. Codes and Frequencies.

Codes	Frequency and Percentage of Students Who Referred to the Code a	Percentage of Meaning Units Referring to Each Category b
Enjoyable program	25 (41%)	I liked the program very much. 140 (41%)
Great idea	36 (60%)
Helping learning	25 (41%)
Satisfaction on completion	12 (20%)
Hoping for more levels/assignments	4 (7%)
Hoping to see it in other courses	7 (12%)
Convenient SpuPilot c	3 (5%)	I liked these aspects related to the program. 19 (6%)
Helpful in-class instructions	3 (5%)
Helpful website/video instructions	5 (8%)
Helpful write-ups	3 (5%)
Not textbook problems	4 (7%)
Mixed feelings	5 (8%)	I did not like the program that much. 49 (14%)
Frustrating	15 (25%)
Difficult challenges	11 (18%)
Not interesting	7 (12%)
Inconvenient SpuPilot	17 (28%)	I did not like these aspects related to the program. 72 (21%)
Insufficient in-class hints/instructions	17 (28%)
Insufficient website/video instructions	10 (16%)
Too difficult write-up assignments	8 (13%)
Traditional homework	4 (7%)
Lack of integrated tips	4 (7%)
Group work needed	5 (8%)
Too frequent surveys	7 (12%)	Some improvements are needed. 51 (15%)
Uneven challenge preference	6 (10%)
Room for improvement	7 (12%)
Minor specific suggestions	7 (12%)
Minor technical issues	11 (18%)
Grading issues	4 (7%)
Good class overall d	3 (5%)	Extraneous 12 (4%)
N/A d	5 (8%)

^aN = 61 students ^b N = 343 meaning units ^c SpuPilot is the interface that allows programming commands, triggers, functions, and so on. ^d comments not related to the Spumone program.

. Forty-one percent of the students stated that they enjoyed the Spumone program, and 60% believed it was a great idea to use such a program in the course. One student shared, “the idea of applying what we learn in class into an interactive virtual world is amazing.” Another offered, “I believe it is an incredible idea to have students learn through this video game program.” In contrast, 25% of students expressed frustration, and 18% expressed difficulty using Spumone. For example, one student said, “I feel Spumone can be extremely frustrating and time consuming at times.”

As also shown in Table 4, when codes were grouped into superordinate categories expressive of students’ general attitude, 41% of the total number of meaning units were considered to reflect the attitudinal disposition of “liking the course” overall, whereas only 14% of them reflected “not liking” the course overall. In addition, 21% of the content from the comments identified specific features of the course that students did not like, whereas 6% identified course features that students liked.

In

Table 5. Overall experience interpreted and rated from student comments.

Score a	Frequency of Students b
5	13 (21%)
4	21 (53%)
3	10 (16%)
2	2 (3%)
1	4 (7%)

^a 5—very positive; 4—somewhat positive; 3—neutral; 2—somewhat negative; 1—very negative; ^b N = 61 students.

, we present the overall experience the students expressed about using Spumone as interpreted from the student comments as a whole and rated on a five-point scale. A score of 5 indicates a “very positive” experience whereas a score of 1 indicates a “very negative” experience. As the table shows, the overall experience of the majority of the students was rated as either “very positive (21%) or “somewhat positive” (53%), demonstrating an overall positive attitudinal bias of students’ experience with using Spumone in the course.

3.4. Areas of Need for Successful Transportability and Implementation

3.4.1. Perspective of Students

In their comments, students suggested that game challenges were not always well aligned to classroom instruction and assignments, making it difficult to see the conceptual understandings that the game was intended to teach. Our analysis of their comments (Table 4) shows that about five percent (5%) of students said that in-class instructions were helpful, but many more (28%) pointed out the necessity of providing more instructions and guidance in class.

The investigator who had created Spumone and obtained the original promising results (fourth author) made available a website that provided guidance for some challenges (http://www.spumone.org/). In their comments, students also suggested the need for more accessible and detailed guidance, perhaps embedded in the game itself, than was provided by this website. About 8% of students stated that the website was helpful, while 16% asserted that more online or in-game guidance was needed. As one student suggested, “adding more information into the game itself would be nice.” Approximately 15% of student comments identified additional specific areas of need for improvement, including technological glitches and grading issues (see Table 4).

3.4.2. Perspective of Adopting Instructors

Based on the observations of adopting instructors, there were many things that needed to be done for Dynamics instructors to adopt, adapt, or reinvent Spumone to make it a regular part of their teaching practice. We identified sets of practices in several different categories. The first set of practices related to the training of adopting dynamics instructors. In the current project, the instructor training primarily consisted of a “crash course” on the Spumone game itself. Adopting instructors indicated that in order to afford them with maximum flexibility and adaptability, there was virtually no training on pedagogy or instructional practices to support student learning with the game-based approach.

Relatedly, a second set of practices needed for successful implementation related to instructional integration. Adopting instructors admitted that they did not put a lot of thought or effort into integrating the game-based learning approach into their instruction. There was also minimal training or guidance on how to do so. Rather, adopting instructors typically demonstrated Spumone in class and left students to use it for certain assignments as an instructional supplement. Instructors stated that they did not adjust instruction for the addition of the Spumone component of the course. Thus, Spumone was being added to the workload that students in previous courses had; nothing was being subtracted from it or substituted to keep the workload comparable. This may have contributed to the frustration expressed by some students.

Both of the adopting instructors identified the need for a more thorough manual on the implementation of Spumone with suggestions of instructional practices supporting student learning. The adopting instructors also expressed the need for continuing support, such as continuing teacher professional development workshops, and perhaps professional learning communities (PLCs) or mini-conferences that could include both instructors and students, and a mix of attendees who have and have not used Spumone.

Upon reflection, the game interface and game experience itself proved to be incompatible for transportability. Based on observation, for example, students did not appear to respond to many game challenges as intended. Thus, improvements to the technological innovation itself may be needed for successful transportability. In order to solicit the desired student thought processes and interactions, for example, the game creator believed that the game interface may need to be made more intuitive to students. Adopting instructors also indicated that there were technological issues that prevented more effective adoption. For example, the Spumone program was sometime prone to crashing.

4. Discussion

In this study, we examined the transportability of a game-based learning approach to teaching the undergraduate mechanical engineering course, Engineering Mechanics: Dynamics. Previous studies indicated that students taking this course and completing assignments by playing the video game, Spumone, made significant gains in terms of conceptual understanding and student engagement compared to a control group [8,40,42]. Results of the present study suggest that those gains were not replicated upon an initial attempt to transport the game-based approach to other instructors and institutions. Gains were made in conceptual understanding throughout the academic year, but no more so than gains made in the control year without the use of Spumone. In addition, there were no significant differences in student engagement when students in the Spumone-based class completed assignments with the game versus without the game. In fact, students reported higher levels of flow, perception of meaningful learning, and overall subjective experience (i.e., all experience sampling items combined in a single scale) when working without the game than with the game. They did report greater enjoyment, and were more likely to identify their homework as like both work and play, when completing assignments with the game-based learning approach, however.

Students who took the game-based course generally indicated enjoying the experience, and, on average, students’ reflections on using Spumone were overwhelmingly positive. The majority of students (60%) expressed the view that the game was a great idea, and 41% believed the game helped them to learn. These results were reflective of the overall positive experience and engagement with learning that much of the literature on educational video games has previously reported [1,2,3,4,13,14,15,16,17,18]. Meanwhile, approximately one quarter of the students (25%) expressed some degree of frustration, while 18% described the game as challenging and difficult. Students also provided a number of constructive suggestions for improvement, as discussed below.

A good question is how to interpret the somewhat mixed overall findings: The game was not as effective with respect to improving conceptual understanding and student engagement once transported to other courses as originally demonstrated, and yet students reported a positive experience with it overall upon reflection. The mixed nature of these findings may suggest that the game is a strong idea with promising upside potential, but that there may be a great deal of personal, tacit knowledge [43,67] regarding the implementation of the innovation as originally designed that needs to be captured, made explicit, and transported, along with the technological innovation itself. Thus, we discuss areas for improvement in the context of principles of implementation science below [45,46,48,50,51].

4.1. Areas of Improvement for Successful Implementation

A critically important, overarching aspect of educational intervention implementation is fidelity, or implementation of an intervention as it was intended [45,52,53]. The Spumone game was designed primarily as a tool to aid in students’ conceptual understanding and engagement by providing open-ended challenges encouraging the design thinking of engineers-in-training. Thus, it was intended to be an instructional aid. To afford maximum adaptability to integrate the game into their courses, the transportability plan consisted chiefly in training adopting instructors on the Spumone game itself. In retrospect, however, the training lacked sufficient adequacy, important for achieving fidelity. Importantly, students suggested that game challenges were not always well aligned to classroom instruction and assignments, making it difficult to see the conceptual understandings that the game was intended to teach. Adopting instructors reflected that the training needs to be better developed and tested, especially to include applications of Spumone as an instructional tool, including how it can be profitably assigned to aid in the instruction of key course concepts. It was also believed that instructor training or professional development needs to be ongoing to support continuing integration of Spumone into instruction and curriculum. Thus, frequency, duration, and sustainability of instructor training and professional development proved to be important factors to guide improvement of transportability [54,55].

Although the evidence of the quality of the intervention based on previous studies was strong, the study exposed important issues related to the game-based intervention quality for the sake of effective transportability. Complexity and adaptability are two important aspects of intervention quality predicting its success [46]. In emphasizing instructor adaptability in our transportability efforts, we underestimated the complexity of the game. The complexity of Spumone challenges appears to necessitate greater structure and guidance. The problem of game complexity was observed when students were not persisting at game challenges as long as the Spumone creator intended, and, consistent with our content analysis of student comments, were sometimes frustrated. We conclude that the complexity of the game may be too high for effective transportability, such that the game interface needs to be made more intuitive for students.

Adopting instructors believed that a more detailed manual would be needed to implement Spumone as a game-based instructional program with the goal of improving student learning. In turn, many students suggested that both in-class instruction and the current website to support Spumone play was insufficient to complete and learn from the challenges. Some also noted a variety of technological glitches. Our emphasizing adaptability provided adopting instructors with a great deal of freedom. However, it is apparent that sufficient structure and guidance is necessary to effectively scaffold intervention adoption for the instructors and game applications for the students. Scaffolding refers to structures or supports, typically provided by a teacher or more advanced peer, that can aid learners in achieving their learning goals en route to developing a skill or competency, or completing learning tasks, independently [68]. Scaffolding in game-based contexts has been shown to positively impact motivation and learning performance [69]. We believe that the level of freedom supporting adaptability needs to be balanced with necessary scaffolds and instructional supports. For instructors, these supports include instructional manuals and continuing professional development. For students, they include accessible online and/or in-game guidance that is continually updated, as well as adequate technological support.

Our impression was that the implementation climate of the Spumone intervention was conducive to effective transportability. For example, the instructors were selected partly due to their receptivity and their demonstration of knowledge and beliefs compatible with successful adoption of Spumone into their courses. Experimenting with game-based learning was also compatible with the existing norms, values, and systems of the adopting instructors’ institutions. However, we learned that successful transportability of a game-based learning approach to teaching mechanical engineering courses based on a complex technological innovation likely necessitates more effort than a one-time instructor training and implementation of the intervention. Consistent with the literature on implementation science, it has proven to require a process including designing, planning, executing, and reflecting upon the intervention [45,46,48,50].

4.2. Study Limitations and Suggestions for Future Research

We note a variety of study limitations. First, the study attempted to transport the game-based intervention in only two other courses at two institutions, and complete data was available for only one of them. In addition, there were some methodological shortcomings with respect to testing or measuring student gains in engagement and learning. For example, our results showed losses in conceptual understanding from the beginning to the end of one course, which was a very surprising result. However, that course was different than the other courses in that it used a 5-week rather than 15-week format, and it combined dynamics and statics. Relatedly, the instructor of this course used a shorter posttest. Fortunately, we could compensate for this latter shortcoming by measuring gains with items common to both the pretest and posttest across all courses. Nevertheless, these issues highlight the need for consistency of the intervention and reliable measures across years and courses in the future. In future studies, it will be essential for adopting courses to be as similar as possible to the course in which a technological innovation was originally tested, and to each other, in order to reliably test for and establish transportability. Our discussion regarding fidelity issues also demonstrates the importance of intermittent fidelity checks throughout intervention implementation [45,46,52,53].

5. Conclusions

We conclude that a game-based learning intervention to teaching undergraduate engineering dynamics did not produce gains in conceptual learning or student engagement similar to those found in its original implementation upon our attempt to transport it to other courses and instructors. However, the vast majority of student participants reported that their experience with the game-based course was positive overall upon reflection after the course. In this study, we noted a number of implementation issues, especially as related to fidelity, intervention complexity and adaptability, the adequacy of training, and ongoing supports for both instructors and students. With respect to supports, we highlight the critical importance of training that addresses instructional integration of a new technology; ongoing teacher professional development; instructional manuals; adequate in-class, online, and/or in-game guidance for students; and sufficient technological support. Because the overall goal of the game-based intervention was to support and improve student learning and conceptual understanding, an important focus of these supports is helping students to see the connections between the computer-based innovation and the concepts that the game was intended to teach.