Applying the Global Change App in Different Instruction Settings to Foster Climate Change Knowledge among Student Teachers

Abstract

This study focuses on learning with the Global Change app, an interactive tool for fostering climate change knowledge. Numerous studies have contributed to the question on what type of instruction is best to achieve learning gains. The findings are mixed. We applied the app in university courses and investigated which instructional setting a discovery learning approach (no supplementary guidance) or an approach that leans more toward direct instruction is more effective (+ supplementary guidance). Thus, we distinguished between conceptual and procedural guidance within our direct instruction approach. Our study was implemented in a digital learning environment with 110 students participating in the study. We applied a 2 × 2 experimental design with different types of guidance as treatment (conceptual and procedural). An online questionnaire was administered in pretest and posttest to measure climate change knowledge as well as different variables. Our results show that the app provided gains in climate change knowledge in a short period of time regardless of treatment. Further, students who received no supplementary guidance acquired more knowledge about climate change than the groups that received supplemental guidance (either conceptual, procedural, or both). Learning gain correlated significantly negatively with cognitive load across the whole sample, but there were no significant differences between groups. This finding might be interpreted in terms of the renowned expertise reversal effect.

1. Introduction

Climate change is clearly one of the most important issues of the twenty-first century (e.g., [1,2]). Therefore, people need to acquire climate literacy, i.e., an understanding of the impact of climate on oneself and society, and vice versa. Accordingly, combating and dealing with climate change plays a key role both within the 2030 Agenda for Sustainable Development ([3], e.g., SDG 13) as well as within the scope of discussing socio-scientific issues within the classroom (e.g., [4,5]). Both the consideration of climate change in the context of sustainability and in the context of socio-scientific issues are important because there is still not a general consensus that man-made climate change is currently occurring. For example, 31% of U.S. science teachers surveyed in his study said that they explicitly address two positions in class as equal and opposite, on the one hand, that global warming is caused by human activities, and, on the other hand, that there are natural causes for rising temperatures [6]. Meanwhile, there is a clear scientific consensus that anthropogenic causes are the main drivers of climate change and that both aspects are not equally important.

The causes and consequences of climate change are difficult for individuals, especially non-scientists, to understand. This concerns both the role of the ozone hole, ocean acidification, and especially cycling matter (e.g., [7,8,9,10,11]). This is significant because the carbon cycle is an important natural system that requires robust scientific knowledge to understand how and why climate change is occurring [12]. An understanding of carbon cycling and the interrelationship with climate change requires a system-thinking approach, meaning the ability to understand complex interconnections among all of the components of the climate system [13,14]. In particular, it requires the ability to describe the carbon and water cycles, to name pools of the cycles and the flows between pools, and to show how they have changed over time, this is important for participating in the climate change debate and making informed decisions. Thus, tools and approaches are needed to support climate change learning and literacy.

The Global Change app was designed precisely for this purpose [15,16]. It was designed through a collaboration of scientists and digital creatives as an education platform to improve students’ fundamental knowledge of the global carbon and water cycles, and how they are impacted by human activity. The app is designed to be used by a wide range of audiences, from middle school students to adults. However, it is unknown how effective working with an app is in the context of climate change.

In particular, it is not known what kind of instructional framing allows students to make the highest learning gains while working with the app. We know from other settings that both degree and type of instruction play a role in the learning process. Opposites are direct instruction with strong guidance and open-ended, exploratory or discovery learning that uses very little guidance. (e.g., [17,18,19,20], for summary [21]). On the other hand, different levels of effectiveness of content-based, procedural, and/or metacognitive instructional approaches have been documented for learning with digital resources [22]. This study therefore investigates which type of guidance achieves the highest learning success for people with appropriate basic knowledge (student teachers in biology) when working with the Global Change App. Due to the COVID-19 pandemic, the study did not take place in a face-to-face setting, but in a digital distance learning environment, taught with specially produced videos.

2. The Global Change App and the Significance of Climate Change Knowledge

The Global Change app aims to promote climate change knowledge and especially climate change literacy through a digital tool [15,16]. It is an interactive teaching tool that illustrates how the biotic and abiotic systems involved in the carbon and water cycles are connected to the stomata and how human activities affect these processes [16]. Therefore, the app is designed as a compact and easy-to-navigate tool, with a focus on conveying how processes are interconnected rather than describing processes in isolation. For easier handling, information on how to use the app can be found in the User Guide. Technical terms used in the app can be looked up in the glossary. After a short introductory animation video, the user can explore three main pages with three content foci, namely the functional importance of stomata as a connecting element of carbon and water cycle, the cycles in preindustrial and present CO₂ concentrations, and the climate connections, which are presented in the broadest sense as a very complex concept map that addresses interrelationships. In the following, the three pages are discussed in more detail (the app is currently only available in the browser version: https://indd.adobe.com/view/c851bd73-efe2-4294-8717-1ba773af4ef5, accessed on 16 August 2021).

2.1. Stomata

The focus of the stomata page is an explanation of the hunger–thirst dilemma of plants and its linkage to the carbon and water cycles. For this purpose, the page What are stomata? describes how opening and closing of stomata is controlled by different environmental conditions. Firstly, students can interactively explore how much water and carbon enters the plant by visualizing different levels of stomata opening. Thereby, students can see how changes in stomatal opening affect the physiological function of the plant and how these changes can in reverse affect the global carbon and water cycles. Secondly, the impact of carbon and water on the physiological function of stomata is illustrated. For example, interactive diagrams describe the effect of the supply of soil water through the vascular system of the plant and the simultaneous exchange of CO₂ and water vapor from the atmosphere through the stomata. On the How tab of the What are Stomata? page, there is an explanation of the influence of carbon and water on the physiological function of the stomata. This allows the so-called hunger–thirst dilemma to be explored interactively: If plants open the stomata, CO₂ can enter the plant, but at the same time they lose water through transpiration. If they close the stomata, they save water, but not much CO₂ can enter (they become “hungry”). The app’s animation and interactivity show how stomata respond to variable water and carbon availability due to stochastic environmental conditions to maximize CO₂ assimilation for photosynthesis while minimizing water loss through transpiration. Finally, the Why page of the app describes key environmental factors that influence stomatal regulation, for example, how assimilation of CO₂ decreases at both low temperatures and high temperatures. Figure 1 shows an example of the stomata page.

2.2. Carbon Cycle and Water Cycle

The second page deals with cycles, pools, and flows. Here the app provides an overview of the global carbon and water cycles and presents the pools and flows in a preindustrial and in present (today’s) context. It is possible to switch interactively between these two levels (preindustrial and present) and to compare carbon pools and carbon flows and how they are affected by rising CO₂ concentrations, an incidentally important competence of systems thinking [8]. Thinking in pools and flows over time is an issue, where also many misconceptions and difficulties in understanding in the public have been described [23]. The design of the cycles page illustrates the key concepts of each component, how the pools (e.g., atmosphere or ocean) are linked by flows (e.g., diffusion) through the system and processes in a way that can be easily understood by students. By linking the pools and flows, students can track carbon and water between the different system components. This offers conceptual connections that show how human activities affect the different components of the cycles. Flows connect the atmosphere, hydrosphere (oceans and surface waters), biosphere (vegetation and soil), and geosphere (sedimentary rocks, calcium carbonate, deep-sea sediments). In addition, basic physiological topics are covered. The processes of autotrophic and heterotrophic respiration, photosynthesis, decomposition, and soil respiration are explicitly addressed at the ecosystem level in the carbon cycles section of the app using illustrations, text tables, and problem tasks (e.g., “Can it be dangerous to sleep in a room full of plants?”). Figure 1 shows an example of the carbon cycle page.

2.3. Climate Change Connections

The final and most complex section of the app is the climate change connections page. This section examines how human influences, such as CO₂-induced increases in radiative forcing, land use change, and fossil fuel emissions affect the global carbon and hydrological cycles, particularly through changes in plant physiological responses. For example, phenomena such as ocean acidification, the importance of albedo, or permafrost melt are explained in the context of climate change. The ocean acidification node, for example, describes briefly how the oceanic carbon cycle is driven by a continuous series of biogeochemical reactions. Students can learn the complexities of the carbonate buffer system, which absorbs atmospheric CO₂ at the air–sea interface. On this side, different feedback loops, also a competence of systems thinking, can be elaborated, which cause difficulties for teachers and learners alike [23]. For example, when sea ice melts, land or open water takes its place. Both land and open water are on average less reflective than ice and thus absorb more solar radiation. This causes more warming. Another important point can be easily illustrated on the connection page. On the one hand, there are direct effects of increasing atmospheric CO₂ concentrations (e.g., ocean acidification), on the other hand, there are indirect or mediated effects by the temperature increase. For example, higher temperatures lead to more soil activity and thus to more CO₂ production. Figure 1 shows an example of the climate change connection page.

2.4. Climate Change Knowledge and Climate Change Learning

Climate change knowledge is an essential part of climate literacy, which is widely discussed in the media and includes aspects of scientific literacy, environmental literacy, and also media literacy [24]. Within this construct, knowledge and understanding of the essential principles of those aspects of the earth system that determine climate events is crucial and can be seen as one dimension of climate literacy (e.g., [25]). This is also true for climate change literacy, i.e., learners should gain an understanding of the basic science of climate and climate change as one aspect of climate change literacy [26]. According to the U.S. Global Change Research Program, the essentials of climate science present information that is considered important for individuals and communities to know and understand the Earth’s climate, the impacts of climate change, and approaches to adaptation or mitigation [27]. This knowledge dimension of climate change is addressed when postulating that, in order to understand climate change, students must first understand climate as a system [28]. Thus, system thinking is one aspect of climate literacy. As the above illustration elucidates, the app provides knowledge that is necessary to understand climate and climate change as a system [16,26,28]. People who are literate in climate science know that climate science can inform many of our daily decisions (e.g., regarding mobility and nutrition). They have a basic understanding of the climate system, including the natural and human-caused factors that influence it [27]. In this respect, it can be assumed that the Global Change app as an interactive tool can make a significant contribution to climate change knowledge and education [29].

Numerous studies can be found in the literature, including suggestions for content and methods of climate education as well as a variety of learning objectives (e.g., [30,31,32]). While there is now a solid body of evidence on the interplay of important personality traits for climate-protective behaviors (e.g., [33]), findings from efficacy studies on effective pedagogical interventions to promote appropriate personality traits in teaching and learning are sparse (e.g., [34,35,36,37,38,39]). In particular, however, there is a considerable research gap with regard to the effects of using digital media. For this reason, we decided to use the Global Change app in this study and test to what extent this medium can be used to promote climate change knowledge.

3. Instructional Settings and Digital Learning

In this study, we question in which instructional setting the Global Change app is best used. In the following two subsections, we will first present the two main instructional settings, namely a more open-inquiry or discovery approach and direct instruction approaches, as they are essential for the design of this study. In a second step, we present the possibilities of structuring direct instruction in terms of conceptual and procedural guidance, which is the second essential distinction for the design of our study. For the relationship of the terms, please refer to the infographic (Figure 2).

3.1. Instructional Settings

Instruction can be multifaceted [40,41], and the question of what type of instruction leads to the best learning outcomes is still debated. Existing approaches can be broadly categorized in terms of guidance. At one extreme of the continuum there are direct instruction approaches, at the other extreme approaches that allow learners much more freedom in the learning process. More precisely, direct instructional guidance, i.e., instruction in which procedures are directly demonstrated to learners, can be contrasted with inquiry-based instruction, in which a procedure is not demonstrated to learners but be discovered [42]. Inquiry learning is closely related to the term discovery learning, which appears more frequently in multimedia literature and is often used synonymously. Discovery learning as instruction asks students to navigate a rich but relatively unstructured learning environment and to learn with minimal instructional support by discovering some or all of the essential information and skills to be learned by discovering or constructing them [43]. We favor the term discovery learning in the study because it fits better to digital learning with the medium app.

With respect to an educational use of the Global Change app, these two contrasting instructional approaches can be applied as follows. From a discovery learning perspective, teacher students of biology would be expected to derive the greatest benefit from working with the app if they are allowed to work with it in a self-determined and self-directed manner and are only given minimal, structuring work assignments at the beginning (for summary, [21]) (note: in our case group D, see Section 5.3) this means the students were given motivation and a work assignment at the beginning, but did not receive supplementary guidance going beyond the app-immanent explanations (please refer to

Table 1. Teaching sequence (see Table 2 for a more detailed description of step 2 “types of instructional guidance”).

Step	Medium	Description	Time (Minutes)
Pretest (online: SoSci Survey)
1a	Video	Motivation: R.A. and N.T. briefly introduce themselves and invite the students to the digital seminar. Then, the importance of the subject matter (Global Climate Change) and its importance for students and the society was briefly explained.	3:09 min
1b		Information about the goal of the teaching unit (R.A. and N.T.): In this module, the goals of the work with the Global Change app were explained: to deepen basic biological processes that are a prerequisite for participating in climate discussions (e.g., knowledge of stomata or carbon pools and carbon fluxes), to sharpen the view of global connections, and to explain causes and mechanisms that lead to climate change.
1c		Activation of prior knowledge: Before working with the app, students are asked to consider what they already know about the following phenomena: Climate Change, Carbon Cycle, Water Cycle, and the importance of plants to the cycles and climate change.
2	Video	Types of instructional guidance	see Table 2
3 *	Video	Work Assignment: “Over the next 50 min, use the Global Change app to elaborate on the scientific basis of climate change. To do this, you are to examine the impact of rising carbon dioxide concentrations and rising temperatures at the different scales. To structure your thoughts, take notes on the 3 different levels of the app. Focus on the key messages and central concepts. In the second level, focus especially on the carbon cycle and less on the water cycle, as the latter is more important for biology classes.”	1:10 min
4	App	Working with the Global Change app	50 min
Posttest (online: SoSci Survey)
5	Zoom	Feedback to the respective group (University of Münster, Freiburg University of Education) as Zoom-Conference (one week after the posttest)

* It is important to mention beforehand that across all groups, time spent using the app averaged 55.9 min (the work assignment was 50 min) and that there were no significant differences in self-reported time spent working with the app between groups (A, B, C, D), which is why we did not further control for this factor (ANOVA: F_3,109 = 1.15, p = 0.33).

for more details and the procedure of the teaching sequence). Greater learning gains might be achieved because basic needs for experiencing competence, autonomy, and social inclusion are of central importance to the learning process [44]. Thus, higher motivation should lead to greater learning gains. On the other hand, independent exploration of the app could be expected to lead to a high cognitive load (overloading of the cognitive working memory) that might inhibit learning (e.g., [45,46]), because students not only have to extract the information from the app (and try to understand the concepts), but also have to determine their own learning path. Moreover, when using the Global Change app with non-native English learners, there is also the additional burden of working with a foreign language app. Thus, direct instruction might be advantageous [17] (note: in our case group A, B, and C, see Section 5.3) this means that students receive supplementary guidance in addition to a motivation, work assignment and app-immanent explanations (please refer to Table 1 for more details and the procedure of the teaching sequence). In brief, the degree of instructional guidance is hypothesized to influence learning gains when working with the Global Change app.

However, it is not so easy to predict when exactly direct instruction and when discovery learning approaches are advantageous. Proponents of direct instruction and discovery learning each claim that their method is most effective under certain circumstances, depending on the domain, the students, the level of expertise of the learners, and other factors such as the time available in the classroom [47,48]. Full and explicit instructional guidance has proven to be very successful especially with novice to intermediate learners [49]. However, the results on the superiority of one of these approaches are not clear. On the one hand, representatives of instructional approaches to teaching and learning emphasize—based on findings from expertise research—the role of subject knowledge [50,51], which can be acquired effectively in more direct instruction [50,51]. Results from the PISA study also correspond very well to this [52]. Here, it was shown that students who reported that their teacher explains scientific ideas “in many lessons” or “in every lesson” perform significantly better in science than students who report that a whole-class discussion takes place “in many lessons” or “in every lesson”. On the other hand, there is research showing that direct instruction can hinder learning. In particular, this is the case when people with a high level of prior knowledge and expertise are instructed in too much detail. This relationship is referred to in the literature as the expertise reversal effect [45,46].

There are a number of factors that might explain why it is not easy to use instructional explanations successfully [20], or why their effectiveness is conditioned by particular instructional contexts [53]. The same applies to the discovery learning approach, which has been shown to be very effective in numerous meta-analyses, but not every type of inquiry teaching is equally effective [54]. As a matter of consequence, it is not possible to say in principle that direct instruction is better than more open formats such as inquiry teaching [54]. In addition to the learners’ prior knowledge, a number of other factors also play a role in the success of the instruction, for example the pedagogical content knowledge of the teacher, but also characteristics such as cognitive abilities, personality traits, learning styles, interests, and motivational states, which can be considered when adapting explanations to the individual learner (for summary [20]).

3.2. Direct Instruction in Digital Learning through Conceptual and Procedural Guidance

In the context of learning with digital media, specifically, it can be assumed that instruction plays a central role for learning especially when it comes to complex scientific content such as global climate change. For example, studies have shown that learning gains were higher when students received content-related instruction in the use of digital media such as computer simulations or mobile plant learning systems (e.g., [55,56]). However, little is known about how exactly such content-based instruction should be structured for science content in digital learning environments. Explanation videos can be regarded as a special form of content-based instruction through videos (e.g., [57]) and can be defined as self-produced, short films in which content, concepts, and contexts are explained (explanation videos in the narrower sense) or in which activities and processes are demonstrated and commented (tutorials, explanation videos in the broader sense). Both explanation videos in the narrower and broader sense pursue the intention of achieving an understanding in the viewer or triggering a learning process [58]. Research on explanation videos, for example, has shown that different types of instruction, e.g., low- and high-quality content-based instruction (the authors used a framework for effective explanation videos: high-quality video fits better to the model than the low-quality video), or instruction with the use of pedagogical agents, differ in their effectiveness (e.g., [22,59]). In summary, content-based instruction through (explanation) videos is one way to support learning with digital media.

Besides content-based instruction, there are a number of other ways to support learning with digital media. In this respect, it is possible to distinguish different subtypes of instructional strategies within direct instruction when working with mobile technology (location, procedural, and metacognitive guidance) [60]. Accordingly, in the context of computer-based scaffolding, other authors define different types of scaffolding strategies, i.e., strategic, metacognitive, or conceptual scaffolding [61]. One difficulty in distinguishing between different types of instruction is that while the terms are similar and used in various studies, they are not always used with the same meaning (e.g., procedural scaffolding/guidance is used differently [60,62]). In this study, we adapt an older but comparatively concise definition for procedural scaffolding and conceptual scaffolding that originated in the field of scaffolding [63], but use the term guidance instead to point out that scaffolding outlines even broader facets than our form of guidance [64,65].

We define procedural guidance as support that emphasizes how to use the available resources and tools. It is oriented to the features and functions of the system and helps learners navigate an open learning environment (OLE). For example, some learners get lost in OLEs; procedural guidance is common to clarify how to return to a desired location.

We define conceptual guidance as support that tells learners where to focus their attention. Sometimes this is accomplished by identifying key conceptual knowledge related to a problem or by creating structures that make conceptual orientation readily apparent (e.g., by providing structure maps and content trees that give explicit cues to students).

More detailed explanations of the implementation related to the work with the Global Change App can be found in Section 5.3. Additionally we illustrated how our two types of guidance, procedural and conceptual, fit into the two types of instructional settings, namely discovery learning and direct instruction. (Figure 2).

4. Research Questions

The aim of this study is to investigate the role of instructional guidance in the acquisition of climate change knowledge using the Global Change app, as an example for gaining complex, interconnected subject content via an interactive digital tool. Our research questions are as follows:

• To what extent is the Global Change app a useful medium for gaining climate change knowledge in general?
• What are the effects of different types of guidance—no supplementary guidance, conceptual guidance, procedural guidance, or a combination of both—on climate change knowledge when working with the Global Change app?
• In particular (within the direct instruction setting): Are there differences in knowledge gains between two types of instructional guidance (procedural guidance vs. conceptual guidance)?
• Which differences are there in knowledge gains between instructional settings, a more open-ended, discovery learning (no supplementary guidance) approach or an instructional setting with a more direct instruction approach (conceptual guidance, procedural guidance, or a combination of both)?
• To what extent do the groups (no additional guidance, conceptual guidance, procedural guidance, or a combination of both) differ on three variables influencing digital learning, i.e., cognitive load, intrinsic motivation, and utility value?

5. Methods

5.1. Sample

A total of 110 pre-service biology teachers (63 students from the University of Muenster: 78% females and 22% males; 47 students from the Freiburg University of Education: 89% females, 11% males; Ø age: 23.76 years) participated in the study. The students from the University of Münster and the Freiburg University of Education are pre-service biology teachers in the master’s program, but study a different focus (Freiburg: lower secondary level: grades 5–10, Münster: higher secondary level: grades 5–12 (13)). Students at both locations, however, have completed a bachelor’s degree in biology and thus have prior knowledge of plant physiology and ecology that is useful for understanding the Global Change app. In order to characterize students in terms of their attitudes and relevant thinking skills related to climate change, two instruments were used in the pretest which are shown in

Table 3. Measures in this study.

Scale	∑ Items	Cronbachs α (Pretest)	Cronbachs α (Posttest)	Min.–Max. (Theoretical Scale Mean)
Climate change knowledge	20	0.583	0.711	0–5 (2.5)
System Thinking revised [71]	10	0.586	-	1–7 (4)
Global warming beliefs [72]	6	0.719	-	1–6 (3.5)
Cognitive load [73] *	4	-	0.706	1–7 (4)
Short-scale intrinsic motivation * [74]
interest/enjoyment	3	-	0.775	1–5 (3)
perceived competence	3	-	0.838	1–5 (3)
perceived choice	3	-	0.791	1–5 (3)
pressure/tension	3	-	0.741	1–5 (3)
Utility value [75]	3	-	no scale	1–7 (4)

* Items are provided in Appendix A.

. It can be stated that, on the one hand, our sample was convinced that man-made climate change exists (5.48 ± 0.48, theoretical scale mean: 3.5, see Table 3), on the other hand, they showed an above-average competence in systems thinking (5.79 ± 0.54, theoretical scale mean: 4.0, see Table 3).

5.2. Teaching Sequence and Digital Learning Environment

Due to the COVID-19 pandemic, at both participating universities, the courses in which our study were conducted were taught in a digital rather than face-to-face learning environment. For this reason, we videographed the elements of our teaching sequence, details are shown in Table 1. The sequence was the same for all students, only step 2 (instructional guidance) was varied, this step is specified in

Table 2. Design of the study: types of instructional guidance. Note: groups A, B, C: with supplementary guidance (in brackets), group D: no supplementary guidance. N: number of students in the respective group.

		ConceptualGuidance(CG) (8:47 min)
		+	−
Procedural Guidance (PG) (8:49 min)	+	Group A	Group B
		(PG + CG)	(PG)
		(N = 25)	(N = 30)
	−	Group C	Group D
		(CG)	(none)
		(N = 28)	(N = 27)

. The pretest was administered one week before, and the posttest was administered immediately after working with the app. We decided to always show the same speakers in the videos, namely one person from the Münster University (R.A.) and one person from the Freiburg University (N.T.). This ensured that students from each location were not solely confronted with a person they did not know, which could have potentially led to confounding effects. The videos and other necessary information were administrated to the students on the respective learning platforms of the two universities Muenster and Freiburg.

5.3. Study Design

An experimental 2 × 2 design, shown in Table 2, was used in this study to investigate learning outcomes with the Global Change app comparing different forms of instructional guidance, systematically varying conceptual guidance (herein after referred as CG) and procedural guidance (herein after referred as PG). Thus, each group received app-immanent instruction just by working with the app (note: any carefully structured learning environment—and we may assume that the Global Change app is one such—has some degree of conceptual and procedural guidance; thus we speak of app-immanent), but in group A, B, and C supplementary instructional guidance was added. Students at both locations were randomly assigned in advance to one of four groups.

As the literature lacks clear statements on the design of instructional videos, we based the design of our procedural and, in particular, guidance instruction videos on the explanation videos and the procedural and conceptual scaffolding definitions described above (Section 3). Specifically, in terms of conceptual guidance, we described which scientific concepts are addressed on each page while procedural guidance helped navigating each page. For example, within conceptual guidance on the C-cycle page, we pointed out that when considering the carbon cycle in the context of global change, the comparison of preindustrial and present CO₂ concentration in the atmosphere is central. Within procedural guidance, at the same point, we just said that there are two buttons on the page which allow switching between “preindustrial” and “present”. We addressed thus the technical interfaces, the navigation, and especially the interactivity (Where to click? Where to get help? How to get back?). The scientific aspects or terminology were not described. An example of the storyboard can be seen in Figure 3.

5.4. Operationalization of Climate Change Knowledge

As presented in the theoretical section, climate (change) literacy encompasses several facets. In a nutshell, climate change literacy includes (1) knowledge of climate system science, (2) understanding of the impacts and threats of climate change, and (3) motivation to make informed decisions to implement mitigative and adaptive solutions to the climate crisis [26]. In this respect, the Global Change app focuses on the first facet—knowledge of the climate system science—in order to understand climate and climate change [15,16].

For our pre–post intervention study, we needed a matching and change-sensitive measure to detect possible effects of our different types of guidance on climate change knowledge. Existing instruments for measuring climate literacy (e.g., [66,67]) are broader in scope and only capture the content of the Global Change app to a very limited extent. Thus, we specifically developed a multiple-choice instrument to test climate change knowledge related to the Global Change app. In line with the different pages of the app, please see Figure 1, a total of 23 items were developed the following the three conceptual foci of the app (see chapter 2): (1) Stomata (4 Items), (2) Cycles and Global Change (comparison of preindustrial vs. present context with special focus on the carbon cycle) (13 items), and (3) Global Change Connections (6 items). To be more precise, a total of three persons (S.L., L.D., and R.A.) examined the content of the app step-by-step and constructed items for the respective content. This provided face validity and ensured that the questionnaire really addressed the app content. An example item for each conceptual focus is given in Figure 4, Appendix B contains the entire instrument. The multiple-choice items were items with multiple answer questions [68].

Each correct answering option (each correct one that was ticked by students and each incorrect one that was not ticked) was scored with one point. Thus, 0 or 5 points (for each individual answer option) could be scored per item. We refrained from scoring the item as correct (entire item correctly solved: 1 point) only if all answer options were correctly ticked. The reasons for this were first that the single items were (completely) solved rather low (in the posttest: Min.: 0.00%, Max: 90.9%; on average, 40.4% of the items were solved completely correctly) and second that it was not possible to form a reliable scale.

We used Cronbach’s α to determine the internal consistency (reliability) of the scale. After excluding 3 items (from an originally 23-item scale) for reasons of low item discriminatory power (r_it ≤ 0.20) from further analysis as recommended by Urbina [69], we were left with a scale with 20 items. The scale showed a good reliability in the posttest (Cronbach’s α = 0.71), but this was weaker in the pretest (Cronbach’s α = 0.58). It can be assumed that parts of the topics addressed in the Global Change app were new to the students in many cases. Therefore, the items in the pretest were answered more heterogeneously (no homogeneous knowledge of the students) and thus the internal consistency was reduced. Within the 20 items, no interpretable structure was revealed by factor analysis; thus we suppose that we are measuring a one-dimensional construct. In this respect, the lower Cronbach’s α value in the pretest seems justifiable [70].

5.5. Additional Measures

To better characterize our sample, we recorded participants’ systems thinking expertise [71] and climate change beliefs [72] in the pretest. Two putative covariates were recorded in the posttest, cognitive load [73] and intrinsic motivation [74]. We also included a measure for utility value [75], first in terms of whether the app is helpful in acquiring climate change knowledge, and second in terms of the instructional videos’ guidance and utility.

Table 3 gives an overview of all measures and their characteristic values measured in this study. In addition to these variables, other descriptive variables such as location of study (Muenster or Freiburg), gender, subject combination in studies, self-reported English skills, age, and effective working time with the Global Change app were assessed.

5.6. Statistical Analyses

There were no significant differences between the two groups (location Muenster and Freiburg) in terms of climate change knowledge; therefore we pooled the two groups and did not differentiate the study location in the following analyses. Likewise, there were no differences in gender with respect to climate change knowledge, so we did not consider gender in the further analyses, this was also true for other factors: the subject combination studied and reported English skills. Therefore, these factors were not considered in the further analysis.

To answer the research questions, we conducted a repeated measures ANOVA in an initial analysis. The dependent variable was climate change knowledge (pre and posttest). Two factors were used as independent variables according to the 2 × 2 design (factor 1: conceptual guidance: yes or no, factor 2: procedural guidance: yes or no, see Table 2).

In a second analysis, we included cognitive load and motivation as covariates in the model (repeated measures ANCOVA). Including covariates allows modeling and adjustment for input variables that were collected in the experiment but not randomized or controlled. Thus, adding covariates can increase the accuracy of the model by reducing the error in the model and thereby increasing the discriminatory power of factor tests. We used two covariates, cognitive load and intrinsic motivation. The latter had no effect on the model parameters (and was not included in the final model), but cognitive load did, so it was included as a covariate in our final model reported below.

The assumptions of the ANCOVA were fulfilled, normality: Shapiro–Wilk test, heterogeneity of variances: Levene’s test (in each calculation). When descriptive values are reported in the following, arithmetic mean and standard deviation are always given (M ± SD).

6. Results

6.1. Gaining Climate Change Knowledge (RQ 1)

Table 4. Results of repeated measures ANOVAs on the score climate change knowledge. A: without covariate, B: including covariate, PD: procedural guidance, CG: Conceptual guidance, η²: effect size: η² = 0.01 indicates a small effect; η² = 0.06 indicates a medium effect; η² = 0.14 indicates a large effect (Cohen 1988).

	A Without Covariate				B Including Covariate
	df	F	p	η2	df	F	p	η2
Source of variation (within subjects)
Time	1	304.7	0.000	0.74	1	54.2	0.000	0.34
Time x Cognitive Load	-	-	-		1	4.20	0.043	0.038
Time × PD	1	0.78	0.380	0.01	1	0.48	0.492	0.01
Time × CG	1	0.37	0.544	0.00	1	0.73	0.395	0.01
Time × PG × CG	1	3.35	0.070	0.03	1	4.15	0.044	0.038
Error	106				105

shows the final repeated measure ANOVA and repeated measure ANCOVA results with type of guidance as fixed factors and climate chance knowledge as dependent variable, once without covariate (A), once with covariate (B). It can be seen that the inclusion of cognitive load as a covariate leads to a model with a higher discriminatory power. In the initial analysis (without covariate) the interaction “Time × PG × CG” showed only a trend (p = 0.070), by including the covariate this trend becomes significant. Consequently, we refer to the model with covariate (B) in the following.

Regardless of the inclusion of the covariate, students gained significant knowledge about climate change in a comparison of the pre and posttest. This is expressed across all groups by a large effect over time (F_1,106 = 304.7, p < 0.0001), i.e., students had more knowledge in the posttest (3.93 ± 0.35) than in the pretest (3.41 ± 0.27). Results are shown in Table 4 and Figure 5.

6.2. Effects of Different Types of Guidance (RQ 2)

With regard to our second research question, it can be stated that the two factors (factor 1: conceptual guidance (CG): yes or no, factor 2: procedural guidance (PC): yes or no, see Table 2) have no influence on learning gains (Time × CG; Time x PD, see Table 4). The covariate cognitive load proved to be significant (F_1,105 = 4.2, p = 0.043, partial η² = 0.038). That is, students who perceived lower cognitive load performed better (small effect) than students who reported perceiving high cognitive load. In other words, a higher perception of cognitive load resulted in lower learning gains. However, as stated before, we did not find effects of different types of guidance on climate change knowledge when working with the Global Change app.

6.3. Effects of Instructional Settings (RQ 3)

With regard to RQ 3, we found a significant interaction time × PG × CG (F_1,105 = 4.2, p = 0.0044, partial η² = 0.038, thus not all groups (“type of instructional guidance”) responded consistently over time. In particular, the group with no supplementary guidance performed better (small effect) than the pooled groups with supplemental instruction (pretest–posttest, no supplementary guidance (D): 3.46 ± 0.29–4.04 ± 0.34, pretest–posttest, with supplemental guidance (A, B, C): 3.57 ± 0.33–4.03 ± 0.35). The results are illustrated in Table 4 and Figure 5. Therefore, we can conclude here that a more open, discovery-based learning (without additional guidance) is somewhat more effective than a more direct instruction when working with the Global Change app in our setting.

6.4. Additional Measures (Cognitive Load, Intrinsic Motivation, Utlity Value, RQ4)

6.4.1. Cognitive Load

Due to the high subject matter complexity, we investigated the level of cognitive load perceived by all students and compared cognitive load within different groups. Averaged across all four groups (A, B, C, and D), the students’ perceived cognitive load was reported to be slightly below average (3.45 ± 1.13), thus moderate to rather low. Descriptively, cognitive load was highest for group C (+ content guidance) and lowest for B (+ procedural guidance). However, these differences were not significant (ANOVA: F_3,109 = 1.22, p = 0.31). However, we found a small negative correlation between cognitive load and pre–post-learning gains across all groups (Pearson, r = −0.188, p = 0.49). That is, regardless of the group (A, B, C, and D), there are individual differences in the sample, when cognitive load is perceived to be higher, learning gains are lower and vice versa.

6.4.2. Intrinsic Motivation

No significant differences were found within the respective dimensions between the type of instructional guidance (A, B, C, D, see Table 2). The app was rated as interesting on average (3.60 ± 0.67), and it was perceived as promoting competency, i.e., students were basically satisfied with their performance (3.26 ± 0.70). The choice, i.e., the autonomous selection of the contents of the app, was rated as high (4.21 ± 0.67) and highest, but not significantly different in the group D, which had no supplementary guidance (4.38 ± 0.53). When working with the app, the students felt relatively little pressure (2.16 ± 0.84).

6.4.3. Utility Value

Table 5. Utility value (mean ± SD), n.s.: not significant, *** p < 0.001, Bold: these groups had the instruction addressed in the respective question, group D: no supplementary guidance.

Item Wording/Group	Group A + PG + CG	Group B + PG	Group C + CG	Group D	Statistics (Between Groups)
(1) How useful did you find the Global Change app in understanding “Climate Change”.	4.84 ± 1.28	5.38 ± 0.98	5.14 ± 1.32	4.93 ± 1.00	F3,109 = 1.22 n.s.
(2) How useful did you find the previous instructional videos for using the Global Change app?	5.60 ± 1.12	5.24 ± 1.33	4.68 ± 1.588	3.96 ± 1.34	F3,107 = 7.17 ***
(3) How useful did you find the previous instructional videos in understanding the content of the Global Change app?	5.44± 1.73	4.69 ± 1.78	5.18 ± 1.63	3.65 ± 1.38	F3,107 = 5.98 ***

summarizes the utility value questions used. The three questions distinguish between the usefulness of the app and of the instructional videos. The app was generally found to be useful for acquiring climate literacy, with no differences between groups (1). However, differences are evident when evaluating the instructional videos. As expected, groups that received procedural instruction (groups A and B) express higher utility in terms of using the app (2) and groups that received content instruction (groups A and C) express higher utility in terms of understanding the content (3). Overall, those who did not receive any supplementary guidance (i.e., only the motivation video, see Table 1) rated the utility of the guidance lowest.

7. Discussion

In this study, we investigated knowledge gains applying the Global Change app in a university setting (RQ 1) and specifically questioned which type of instructional guidance is most effective when working with an app that addresses a complex biological topic, namely climate change (RQ 2), and in how far applying the Global Change app in a more open-ended instructional setting or a more directive instructional setting is more beneficial to learning for biology students (RQ 3). In the following, we first summarize and interpret the results to these main questions. Then we present overarching reflections on the results of our study relevant to all research questions.

In relation to RQ 1, the results of this study show that the Global Change app is suitable to foster knowledge gains about global climate change in a relatively short period of time and is thus also suitable as a tool to support science teaching [15]. The question to be answered is how to structure the specific application of the app in a learning setting. The present study provides first results in this regard.

Based on the literature on learning with digital media [60,61,63] we identified conceptual and procedural guidance as important for working with the Global Change app and systematically varied these two factors (RQ 2). However, in the data of our study, we did not find differences in learning gains due to these factors. Neither conceptual guidance explaining the main concepts nor procedural support for operating the app interface resulted in higher learning success. There are examples in the literature that procedural scaffolding is beneficial for learning [62]; however, the opposite effect is also shown [59], in which a procedural scaffold can lead to lower learning performance. Among possible reasons, cognitive overload is one reason often discussed [59] (see in more detail below). Similarly, our conceptual guidance has not been effective. This finding is in contrast to another study in the context of climate change knowledge [5], that shows that it can be beneficial to support students in learning with the relevant concepts, as this ensures the highest learning success [5]. However, this study did not examine the work with an app, but with a concept mapping tool.

With regard to RQ 3, we assumed that supplementary guidance might in principle reduce the cognitive load and could lead to higher knowledge gains than a more open discovery learning setting [17]. To the contrary, in our sample, we found that the group that did not experience supplementary guidance performed better in gaining climate change knowledge than the groups with supplementary guidance, though the observed effect was small. We discuss this finding in terms of expertise reversal effect [47,48] (see Section 2) and cognitive load below.

In summary, the results are more in line with the research direction that postulates a superiority of open-ended, discovery-oriented procedures than with direct instruction approaches when working with learners who are not novices and already have a higher level of expertise. Possible explanations for these main results fall along three directions: (a) increase of cognitive load through supplementary guidance, (b) redundancy of guidance due to the level of expertise of the sample (expertise reversal effect), or (c) insufficient quality of the instructional videos. These ideas are further elaborated below.

7.1. Cognitive Load

We did not find any systematic group differences in cognitive load, which was generally rated as moderate to low by all participants. According to the cognitive load theory [42,76], higher cognitive load leads to lower learning gains. In fact, we find exactly this relation within our data as well (across the entire sample). It would therefore be conceivable that an increase in cognitive load due to the supplementary instructional guidance would lead to lower learning performance. Such a finding is described, for example, by [59]. The authors investigated the influence of a virtual human hand (demonstrating and modeling) in an instructional video about friction on inclined planes in the context of animated pedagogical agents. In the control group, explanation occurred without a virtual human hand. Results showed increased learning benefits for participants in the control condition. The authors attributed their findings to cognitive load theory, namely that learners can only process so much information at once and exceeding this cognitive capacity can inhibit learning [42,77], which was the case with the animated human hand as participants were distracted. Interestingly, the animated hand was subjectively rated as very valuable by the experimental group (because the video with the hand was more humanlike and engaging).

However, the data of the present study do not reveal an increase in cognitive load due to supplemental guidance. Although we found a significant correlation between learning gain and cognitive load, we did not find significant differences in cognitive load between the four groups. In particular, the group with the dual instruction (conceptual + procedural guidance) did not show significantly increased scores compared to the group without supplemental guidance. Both our conceptual and procedural guidance thus apparently did not help reduce cognitive load, but cognitive load was not increased either. Comparable to the study with the virtual hand (see above [59]), our videos were rated as very helpful by the respective groups. In contrast to the study with the virtual hand, however, they did not lead to an increase in cognitive load. Thus, our results cannot be directly explained by the cognitive load theory.

7.2. Expertise Reversal

Expertise reversal is a well-known effect related to instructional effectiveness. It states that individuals with high expertise tend to benefit less from additional instructional interventions than novices (see Section 3). In our study, there are several indicators that the level of expertise of the master’s students studied tended to be high in this regard. On the one hand, the pretest scores show comparatively sophisticated beliefs about global warming even before the intervention, as well as good systems thinking skills. On the other hand, the pretest scores in the knowledge measure show good existing prior knowledge (the scores achieved in the pretest are significantly higher than the guessing probability for our multiple-choice questions, which is 2.5 points per item). Additionally, the scores regarding the cognitive load of working with the Global Change App are—across all groups—rather low. It therefore seems plausible that that our results can be attributed to an expertise reversal effect [45,46], i.e., that the participants did not benefit from the supplementary guidance, even if they subjectively perceived it that way (utility value). To test this expertise reversal hypothesis it would be useful to apply the instructional videos in a follow-up study with novices (e.g., with high school students). It might be assumed that novices benefit more from our supplementary guidance due to lower prior knowledge.

7.3. Quality of Supplementary Guidance

A third potential reason for the finding that students show better learning performance without further instruction could be the quality of our videos, because finally our results show that the videos we produced for conceptual and procedural guidance were not as helpful to students as we intended. One reason could be that our guidance was quite long overall, longer than recommended in the literature for explanation videos [77]. Although too detailed instructional guidance can lead to fatigue and mental saturation [78] and thus higher cognitive load, the perceived cognitive load of our sample did not—as said before—differ between groups, so the length of the videos does not seem to be the determining criterion here. Of course, we cannot conclude that supplemental instruction leads to lower learning gains in general when working with the Global Change app, precisely because we found only a small statistical effect. With more focused and parsimonious guidance, we cannot rule out that our sample would have benefited as well.

Another point is that we lacked clear instructions on how to design the videos; although we referred to the literature for the definition of conceptual and procedural guidance, there was understandably no specific instruction given on how to design the videos [63]. We find evidence in the data that our sample fails to process the content from the videos. This is consistent with research on instructional explanations [20], namely that students often do not learn through instructional explanations (even in the special case of explainer videos). In this context, it is interesting to note that in our conceptual guidance video, the solution for three items of the climate change knowledge questionnaire was in principle already given. However, these three items were not solved better by the two groups with conceptual guidance than by the other two groups. For example, in the cycles, the three largest C pools were explicitly mentioned in the video, but the item on this question was not solved better by the students with conceptual guidance than by the other groups.

It is possible that students in our setting had difficulties correctly implementing the additional instruction when working with the app; this is true for both conceptual and procedural instruction. In this regard, we may also have paid too little attention to elements of scaffolding such as asking questions, hinting, prompting, or simplifying problems [61,79], especially in the conceptual instruction. Our conceptual and procedural instructions were neither explanatory videos in the strict sense [22], nor instructional explanations in the narrower sense [20], but most certainly represent a kind of scaffolding. For example, we provided students with explicit hints, but did not provide structure maps and content trees (see Section 3) [63]. It is possible, nevertheless, that our procedural guidance is simply redundant for students familiar with mobile devices, and that conceptual guidance is too unspecific. For example, it might be more effective to incorporate tasks that give students step-by-step solutions as they work with the app [21] rather than showing a longer video before working with the app.

8. Limitations

There are at least three methodological limitations to our study. First, due to our 2 × 2 design systematically varying two video modules (conceptual guidance, procedural guidance) it was not possible to control for time of the respective guidance. Although we designed and optimized the two modules so that their length was quite comparable at 8:49 min vs. 8:47 min, the dual guidance group in particular received significantly longer instruction while the group without supplementary guidance received shorter instruction. Although not statistically evident, there may have been slight confounding effects here with respect to interest and motivation, possibly also cognitive load. Second, in a digital environment, we could not fully control the use of the videos or the work with the app. However, students had to state their working time with the app which was similar in all groups. Additionally, after the posttest, we asked each of the students if they really wanted their data to be included in the analysis to make sure they worked diligently and accurately. A total of 93.3% of the students answered “yes.” Feedback in the final zoom sessions was also quite positive, therefore, there is no evidence that the students have not worked thoroughly. Lastly, the sample under investigation was relatively small. We see in the empirical data that there are no major group differences in climate change knowledge. It could be that there are small significant differences if we look at a larger sample. To confirm a small difference, even with a power of 0.80, we would need a sample of n = 200 (2 groups) or n = 300 (4 groups) of students (results of a post-calculated power analysis).

9. Conclusions

In short, within a digital learning environment, we examined different instructional settings when working with an innovative digital tool (i.e., the Global Change app) in terms of learning gains in climate change knowledge and found an advantage of the more open-ended discovery setting compared to the more directive settings with supplementary guidance. The following can be concluded for working with the Global Change app, namely that with learning groups comparable to ours, the app should be used more without previous video guidance. The data suggest that the level of competence of our students is causal for this. For the use of instructional measures in the context of (digital) media, this means that the level of expertise should definitely be taken into account: groups with higher expertise might tend to benefit from less instruction, novices from more instruction. In terms of the transferability of our findings, it can be assumed that they can be applied to other digital tools as well. It can be concluded that learning groups with a good level of prior knowledge are able to work out even complex issues independently in a relatively short time within the framework of discovery learning. However, it is conceivable that the effects could be different for other samples (students of non-science subjects, students with lower prior knowledge, or other nationalities), in this respect the study would have to be replicated with a similar design but different samples.

A final point concerns implication for instructional design when using the Global Change app or other apps. We were directed by explanation videos in our study design, but found as we progressed that the advanced organizer or scaffolding might have been more appropriate [61,80]. It should be investigated which results in terms of learning success are provided by instructions that are more oriented toward the advanced organizer or even more in terms of scaffolding that integrates the stimuli relevant to learning into the learning process.

Overall, the study contributes to the open discovery learning versus direct instruction debate and suggests that learner proficiency levels should not be neglected.