MAD+. Introducing Misconceptions in the Temporal Analysis of the Mathematical Modelling Process of a Fermi Problem

Abstract

This work describes how the combination of the mistakes committed by a group of pre-service teachers when solving a Fermi problem, with the representation of the temporal analysis of their resolutions, can offer more in-depth information about their conceptual misconceptions regarding mathematical and modelling concepts. The combined representation allows knowing when mistakes occur and provides a powerful tool for instructors to adapt the teaching–learning processes of mathematics at all levels of education. Our study is based on a recent categorisation of students’ mistakes, together with the creation of a new representation tool, called MAD+, that can combine all this information. The macroscopic view provided by the MAD+ diagrams gives insight into the context in which the mistakes take place and makes the analysis of the resolution of a Fermi problem more efficient.

1. Introduction and Theoretical Framework

In recent years there has been an increasing interest in the introduction of real context and modelling activities in training at all educational levels [1]. There is widespread agreement in the international education community that applied problems and mathematical modelling play an important role in education at all levels. The inclusion of real-world problems in mathematics education has proven to be an effective tool in the education of future citizens [2]. However, behind this consensus there are also many discussions on how to integrate mathematical modelling into mathematics teaching and learning processes. There is still a lack of research and evidence on the effects of integrating modelling tasks in schools, although some medium and large-scale projects have already been carried out [3,4,5,6,7,8]. Rita Borromeo, in her book Learning how to teach mathematical modeling in school and teacher education [9] shows how important it is for future primary school teachers to learn how to teach mathematical modelling. This process involves the acquisition of modelling skills and their associated teaching skills. However, these competencies cannot be acquired if the associated mathematical competencies are not achieved. That is to say, those teachers who have conceptual and procedural mathematical errors will see their ability to apply modelling in their future classes diminished.

Mathematical modelling research studies show that, although there is consensus that modelling processes have a cyclical character, this way of representing these processes do not explicitly show how students perform their work while solving a modelling task [10,11,12]. In particular, Ärlebäck’s Modelling Activity Diagrams (MAD) [13] are representations that provide a view of the macroscopic behaviour displayed by learners when solving a Fermi problem. Ärlebäck also states that the cyclic nature of the modelling process is somehow artificial and by using his representation the complexity when modelling can be simplified.

The process of solving a mathematical modelling task can be represented by different modelling cycles and some of them include representations of students’ cognitive processes [14,15]. However, this information may not be sufficient to observe all the decisions and mistakes that take place when solving a modelling task. To overcome this limitation, in this paper we want to use a temporal analysis of students’ resolutions when solving a Fermi problem paying special attention to the mistakes students make during its resolution process and to the moment in which they occur.

1.1. Estimation, Mathematical Modelling and Fermi Problems

Hogan and Brezinski [16] refer to numerosity as the estimation of magnitudes and they state that this involves a single estimation skill. They claim that measurement estimation should be approached as an independent skill as it does not automatically follow from general mathematical development. However, Albarracín and Gorgorio [17] stated that another type of estimation activity is the calculation of quantities in predictive or estimation activities using models that represent the real situation. The authors defend that, in the absence of a specific procedure to determine the exact value of a quantity, the situation can be simplified to an equivalent one that represents it. This may be the case when estimating the number of objects in a container. In addition, there are real-life situations that do not require an exact quantity to be estimated, but rather in which an estimate based on a range of acceptable values can be accepted.

When a situation is simplified, a mathematical model is produced. This allows us to obtain an approximation that can be somehow valid depending on whether the model chosen corresponds to reality. In this sense, the result obtained is an estimate of the quantity requested. In our research study, we use a problem related to this type of activity that links the estimation of a quantity with the representation of a real situation through a model.

Ärlebäck [13] defines Fermi problems as “Open, non-standard problems requiring the students to make assumptions about the problem situation and estimate relevant quantities before engaging in, often, simple calculations” (p. 331). The problem proposed in our work satisfies this definition and therefore both the results of the literature concerning modelling tasks and Fermi problems can be applied.

Turning attention now to the studies that have been carried out on modelling skills [18,19], modelling competencies involve the steps of the modelling process but go beyond this to “include skills and abilities to perform modelling processes appropriately and goal-oriented as well as the willingness to put these into action” [20] (p. 117). The focus of our study is centred on the modelling processes and, even though we can find the use of modelling routes ([21,22,23,24]), we will centre on the Ärlebäck [13] Modelling Activity Diagram framework as we will see in the following sections.

The reason why we choose Arleback’s [13] approach instead of Borromeo Ferri’s modelling cycle [22] is that in MAD, the “making model” activity incorporates parts of both “simplifying the task” and “anathematising” of the modelling cycle. In our particular problem is hard to separate these two activities. The same happens when considering the “interpreting” and “validating” activities that in the modelling cycle are clearly separated. In fact, the basis of Arleback’s contribution is, as in our case, to avoid ambiguity to separate the mentioned activities.

Different authors have characterised the strategies which appear in the schemes presented by the students when solving Fermi problems regarding estimation of quantities [17,25,26]. These strategies will be considered in the analysis of the students’ productions during the development of this research work. In particular, the concentration measurements strategy (density), the iteration of a unit strategy and grid distribution model. As we will see when we analyse the students’ resolutions, there is another model that arises in this particular case, the surface division model, that is, to divide the surface of the ball pit by a projection of the ball.

1.2. Difficulties, Mistakes and Obstacles When Solving a Modelling Task

In the last decades, several authors have worked to identify the difficulties that solvers have with certain stages of the modelling process during a modelling task. Julie and Mudaly [27], in an experiment with secondary school students, revealed an absence of the processes of returning to the real problem situation and checking the suitability of the solution. Kaiser, Schwarz and Tiedemann [28] found that most of the prospective teachers of their experiment, who had knowledge about modelling at various levels, did not consider the validation phase in the modelling process. Biembengut and Faria [29] concluded that most of the teachers and students of mathematics teacher-training participating in their study had difficulties not only in reading and understanding the prescribed modelling task, but also in remembering the mathematical knowledge required to solve it. Socas, Ruano and Domínguez [30] analysed the mathematical modelling process in terms of the difficulties and errors of a group of secondary school students. They conclude that a high number of errors were due to not completing the problem by giving a solution and that, in addition, a considerable number of errors occurred due to only representing the situation graphically without knowing how to interpret it in order to answer the given question. Şen Zeytun, Çetinkaya and Erbaş [31] determined that only some teachers used the original context to verify their results.

Additionally, in the sense of blockages during a modelling task, Stillman, Brown, and Galbraith [32] applied the Galbraith et al. [33,34] framework for identifying these blockages that hinder progress in transitions in the modelling process applied to secondary school students.

Taking one step further, there exist some attempts to organise, classify or categorise the origin and kind of mistakes when solving a modelling task. Couch and Haines [35] analyse the skills of mathematics undergraduate students while studying a course in mathematical modelling. They are at an intermediate stage between the expert and the novice, whose modelling capabilities are affected by the limitations of their domain knowledge in both organisation and content. They identify four areas in which these students still have difficulties and their answers are classified as follows: difficulty in maintaining the balance between the real world and the abstract model; choosing an appropriate mathematical model; deploying appropriate mathematical concepts and procedures to solve the problem; not relating the model results to the real world.

Große [36] analyses the difficulties with real problems and identifies the following aspects as the origin of these obstacles: (a) learners do not adequately meet the intra-mathematical requirements to deal with the problem; (b) students have difficulties in extracting the relevant information, and (c) learners are overwhelmed by the diversity of requirements in the sense of the number of phases in a modelling process since in general terms they are used to doing only repetitive exercises of certain processes.

Klock and Siller [37] developed a categorisation that provides an overall picture of the difficulties students have in solving a modelling task at each phase of the cycle. They based their framework on the Maaß [38] classification and all named mistakes, blockages [33,34] and difficulties [32] were classified.

Most recently, Moreno, Marín and Ramírez-Uclés [39] established a system of mistake categories in the different phases of the modelling process and the coherence between them, based on the analysis of the modelling models and the background studies. This is organised in the competencies of simplifying, mathematising, solving and interpreting. This categorisation, very similar that the one used by Klock and Siller, has recently been used in a research study involving pre-service teachers and the resolution of a Fermi problem [40]. This categorisation will be considered also in the analysis of results presented in this paper.

1.3. Temporal Analysis of the Resolution Process

As we stated before, the resolution process of a modelling problem is dynamic and cyclic. The students can shift the task with no completion of the previous phase and they can repeat the whole process several times before the problem is completely finished [41].

Several authors have determined the different phases in which problem solving resolution, and in particular modelling solving, are organised over time. Schoenfeld [42,43] analysed the behaviours that solvers develop when working on problems that cannot be solved simply by remembering and applying known solving patterns. In this way, Schoenfeld tried to categorise the behaviours in periods of time, during which an individual or a problem-solving group is engaged in one large task or a closely related body of tasks in the service of the same goal. These periods are reading or rereading the problem, analysing the problem, exploring aspects of the problem, planning all or part of a solution, implementing a plan and verifying a solution. The introduction of the time as a variable, together with the presentation of the periods, helps to understand the non-linear nature of the problem-solving processes. Goos and Galbraith [44] used a variation of Schoenfeld’s analysis to study the nature of individual and interactive strategies used when learners work together on solving problems. Additionally, Scott and Stacey [45] followed Schoenfeld’s methodology to study how students use the strategies in a problem solving situation.

Later, Brown [46] performed a timeline diagram that records the detail of the Schoenfeld periods and shows the progress of the solution as the students proceed. This diagram allows the researcher to focus more globally on the periods to find cognitive and meta-cognitive differences in the resolutions made by different students. In addition, Ärlebäck [13] developed and made use of an adapted version of the representation of the Schoenfeld periods over time to get a schematic picture of the problem-solving process of students working on a Fermi problem, the so-called Modelling Activity Diagrams (MAD). Ärlebäck identifies six modelling sub-activities to be used as codes for the activities done by students when solving a Fermi problem. These are reading (R), making model (M), estimating (E), calculating (C), validating (V) and writing (W). Recently, Albarracín, Ärlebäck, Civil and Gorgorio [11] extended Ärlebäck’s diagrams representing the sub-problems elaborated by the students to achieve their objective within the activity.

Albarracín et al. [11] affirmed that, although the modelling process involved in arriving at a solution to a Fermi problem is complex, the extended MAD allows the researcher to reveal the internal structure of the proposed problem, thus more clearly showing the work pattern of the students. Cai et al. [47] explained that the modelling cycles unintentionally hide much of the real work of mathematical modelling. In this line, Albarracín et al. [11] supported that the extended MAD diagrams can be a helpful tool to visualise some of the hidden elements, and that “previously seemingly messy and haphazard problem-solving processes can be understood as highly structured” (p. 228). Thus, MAD diagrams can give a clue as to where student mistakes may be and why they change of phase or why they start another modelling cycle.

However, we propose an extension of Ärlebäck’s [13] Modelling Activity Diagrams that includes the temporal location of the mistakes, by following the classification of Moreno, Marín and Ramírez-Uclés [39], so that more information about Fermi’s problem-solving process can be obtained. This extension can be made due to the correspondence between MAD sub-activities and Moreno, Marín and Ramírez-Uclés categories. According to these authors, errors cannot always be identified with specific stages of the modelling cycle and therefore they focus more on the actions made by the students, as reported by MAD. Figure 1 contains information on how the mistake classification, the MAD sub-activities and the transitions in the modelling cycle relate to each other. Although we will go into these relationships in more depth later, we will briefly outline them now.

The (a) Idealising transition on the modelling cycle corresponds to Type 1 mistake and contains Estimating and Modelling sub-activities in MAD. The (b) Mathematising transition corresponds to Type 2 mistake and part of the Estimating and Modelling sub-activities in MAD. The (c) Investigation of the model transition on the modelling cycle corresponds to Type 3 mistake and part of the Calculating sub-activity in MAD and the (d) Interpretation transition on the modelling cycle corresponds to Type 4 mistake and part of the Validating sub-activity in MAD.

As we can observe, MAD sub-activity Modelling incorporates parts of both “Idealising” and “Mathematising”. As Ärlebäck [13] states, the reason of this fusions is that is often difficult to separate these two actions. His decision to separate the estimation sub-activity, which is completely in line with our view, comes from the importance that this action has in relation to Fermi problems and is therefore analysed separately in his diagrams.

In the same way, the distinction in the error classification of the models chosen by the students when solving the task and which are reflected in the category Type 2, will allow us to make a more detailed study, which we will see later in Section 3.

In the works mentioned before, the authors do the coding and graphical representation of the diagrams manually, which requires an investment of time and effort to find additional graphical representation tools when analysing the student’s behaviour. Pla-Castells and García-Fernández [41] presented a tool that simplifies the registration of a problem-solving session and the representation of the corresponding diagrams. This tool is a computer application which they call Task Time Tracker (TTT).

The aim of the present research work is to investigate the potential of enhanced Modelling Activity Diagrams (MAD+) in the detection of obstacles and conceptual and procedural mistakes made by students when solving Fermi problems. For this purpose, a tool has been developed that allows these diagrams to be modified, adding a visualisation layer of the mistakes made by students in each of the phases.

The research question that arises at this point is: “Can MAD diagrams combined with the temporal representation of mistakes provide more in-depth information on Fermi problems’ resolution processes?”.

This objective will be addressed through the following questions:

-

Can the classification made by Moreno, Marín and Ramírez-Uclés [39] be adapted to the problem proposed in our research? Additionally, if this is the case, can this classification be used as the basis to detect students’ mistakes?

-

Does the MAD+ diagrams allow us to see which phases are the most conflicting? What interpretation can we give to the presence of mistakes of a category in the different phases of the process described in the MAD+?

2. Materials and Methods

In this section we present the sample of students we work with and the specific Fermi problem they had to tackle.

2.1. Sample

The study was carried out on 12 third grade prospective teachers of the degree in Primary School Education of a public university of Spain. It was performed by four groups of mixed gender students (ages 20 to 21) randomly distributed among the students of the natural class group, which in the following will be referred to as Groups A, B, C and D, respectively.

The Fermi problem proposed for this study was distributed as a voluntary assignment as part of the course activities and was carried out by video conference through a virtual meeting that was recorded. Students were also asked to submit the written resolution of the assignment as part of the activity.

2.2. The Fermi Problem

The problem provided to the students was about estimating the number of objects needed to fill a large volume and was presented to the students using a familiar context. The problem statement was the following:

Vincent takes his little brother to a ball pit to celebrate his ninth birthday. When they enter, they realise that the pool has been filled with new balls. Vicent’s brother thinks that they’ve bought a lot of balls, probably more than 1000! Do you think Vicent’s brother has exaggerated the number? Calculate approximately how many balls fit in the pool if you have the diagram shown in Figure 2.

The students solved the task without any help from the instructors. At the time of the online meeting, the students were alone and the only clarification accompanying the task statement was the following: “The task is solvable. All data can be obtained from the problem statement and the scheme”.

2.3. Methodology

As we stated before, the aim of our research is to provide new tools to improve the training of prospective teachers. All the analysis that will be discussed in this work is based on the interactions between the student members of the groups when solving a Fermi problem. The study of the discussions and interactions during the problem-solving process within a workgroup can be of great interest [50,51,52].

Although the written productions of the groups were collected to analyse what the students considered to be their best resolution, the analysis in this research work is based mainly on the video recordings and on the transcription of the discussions held during the resolution of the task. On the one hand, it is necessary to detect the moment in time when the mistakes occur, and, on the other hand, it is important to analyse the resolution processes carried out by each of the groups.

For the detection of mistakes, it was necessary to have transcriptions of the discussions recorded in the video conferences. These were made by using specific software and reviewed by the authors to adapt the interventions to natural language in order to avoid informal comments and grammatical mistakes. We detected the mistakes in the text of the transcript so that their timestamp would be recorded. The examples presented in this paper have been translated from Spanish for ease of understanding, and the original dialogues can be found in Appendix A.

The MAD+ Diagrams will be constructed using the TTT software [41] which, allows us to create time diagrams that superimpose a layer with the mistakes observed in the analysis of the student’s resolution processes.

Our study has two differentiated steps. On the first step, we adapt the classification made by Moreno, Marín and Ramírez-Uclés [39] to the problem proposed in our research to facilitate the implementation of the research study. On the second step, an additional layer containing the mistakes made by the students’ groups will be added to the MAD diagrams. This will allow us to obtain valuable information on the mathematical modelling processes together with possible conceptual and procedural mathematical shortcomings.

3. Results and Discussion

In this section we will discuss the results observed during the analysis of the types of mistakes that appear in the different phases of the modelling process detected in the transcriptions. In this first discussion, the mistakes are aggregated, considering only if they appear or not during each phase, even though some particular examples are presented. Then, we address the analysis of the performance of each group separately by means of the MAD+ diagrams.

3.1. Mistakes and Misconceptions Detection Using the Transcriptions

As we stated before, Moreno, Marín and Ramírez-Uclés [39] based their categories around four competencies: simplifying, mathematising, solving and interpreting. The particular wording of our task makes this categorisation concrete as shown in

Table 1. Concretion of the mistake categorisation [39] when applied to the formulation of the task used in this study.

	Mistake Description	Students’ Considerations
T1. Simplification	1.1. Incomplete real model associated with the lack of consideration of elements of reality	Any of the following elements are not considered: - Space occupied by the ladder, slide, and doors. - Curved shape of a corner of the enclosure. - Depth of the pool. - Perimeter of the pool. - Radius of the ball. - Scale of the scheme.
	1.2. Incomplete real model due to inconsistencies in the relationships between the elements of reality considered	- Measure estimation mistakes. - The learners are aware of the necessary variables, but they do not study them because they do not know how to do it or they do it incorrectly. - They do not relate the variables well.
	1.3. Does not develop an objective function for the real model that can then be expressed in mathematical terms.	- They do not plan any of the unit iteration, density or grid models.
	1.4. Does not build a real model.	- They do not try to deal with the problem. They do not suggest any strategy.
T2. Mathem.	2.1. Mathem. model inconsistent with reality	- Use of the grid model. - Use of the surface division model.
	2.2 Incomplete mathematical model	- Use of the unit iteration model without considering the gap between the balls.
	2.3. Does not build any mathematical models	- Try to solve the problem without doing any mathematical modelling.
T3. Solve	3.1. Conceptual mistakes: failure in relation to the associated mathematical object	- Not knowing how to use the scale or considering the scale 1:100. - Use of cm instead of cm${}^2$ when calculating areas. - Reversal error, that is, multiplying instead of dividing or vice versa [53].
	3.2. Procedural mistakes in calculations	- Not knowing how to convert cm${}^3$ to m${}^3$ or vice-versa.
	3.3. Incomplete resolution of the proposed mathematical model.	- Not knowing what to do with the curved shape.
T4. Interpret.	4.1 The results obtained are not interpreted on the basis of the proposed model	- Overestimated or underestimated results not recognised as a bad result. - Incorrect or lack of validation of the partial results in each of the phases.
	4.2. The limitations of the mathematical model in the original situation are not recognised.	- They do not recognise that calculations are approximate, and that result cannot be exact. - They do not realise that they should have considered the space between the balls.

.

This table is the guide to specifically and unambiguously detect mistakes in the categorisation mentioned above. In the first column, the main category is named and the statement of the mistakes included in the original categorisation is included. Finally, in the second column we present a description of the form in which the mistakes of each category appear during the solving process of our Fermi problem. This second column has been obtained in several stages. In the first place, the authors solved the experimental problem and made a hypothesis about the possible errors that students could make. The errors in this list were then placed in Moreno’s categorisation. Finally, the content of this particularisation was refined by listening to all the recordings of the experiment. The rest of this subsection presents different transcripts of students’ dialogues that illustrate how this classification was particularised using the empirical data.

We have to focus on the importance the model chosen by the students has in Moreno, Marín and Ramírez-Uclés’ categorisation [39]. As we saw in Section 1.1, the resolution strategies used in problems similar to the one presented in this paper have been extensively studied in previous works [17,25,26] and we think that it can open a line of research that allows us to understand the resolution strategies and their relationship with the obstacles and difficulties presented by the students.

Once we were clear about the guidelines for detecting mistakes, we looked for them in the transcripts of each of the groups. The classification of the mistakes of the different groups is included in

Table 2. Mistakes made by group in each category.

	T0	T1				T2			T3			T4
	Reading	Simplification				Mathematisation			Solve			Validate
	0	1.1	1.2	1.3	1.4	2.1	2.2	2.3	3.1	3.2	3.3	4.1	4.2
Group A			x				x		x	x	x	x
Group B	x		x			x			x	x	x	x	x
Group C		x	x				x		x			x
Group D		x	x			x		x				x	x

. As we can see, all the mistakes in Moreno, Marín and Ramírez-Uclés’ categorisation [39], except 1.3 and 1.4, appear in our experiment, with 1.2 and 4.1 being present in all groups.

Hereafter, we comment on some relevant examples of the different mistakes committed by the groups in each category. We show several examples of the transcriptions of the students’ resolutions with the exact timestamp (hh:mm:ss). The students’ interventions are labelled as GXSY, being X the name of the group (A, B, C or D) and Y the code of each student (1, 2 or 3).

We remark that we have added one more category to the classification of Moreno, Marín and Ramírez-Uclés [39] because one of the groups did not properly understand the statement of the problem and they based their resolution on incorrect hypotheses. We call it “Type 0—Reading” mistake. In the following dialogue, it can be seen how Group B does not understand the statement of the task and therefore they are confused about the validation data.

00:07:46 GBS2 No, because it says: how many balls did they have to buy? I’m sure it’s more than 1000. It’s like they’ve added more balls.

00:07:50 GBS1 Yeah, yeah.

00:07:54 GBS3 So maybe there’s a catch there.

00:07:57 GBS2 Of course that’s what it is. It’s not that there’s 1000, it’s that there’s 1000 more than there were before.

In the simplification (estimation) phase, most groups knew which variables had to consider when solving the task, but they were discouraged from using them because they did not know how to deal with their relationship or their use during the task resolution. For example, Group B realised that, since they had a drawing of the shape of the ball pit, they had to use a scale. This reasoning ended quickly when they realised that they didn’t know how to manage it.

00:02:45 GBS1 And we could do it scaled because you have a drawing. I mean, from the drawing you can measure, and you do it scaled but I don’t know if that would be okay.

00:03:05 GBS1 Because what scale do you use? I don’t know, I don’t remember how that worked.

00:03:31 GBS1 We’ve said that you must calculate the volume by taking out the slide, the staircase, and doors, right?

00:04:24 GBS3 Maybe it’s crazy to take a ruler and then scale it.

00:04:46 GBS1 Of course, I think the right thing to do is to do the scale, but I don’t know, is there a fixed scale for the drawing or something?

00:04:57 GBS2 No. You can do whatever scale you want. I don’t think you don’t gain anything by doing the scale because the scale is whatever you want.

In the mathematisation phase, Groups A and C chose the unit iteration model. In this case, the resolution process corresponds to the division of the volume of the pool by the volume of a ball. Furthermore, Group A realised that the space between the balls had to be removed, but did not know how to do it. Group C also took this space into account but estimated, without any calculation, that 2000 balls had to be removed (see example below). In Group C, in addition, the density model appears, but they make the mistake of thinking about how many balls fit in a square meter, not in a cubic meter, and they quickly abandoned this reasoning because they did not know how to work it out mathematically.

00:19:35 GCS2 What I would say is that the total number of balls that we’ve got is 7458, but as the balls are not going to take up all the available space, we could subtract 2000 of those balls, at a guess…

00:19:47 GCS3 Yeah…2000 will certainly be left out.

00:19:51 GCS2 Because, anyway, we don’t know how many balls fit in 1 square metre, we’d have to assume that…Well, yeah, we could figure it out, you know, by having the diameter of the sphere…

00:20:08 GCS3 Yeah, but you can’t work out how much the little gaps take up either…

00:20:13 GCS2 No, no, no…The volume that it takes up in those holes, we can’t calculate that.

00:20:24 GCS3 Subtract about…2000…

00:20:27 GCS2 Yeah, it’s by estimation…as these are modelling problems…

00:20:44 GCS1 …minus 2000, so 5458 balls. So…, so we could say that there’s room for more than 1000 balls.

Continuing with the modelling phase, Groups B and D formulated models that were not appropriate for solving the problem. Group B proposed to calculate the volume of one ball and then calculate the volume of the 1000 balls mentioned in the statement, but they did not know what to do with this information and abandoned the solution. They solved the statement using an area division model by considering only one layer of balls. They realised that ball pits have more than one layer, but they rejected the solution because they did not know how to approach it mathematically. Group D, meanwhile, tried to estimate the number of balls that would fit in a ball pit by comparison against the room they were in, and guessing how many balls would fit. All attempts at estimating lengths and doing mathematical calculations were soon abandoned because they felt it would be difficult.

In the resolution (calculation) phase, many mistakes occurred when using the cubic units of magnitude and their conversion into sub-units. In addition, the reversal error appears when multiplying the volume of the ball by the volume of the pool in the unit iteration model. All the groups considering that the scheme is scaled, take the scale of 1:100 by automatically going from centimetres to metres without keeping in mind the elements of reality present in the scheme. In the following dialogue from Group A’s resolution, we find that the students referred to metres and centimetres instead of cube units and they divided by 1000 instead of 1,000,000 to go from cubic centimetres to cubic metres.

00:48:03 GAS3 37 million 44,000

00:48:11 GAS1 Shall we convert it to metres?

00:48:14 GAS2 Are you doing it in centimetres?

00:48:29 GAS3 Okay, now better 37,044 meters.

00:48:36 GAS1 cubic meters.

All groups present mistakes in the validation phase. In particular, those corresponding to an overestimated volume of the pool and the ball were due to mistakes in the change of magnitudes. One of the groups finds that the volume of the ball is greater than the volume of the pool, but they do not consider this to be a mistake and they think the problem is solved without calculating what was requested in the task. In the transcript of the following dialogue, Group B gives an estimate of the ball pit pool of 13.5 square metres while implicitly assuming a scale of 1:100 as they had measured the scheme in centimetres.

00:35:14 GBS1 Okay, it is 13 point 5, the whole area [of the pool picture]

00:35:20 GBS2 Thirteen point 5 square meters. Ok.

It should be noted that some of the mistakes described in Table 1 are caused by the omission of fundamental elements in the mathematical modelling processes. Therefore, they are not labelled with a specific timestamp, even though they are marked in Table 2.

To sum up, we have mapped the categorisation of Moreno, Marín and Ramírez-Uclés [39] to the Fermi problem proposed in our research work and it can be used as the basis to detect students’ mistakes. This result allows us to use this categorisation overlaid in the MAD+ diagrams that will be described in Step 2.

3.2. Analysis of the Mistakes’ Timeline. Construction of the MAD+ Diagrams

As we stated before, the aim for this part of the research is to show how the inclusion of a layer in the MAD diagrams, that includes the mistakes committed by the groups of students when solving a Fermi problem, can help to understand the misconceptions they have regarding modelling processes and mathematical concepts and procedures. In order to describe the creation of the MAD+ diagrams, we have to clarify what are the actions that students are taking during each of the phases. We have adapted the scheme by Ärlebäck [13] defined to our particular problem, removing the writing phase. The construction of the MAD+ diagrams is based on the description of the categories shown in

Table 3. Characterisation of students’ mathematical thinking in the context of the proposed problem.

MAD Categories	Description of Modelling Activities of Students from the Problem in Context
Reading	Understanding the task: The students must understand the context of the problem and to focus on the main question of the task. How many balls can fit into the pool of the ball pit? Is this number lower or greater than 1000?
Modelling	Transitioning from a real-world context to a mathematical interpretation of the task: Students find out how to calculate the number of balls fitting in the pool, asking themselves how to obtain the result. They have to think if any of the unit iteration, density or grid models can lead to a solution, even if they are not explicitly aware of that decision. They list the quantitative variables to be estimated.
Estimating	Making sense of the quantitative estimates in the problem in context: Students estimate the quantities involved in the model.
Calculating	Using simple mathematical concepts to calculate the missing information on the sketched diagrams or figures: Students calculate the volume of the pool and the volume of the ball, convert cm${}^3$ into m${}^3$ (or vice versa), use the proportion to calculate the scaled measures, etc.
Validating	Interpreting, verifying and validating the mathematical calculations: Students make sense of the mathematical results, including calculations, within the problem in context. They compare their calculation with the number given in the task formulation.

.

We consider the four general categories from Moreno, Marín and Ramírez-Uclés [39] presented in Table 1. The correspondence of Table 1 categories and the ones in Table 3 is as follows: T1—Simplification corresponds to Estimating and Modelling in MAD; T2—Mathematisation corresponds to Estimating and Modelling in MAD; T3—Solve corresponds to Calculating in MAD; T4—Validate corresponds to Validating in MAD as seen in Figure 1. In order to include the mistakes in the MAD+ diagram we will use the codes specified in

Table 4. Mistake codes.

Type of Mistake	Code
0. Reading	Square (∎)
1. Simplification	Triangle (▲)
2. Mathematisation	Diamond (♦)
3. Solve	Cap (⯊)
4. Validate	Star (★)

.

Additionally, as we explained before, we add the “Type 0—Reading” category.

When listening to the audio of each group, and by using the TTT software, the coder can indicate the exact moment in which there exist a jump of phase in the modelling cycle and, furthermore, the instant in which a mistake is detected. By using the codification of Table 4, we can specify the mistake encountered. Note that the MAD+ diagram can be obtained in the same process of reviewing the audio recording, without the need to go through the transcription in an additional step, thus making the analysis more efficient. As a result, we obtain the output shown in Figure 3, Figure 4, Figure 5 and Figure 6.

Next, we describe how the MAD+ diagrams can be used to obtain an analysis of the different mistakes that appear during the resolution process. First, we present a synthetic analysis of the kind of mistakes appearing in each phase of the modelling cycle. For this purpose,

Table 5. Mistakes detected in each phase of the modelling cycle. o = expected mistake; x = other phases’ mistakes. Description of T0–T5 can be found in Table 1.

	T0	T1	T2	T3	T4
V					o
C			x	o	x
E		o		x
M	x	x	o	x	x
R

shows the different mistakes that have been detected in each phase. Then, we explain the reasons why each mistake appears in a particular phase. Some of them are expected to appear in certain phases due to their nature and we will call them “expected mistakes”. We use a circle to indicate this kind of mistake and a cross for those mistakes that appear during a phase in which they are not expected to be.

From Table 5, we observe that the students do not make any mistakes during the reading phase, not even Type 0 (∎) mistakes, which would be the expected ones. Type 0 mistakes, however, do appear during the modelling phase.

Indeed, it can be observed that during the modelling phase all types of mistakes appear at some point in the construction of the needed model. Type 2 (♦) is the expected mistake, and it occurs because the students realised that they could have chosen a better model, even though they did not know how to approach it. For instance, Groups A and C are aware of the need to take into account the gap between balls, but they don’t find a good strategy, such as using the concept of density model, to address it. Additionally, Type 0 (∎) emerges because they had not understood the statement of the problem while they were thinking about the strategy to solve it. Furthermore, Type 1 (▲) appears because, while they were building the model, they missed variables that should be considered. In addition, Type 3 (⯊) happens because the students had conceptual mistakes while building the model. Finally, Type 4 (★) is detected due to the incorrect validations that had to be made after each partial result the students obtained.

In the estimating phase, we find the Type 1 (▲) mistake, which is the expected one, mainly because the students fail to obtain a good estimation of a required measure. In addition, Type 3 (⯊) occurs because they make conceptual mistakes while estimating.

Regarding the calculation phase, Type 3 (⯊) is the expected mistake and, indeed, it happens frequently in all of the groups except for Group D. Additionally, Type 2 (♦) appears because, when calculating, the students make statements that are inconsistent with the real model. For instance, they express the volume in litres, which makes no sense in the context of the problem. Additionally, in this phase, Type 4 (★) mistakes appear due to erroneous partial validations.

Finally, in the validation phase, Type 4 (★) mistake, which is the expected one, is the unique we have detected.

In conclusion, we observe that any kind of mistake can appear in any phase of the modelling cycle, the modelling phase being the most conflictive one, in the sense of being the one with the highest frequency and variety of mistake types. This observation makes sense if we take into account that the group of students that conformed to the sample under analysis were novice students solving a modelling task, and, thus, they presented difficulties when moving from the real world to the model [54,55]. This is also consistent with the idea that the highest frequency of mistakes occurs in the phases in which the original real situation appears, for instance, during the simplification phase in which they try to obtain the real model [39]. However, in the validation phase, when the students interpret the results to come back to the real world, we neither detect as many mistakes nor as varied as when the students are involved in modelling activities. In our opinion, this does not mean that they are better at validation than at modelling, but rather that, as explained above, they hardly deal with this phase and consequently, the omission of the work means no mistakes are made. This agrees with previous studies asserting that students do not even consider the need for a validation phase when they get a result [28]. Note also that all the expected mistakes, except in the case of Type 0 (∎) mistakes, appear in their corresponding phase.

In the analysis so far, we have addressed the overall distribution of mistakes during the different phases, considering the groups altogether. Next, we use the individual MAD+ diagrams to revise how the modelling process is developed in each case. We shall see that the usage of the MAD+ diagrams will help us detect the most significant aspects to pay attention to, without having to go through the recordings or the transcriptions again. It is possible to see, in a particular way, what type of mistakes are the most frequent in each group and, consequently, which ones should be worked on to improve the teaching-learning processes.

Figure 3 shows the appearance of many Type 1 (▲) mistakes, which indicates that Group A presents estimation problems, not only in the corresponding phase, but also at the time of modelling. This could be reflecting a commonly observed problem regarding the lack of estimation activities during primary and secondary studies [56,57,58]. If the estimations are not done correctly, the validation process may be flawed since the results obtained would entail erroneous conceptualisations that do not correspond to the real model.

From Figure 4, we observe that Group B has many problems in the modelling and the calculation phases. They present a lot of conceptual and procedural problems that need to be addressed in their mathematics teaching-learning processes.

Similar to the case of Group B, Group C has, primarily, estimation gaps that need to be addressed. These estimation mistakes are translated into validation mistakes the two times they try to validate. As a result of these mistakes, students finish the resolution process without a final validation of the result, as it can be seen in Figure 5.

In Group D (Figure 6), despite being the group with the least number of mistakes in the graph, there are two direct jumps between the modelling and validation phase without going through estimation and calculation. This occurs because, although they make attempts to create a model, they do not manage to obtain it, and try to solve the problem without having a model.

These results show, on the one hand, evidence that the modelling phase is the most difficult one. Particular emphasis needs to be placed on training teachers to learn how to move correctly from the real model to the mathematical model. On the other hand, the MAD+ diagrams allow us to see at a glance what kind of mistakes are most frequent in each group and this will allow us to focus teaching on the most significant deficiencies in each case.

One of the key benefits of the MAD+ diagrams is that it provides a visual representation that informs about the modelling phase in which the students are when the mistakes arise. By means of the MAD+ diagrams, this contextual information can be grabbed without having to dive into the recordings or the transcriptions of the problem resolution to find it.

Note, however, that the original information of the resolution is still necessary for a proper analysis of the activity. Not only because the recording has to be reproduced entirely to build the MAD+ diagram, but also because it can be necessary to focus on specific mistakes to understand their origin and their relationship with other mistakes or with other aspects of the modelling processes, such as phase transitions. Nevertheless, the MAD+ diagram will help find the appropriate section of the recording or transcription very efficiently.

4. Conclusions

As we have seen in the previous section, MAD+ diagrams can provide a deep insight into the processes of solving Fermi problems. To achieve this, we have adapted a relevant mistake classification to the problem proposed in our research and used it as the basis to detect students’ mistakes during the resolution of the proposed Fermi problem. Furthermore, a MAD+ diagram has been constructed for all the productions made by the students in order to give an interpretation of the resolution processes steps and the misconceptions produced.

The combination of the mistake information together with the MAD diagrams, to build the MAD+ diagrams, makes it possible to efficiently analyse the process of solving modelling activities and, in particular, a Fermi problem. These diagrams offer enough information to analyse the resolution process followed by the students, as well as their misconceptions regarding mathematical concepts and procedures, without the need to revisit the transcriptions or the recordings of the activity.

In summary, this work allows us to open a line of research that relates the mistakes made by students when solving a Fermi problem with the temporal processes of solving it. In this way, we can obtain a macroscopic picture of where the weaknesses of each group of students may lie and focus the teaching–learning processes of mathematics on the solution of these deficiencies.

Furthermore, in order to particularly analyse in more detail the cognitive processes appearing when solving a Fermi problem, we could focus on developing an analogous tool of representation, with sub-activities and mistakes, based on the modelling cycle instead of Arleback’s approach. Then we could consider the “understanding”, “simplifying” and “mathematising” activities separately. These require the most complex cognitive processes, so an exhaustive analysis along this line made more sense in this framework.