Innovation of Teaching Tools during Robot Programming Learning to Promote Middle School Students’ Critical Thinking

Round 1

Reviewer 1 Report

Dear Authors,

Thanks for submitting your manuscript. It is an interesting paper with classic studies related to the questions guiding the study; however, arguments and discussions of the findings must be significantly improved. Little description of the results are provided. 

Also please specify if the questionnaires and the study have been ethically approved?

Some other minor changes/suggestions are detailed below:

• The degree of significance in lines 22 and 23 needs to be defined by p value.
• The sentence in lines 39-41 is grammatically incorrect (“which” to be removed)
• Reference 66 in line 41 does not follow the reference number order
• Line 150 needs to be referenced.
• Typo error in line 245 “usedd"
• Subtitles are full sentences describing results – perhaps can be summarized

Regards

Author Response

1. Thanks for submitting your manuscript. It is an interesting paper with classic studies related to the questions guiding the study; however, arguments and discussions of the findings must be significantly improved.

Reply: 

Thank you very much for your comments.

Section 5.1 (P. 13, lines 374-404) has been revised to describe the findings with the result.

1. 13, lines 374-404:

In this study, students’ critical abilities included the five sub-dimensions of recognition of assumptions, induction, deduction, interpretation, and evaluation of arguments. After the 6-week experiment, the five abilities of critical thinking had all improved. For recognition of assumptions, it means the ability to recognize hidden premises. In the 6-week robot programming learning, the functions of the robot to be built were imparted to students, which could help give them the prerequisite experience [56] so as to develop their recognition of assumptions ability (M = 4.14, p = 0.000). Induction refers to the ability to conclude the results according to the known information. Robotics education can attract students’ interest in learning, provide them with full space for imagination, and cultivate their innovative thinking, practical ability, and comprehensive application ability [36]. Therefore, their induction ability also improved during the robot programming education (M = 4.49, p = 0.000). Deduction ability can help learners to identify the latent relationships between description and previous description and to extrapolate from general principles to conclusions in special cases. In the 6-week experiment, students were allowed to construct the program model according to the robot functions, and constantly deepen their understanding of the robot program, so as to cultivate their deduction ability (M = 3.89, p = 0.000). Interpretation is the ability to find evidence from statements and evaluate the possibility of the induction. The learning objective of the 6-week robot programming learning was to make a manipulator based on the robot program. The objective required students to design and construct the robot based on an understanding of the program. Therefore, students needed to interpret the meanings of the robot program so that their interpretation ability could be developed (M = 4.45, p = 0.000). As for evaluation or arguments, it means the ability to evaluate the support level of arguments in a question. In the 6-week robot programming education, Construction-Criticism-Migration (CCM) was applied by students in the experimental group. They were allowed to independently construct the robot, discuss and argue with each other, and draw lessons from their failure experience for migration purposes. During this process, students’ evaluation or argument ability improved (M = 4.47, p = 0.006).

2. Little description of the results are provided. 

Reply: 

Thank you very much for your comments.

Parts of Section 4 and 5 have been revised to describe the results and findings in detail.

3. Also please specify if the questionnaires and the study have been ethically approved?

Reply: 

Thank you very much for your comments.

In Section 3.1, there is some information for the explanation of participants’ agreement of participation (P. 5, lines 172-174). Some statements were provided in lines 503-504.

1. 5, lines 172-174:

Participants were told that they were participating in an experiment, the result of which may be published, and their privacy was guaranteed. All participants agreed to participate in the experiment.

1. 15-16, lines 503-504:

Institutional Review Board Statement: Not applicable.

Informed Consent Statement: Not applicable.

4. Some other minor changes/suggestions are detailed below:

The degree of significance in lines 22 and 23 needs to be defined by p value.

Reply: 

Thank you very much for your comments. In Table 5 and 7, p value for the five sub-dimension was described. There were 5 numbers for the p value of five sub-dimension. If there were added in the abstract, it may be confused by the readers. Thus, we decided not to add the values in the abstract.

5. The sentence in lines 39-41 is grammatically incorrect (“which” to be removed)

Reply: 

Thank you very much for your comments. The sentence was revised (P.3, lines 44-46).

P.3, lines 44-46:

Critical thinking ability is an important ability of students which has an impact on their learning, future work, and lives [6].

6. Reference 66 in line 41 does not follow the reference number order

Reply: 

Thank you very much for your comments. The number was changed to 6 (P.3, lines 44-46).

P.3, lines 44-46:

Critical thinking ability is an important ability of students which has an impact on their learning, future work, and lives [6].

7. Line 150 needs to be referenced.

Reply: 

Thank you very much for your comments. The sentence is to describe the aim of this study.

8. Typo error in line 245 “usedd"

Reply: 

Thank you very much for your comments. The word has been revised.

9. Subtitles are full sentences describing results – perhaps can be summarized

Reply: 

Thank you very much for your comments. The subtitles of Section 4 were revised as full sentences (P.10, line 303 and P.11, line 332).

P.10, line 303:

4.1 All students’ critical thinking improved after the six-week experiment

P.11, line 332:

4.2 Significant differences in students’ critical thinking existed in the two groups

Author Response File: Author Response.docx

Reviewer 2 Report

The paper presents an experiment questioning the influence of CCM and Robot education in critical thinking of learners.

The article is very well written.

The introduction is well structured and reaches clearly the objective of the research. Just notice that reference [66] does not exist and authors may be referring to [6].

The literature review is clear and states adequately the basic concepts underlying the research work. However, when defining critical thinking, authors state "...human beings are not born with the ability of critical thinking..." which is not supported by the provided references. Moreover, when explaining robotics education, authors state "Previous studies have shown that robotics education can reduce cognitive impairment and enlighten children"; the statement is too general and is not supported by the following cites.

Regarding the experiment procedure section, what does "Students in the experimental group with independent learning constructed the robot learning task mainly by themselves" mean? Moreover, in order to guarantee the reproducibility of the experiment, the concrete activities developed by both groups of students must be provided.

Also for the sake of reproducibility, the instrument section must include the questionnaires.

In relation to the discussion section, when addressing the relationship between robotics education and critical thinking (5.1), the second paragraph only includes cites of other research works but they are not related to the experiment in the paper. Moreover, again the authors introduced a statement about the absence of critical thinking at birth.

Conclusions include the recommendation "Therefore, CCM instructional design can be applied more often in robot programming education". According to the obtained results, authors should be more assertive.

A limitation of the work that the authors do not mention is the lack of a control group without robot education during the same period.

Author Response

Dear Editors and Reviewers,

We would like to thank the editor for handling our paper and the reviewers for their hard work and constructive comments. We found the reviews to be helpful in further improving the quality of our manuscript. We have replied to the reviewers' comments, and we have updated the revised manuscript with yellow highlighting that indicates the changes. Finally, we have improved the language with the help of a professional language editor to make it more logical and flow better.

Many thanks and best regards.

Yours sincerely,

Hehai Liu, Jie Sheng, Li Zhao

1. The introduction is well structured and reaches clearly the objective of the research. Just notice that reference [66] does not exist and authors may be referring to [6].

Reply: 

Thank you very much for your comments.

The error has been corrected (p. 3, lines 44-46).

1. 3, lines 44-46:

Critical thinking ability is an important ability of students which has an impact on their learning, future work, and lives [6]. 

2. The literature review is clear and states adequately the basic concepts underlying the research work. However, when defining critical thinking, authors state "...human beings are not born with the ability of critical thinking..." which is not supported by the provided references.

Reply:

Thank you very much for your comments.

In the available studies about critical thinking, there is a debate about whether critical thinking is an innate skill or simply the product of education. Thus, to guarantee the rigor of this current study, the description with ambiguity mentioned above has been deleted.

3. Moreover, when explaining robotics education, authors state "Previous studies have shown that robotics education can reduce cognitive impairment and enlighten children"; the statement is too general and is not supported by the following cites.

Reply:

Thank you very much for your comments.

In the sentence, we have added more description to explain what children can be enlightened of through robotics education. In addition, we have cited two papers to support this statement (p. 4, lines 118-120).

1. 4, lines 118-120:

Previous studies have shown that robotics education can reduce cognitive impairment and encourage children to be more engaged in their learning activities [32, 33].

4. Regarding the experiment procedure section, what does "Students in the experimental group with independent learning constructed the robot learning task mainly by themselves" mean?

Reply:

Thank you very much for your comments.

We revised this sentence. It means during the robot programming learning process, students in the experimental group would design, build, and program the model according to the robot functions by themselves without the teacher’s step-by-step guidance (p. 6, lines 195-197).

1. 6, lines 195-197:

Students in the experimental group conducted independent learning, and completed the robot learning task mainly by themselves without the teacher’s step-by-step guidance.

5. Moreover, in order to guarantee the reproducibility of the experiment, the concrete activities developed by both groups of students must be provided.

Reply:

Thank you very much for your comments.

We have described the activities in detail in Section 3.2 (pp. 5-6, lines 175-202), redrawn Figure 1 (p. 6, lines 204-205), added the learning task and equipment, and added Figure 2 (p. 6-7, lines 206-207) to show the learning procedure.

Pp. 5-6, lines 175-202:

3.2. Experiment procedure

The experiment lasted for 6 weeks (see Figure 1). Both groups were instructed in the content of “making a robot through programming and construction” in the robotics course. The specific learning objectives of the current robot programming learning are to make a manipulator, build the foundation of the robot, combine programming, and judge the robot direction and distance during the robot programming learning. Students in both groups were provided with the same robot programming learning environment. They moved to the robot education classroom equipped with blocks, pads, and computers to learn the same content, and their learning objectives and tasks were the same. Students in the same robot programming learning environment had similar experiences to enhance their knowledge of robot education. In previous robot programming environments, students were confronted with the same cognitive load [52]. 

In the first week, the teacher conducted the pre-test of critical thinking for the two groups and presented the finished work of the robot. In the second week, the teacher explained the functions of the robot, the required electronic components, and the programming for both groups. In the following 3 weeks, the experimental group students independently designed, built, and programmed the model according to the robot functions, while the control group students followed the teacher's guidance at each step and created a similar robot by imitating the teacher. Students in the experimental group conducted independent learning, and completed the robot learning task mainly by themselves without the teacher’s step-by-step guidance. They encountered many failures. However, they discussed and argued with each other and found new solutions to try again and again. They could ask the teacher for help as well. Their failure experience could be migrated to arouse new ideas for their next attempt. In the last week, the two groups presented and shared their final works. The post-test and evaluation were then conducted (see Figure 1, Figure 2).  

6. Also for the sake of reproducibility, the instrument section must include the questionnaires.

Reply:

Thank you very much for your comments.

More details of the questionnaire have been added in Section 3.3 (p. 7, lines 218-225). The scale was added in Appendix A (p. 16, line 508).

1. 7, lines 218-225:

 Finally, the scale comprised five dimensions: recognition of assumptions (5 items, e.g., When I do something, I will think about what I really need to learn.), induction (5 items, e.g., I can always figure out how to do it in an example.), deduction (5 items, e.g., I will choose to think for myself instead of turning to the teacher when I am in trouble.), interpretation (5 items, e.g., I can understand the information (icon, text, etc.) provided in the question correctly.), and evaluation of arguments (5 items, e.g., I can control myself and keep working towards my goals.) (see Appendix A).

7. In relation to the discussion section, when addressing the relationship between robotics education and critical thinking (5.1), the second paragraph only includes cites of other research works but they are not related to the experiment in the paper.

Reply:

Thank you very much for your comments.

More discussion about the experiment of the study was added according to the results (pp. 13, lines 374-404).

Pp. 13, lines 374-404:

In this study, students’ critical abilities included the five sub-dimensions of recognition of assumptions, induction, deduction, interpretation, and evaluation of arguments. After the 6-week experiment, the five abilities of critical thinking had all improved. For recognition of assumptions, it means the ability to recognize hidden premises. In the 6-week robot programming learning, the functions of the robot to be built were imparted to students, which could help give them the prerequisite experience [56] so as to develop their recognition of assumptions ability (M = 4.14, p = 0.000). Induction refers to the ability to conclude the results according to the known information. Robotics education can attract students’ interest in learning, provide them with full space for imagination, and cultivate their innovative thinking, practical ability, and comprehensive application ability [36]. Therefore, their induction ability also improved during the robot programming education (M = 4.49, p = 0.000). Deduction ability can help learners to identify the latent relationships between description and previous description and to extrapolate from general principles to conclusions in special cases. In the 6-week experiment, students were allowed to construct the program model according to the robot functions, and constantly deepen their understanding of the robot program, so as to cultivate their deduction ability (M = 3.89, p = 0.000). Interpretation is the ability to find evidence from statements and evaluate the possibility of the induction. The learning objective of the 6-week robot programming learning was to make a manipulator based on the robot program. The objective required students to design and construct the robot based on an understanding of the program. Therefore, students needed to interpret the meanings of the robot program so that their interpretation ability could be developed (M = 4.45, p = 0.000). As for evaluation or arguments, it means the ability to evaluate the support level of arguments in a question. In the 6-week robot programming education, Construction-Criticism-Migration (CCM) was applied by students in the experimental group. They were allowed to independently construct the robot, discuss and argue with each other, and draw lessons from their failure experience for migration purposes. During this process, students’ evaluation or argument ability improved (M = 4.47, p = 0.006).

8. Moreover, again the authors introduced a statement about the absence of critical thinking at birth.

Reply:

Thank you very much for your comments.

In the available studies about critical thinking, there is a debate about whether critical thinking is an innate skill or simply the product of education. Thus, to guarantee the rigor of this current study, the description with ambiguity mentioned above has been deleted in Section 5.1.

9. Conclusions include the recommendation "Therefore, CCM instructional design can be applied more often in robot programming education". According to the obtained results, authors should be more assertive.

Reply:

Thank you very much for your comments.

The sentence was revised according to your comment (p. 15, lines 461-464).

1. 15, lines 461-464:

Thus, in future robot programming education, it would be worth considering adopting Construction-Criticism-Migration (CCM) as a common teaching tool to help students develop their various abilities.

10. A limitation of the work that the authors do not mention is the lack of a control group without robot education during the same period.

Reply:

Thank you very much for your comments.

The current study focuses on whether the use of two types of teaching tools, Construction-Criticism-Migration (CCM) and demonstrate-practice during the robot programming learning will help promote students’ critical thinking. Therefore, robot education is the common learning environment for students in both groups, and for the experimental group and control group, the only difference is that they applied different types of instructional design.

Reviewer 3 Report

A theoretically well-founded work with adequate references to the problem being addressed, which, with a series of adjustments, can be a scientifically interesting article.  Accordingly, we propose the following considerations:

1. The scientific protocol requires a series of conditions that must be met: the problem, the objectives, the hypotheses, the sampling, the procedure and the instruments, the results and the conclusions discussed, all are required requirements since a scientific work, which is done public, must observe the possibility of being replicated, for this reason must keep a scientific organization.
2. There is no clear determination on the type of methodology used.
3. The results must be clearly expressed in terms of the hypotheses and not those of the statistical test used.
4. The Robot Programming Learning environment should describe more detail. Ex: What are functions that enhance scientific literacy and cognitive load of students?
5. Where the questionnaires used in the present study validated with populations?
6. Reading the paper I am unsure as to what components are incorporated into the “Robot Programming Learning environment” It would be helpful to have a high level description before hitting what is now Section 4.  As is, it is hard to understand the impact of Section 4 and which undermines the information in Section 5.
7. I think it would be important for the authors to clearly define the goals of the Robot Programming Learning system and the objectives they are trying to achieve. This will aid the author in understanding the goals of the methods presented.

Author Response

Dear Editors and Reviewers,

We would like to thank the editor for handling our paper and the reviewers for their hard work and constructive comments. We found the reviews to be helpful in further improving the quality of our manuscript. We have replied to the reviewers' comments, and we have updated the revised manuscript with yellow highlighting that indicates the changes. Finally, we have improved the language with the help of a professional language editor to make it more logical and flow better.

Many thanks and best regards.

Yours sincerely,

Hehai Liu, Jie Sheng, Li Zhao

1. The scientific protocol requires a series of conditions that must be met: the problem, the objectives, the hypotheses, the sampling, the procedure and the instruments, the results and the conclusions discussed, all are required requirements since a scientific work, which is done public, must observe the possibility of being replicated, for this reason must keep a scientific organization.

Reply:

Thank you very much for your comments.

In Section 2.4 there were questions proposed as the research problem and objectives. No hypotheses were proposed in the study. In Section 3, we described the instrument and experiment procedure and provided the questionnaire in detail, which could support the possibility of being replicated.

2. There is no clear determination on the type of methodology used.

Reply:

Thank you very much for your comments.

The specific type of methodology used in this study have been added in the abstract (P. 1, lines 17-25). Section 3.4 also described the methodology used in the study (P. 10, lines 297-301).

1. 1, lines 17-25:

Before the experiment, a pre-test was conducted to measure students’ critical thinking ability. Then all students were divided randomly into two groups, one as an experimental group with the teaching tool of Construction-Criticism-Migration (CCM) instructional design, and the other as a control group with the traditional teaching tool of demonstrate-practice instructional design. After a 6-week experiment, the measurement of critical thinking was applied as the post-test. SPSS was used to conduct the Independent sample t test and One-way ANOVA to explore whether students’ critical thinking ability had improved and whether differences were found between the experimental group and the control group after the 6-week experiment.

1. 10, lines 297-301:

3.4 Data analysis

The scale of the critical thinking test was distributed in the form of a paper version, and the data were entered and coded with the help of SPSS 24.0. One-way ANOVA and the independent sample t test were used to explore the difference between the two groups and within single groups before and after the experiment, respectively.

3. The results must be clearly expressed in terms of the hypotheses and not those of the statistical test used.

Reply:

Thank you very much for your comments.

Section 4 is the results section. It has been clearly expressed in terms of the two questions proposed in Section 2.4, and Section 4 is divided into two parts: 4.1 All students’ critical thinking improved after the six-week experiment; 4.2 Significant differences in students’ critical thinking existed in the two groups. Section 4.1 is consistent with the first research question: whether both groups’ critical thinking was promoted during the robot programming learning. Section 4.2 corresponds to the second research question: whether there was any difference between the two groups.

4. The Robot Programming Learning environment should describe more detail. Ex: What are functions that enhance scientific literacy and cognitive load of students?

Reply:

Thank you very much for your comments.

More description of the robot programming learning environment has been added in Section 3.2 (pp. 5, lines 178-187). However, the scientific literacy and cognitive load of students in different groups were not the focus of the study. Whether students with different teaching tools would have different scientific literacy and cognitive load may be explored in future studies (pp. 15, Lines 494-497).

Pp. 5, lines 178-187:

The specific learning objectives of the current robot programming learning are to make a manipulator, build the foundation of the robot, combine programming, and judge the robot direction and distance during the robot programming learning. Students in both groups were provided with the same robot programming learning environment. They moved to the robot education classroom equipped with blocks, pads, and computers to learn the same content, and their learning objectives and tasks were the same. Students in the same robot programming learning environment had similar experiences to enhance their knowledge of robot education. In previous robot programming environments, students were confronted with the same cognitive load [52].

Pp. 15, Lines 494-497:

Finally, since scientific literacy and cognitive load would affect the learning effectiveness in robot education, whether students learning with different teaching tools would enhance different aspects of their scientific literacy or have different cognitive loads may be explored in future studies.

5. Where the questionnaires used in the present study validated with populations?

Reply:

Thank you very much for your comments.

In the present study, three experts who have studied critical thinking and five teachers who have taught robot programming were invited to review all items and give feedback (p. 7, 214-218).

1. 7, lines 214-218:

Furthermore, in order to ensure the content validity and face validity of the questionnaire, three experts who had studied critical thinking and five teachers who had taught robot programming were invited to review all items and give feedback. The research team compared the similarities and differences in the experts’ opinions and made joint decisions.

6. Reading the paper I am unsure as to what components are incorporated into the “Robot Programming Learning environment” It would be helpful to have a high level description before hitting what is now Section 4.  As is, it is hard to understand the impact of Section 4 and which undermines the information in Section 5.

Reply:

Thank you very much for your comments.

In the robot programming learning environment, two different teaching tools were incorporated. For the control group, demonstrate-practice was applied, and students followed the teacher’s guidance at each step and imitated the teacher to produce a similar robot, while for the experimental group, students were taught by the CCM, which allowed them to construct independently, and to be full of criticism. Also, apart from the differences in teaching tools, all other equipment was the same for both groups. A more specific description of what components are incorporated into the “Robot Programming Learning environment” has been added in Section 3.2 (pp. 5-6, lines 176-202). Figure 1 was revised to show the detailed procedure and Figure 2 was added to show the learning procedure.

Pp. 5-6, lines 176-202:

The experiment lasted for 6 weeks (see Figure 1). Both groups were instructed in the content of “making a robot through programming and construction” in the robotics course. The specific learning objectives of the current robot programming learning are to make a manipulator, build the foundation of the robot, combine programming, and judge the robot direction and distance during the robot programming learning. Students in both groups were provided with the same robot programming learning environment. They moved to the robot education classroom equipped with blocks, pads, and computers to learn the same content, and their learning objectives and tasks were the same. Students in the same robot programming learning environment had similar experiences to enhance their knowledge of robot education. In previous robot programming environments, students were confronted with the same cognitive load [52].

In the first week, the teacher conducted the pre-test of critical thinking for the two groups and presented the finished work of the robot. In the second week, the teacher explained the functions of the robot, the required electronic components, and the programming for both groups. In the following 3 weeks, the experimental group students independently designed, built, and programmed the model according to the robot functions, while the control group students followed the teacher's guidance at each step and created a similar robot by imitating the teacher. Students in the experimental group conducted independent learning, and completed the robot learning task mainly by themselves without the teacher’s step-by-step guidance. They encountered many failures. However, they discussed and argued with each other and found new solutions to try again and again. They could ask the teacher for help as well. Their failure experience could be migrated to arouse new ideas for their next attempt. In the last week, the two groups presented and shared their final works. The post-test and evaluation were then conducted (see Figure 1, Figure 2).

7. I think it would be important for the authors to clearly define the goals of the Robot Programming Learning system and the objectives they are trying to achieve. This will aid the author in understanding the goals of the methods presented.

Reply:

Thank you very much for your comments.

The learning objective of the robot programming learning system has been added in Section 3.2 (pp. 5, lines 178-187).

Pp. 5, lines 178-187:

The specific learning objectives of the current robot programming learning are to make a manipulator, build the foundation of the robot, combine programming, and judge the robot direction and distance during the robot programming learning. Students in both groups were provided with the same robot programming learning environment. They moved to the robot education classroom equipped with blocks, pads, and computers to learn the same content, and their learning objectives and tasks were the same. Students in the same robot programming learning environment had similar experiences to enhance their knowledge of robot education. In previous robot programming environments, students were confronted with the same cognitive load [52].

Author Response File: Author Response.docx

Round 2

Reviewer 3 Report

Authors have addressed the major question very carefully and also added new materials in the revised manuscript. I have no technical suggestions to make regarding the revised manuscript. This paper can be accepted for publication now.