Becoming an Expert Teacher: Assessing Expertise Growth in Peer Feedback Video Recordings by Lexical Analysis
Round 1
Reviewer 1 Report
Title: Becoming an expert teacher: assessing expertise growth in peer feedback video recordings by lexical analysis
This paper aims is to research pedagogy for using video in supporting collaborative learning. It is reported an experiment in bachelor-level courses of a Vocational Education and Training (VET) teacher education curriculum in the Netherlands. This testing aimed to support student teachers’ development from ’novice’ to ’starting expert’ by using students’ video recordings and peer feedback in a blended curriculum. The paper try to answer to the question: “is students’ transformation from novice to expert reflected in an increase in lexical richness, semantic cohesion and constructive use of key terms from the literature they study?”
The study is based in a sample constituted by 10 males and 5 females.
Overall, the results found allow the conclusion that the students' lexical richness the feedback and reflections of peers increased, indicating that these students were evolving from novices to experts.
In my opinion this article does not have enough scientific value to be published in Education Sciences. The sample size is too small to make any kind of inference; furthermore, the behaviour of students who know they are being filmed is not controlled, which may bias the results; as recognised by the authors there was no a control group which could allow an assessment of whether or not the students have actually progressed. Additionally, “there was only one teacher trainer involved and the influence of teaching style, experience and other personal characteristics was not investigated. Furthermore, the influence of collaborative learning by peer feedback was not investigated at an intensive level of collaborative learning as was intended.” Consequently, there may be high subjectivity in peer feedback.
Author Response
Answer:
In general, thank you for the comments on our submission. It helps us to critically reflect again on our article and improve the quality of the publication.
Reviewer
This paper aims to research pedagogy for using video in supporting collaborative learning.
It is reported an experiment in bachelor-level courses of a Vocational Education and Training (VET) teacher education curriculum in the Netherlands. This testing aimed to support student teachers' development from' novice' to' starting expert' by using students' video recordings and peer feedback in a blended curriculum. The paper try to answer to the question: "is students' transformation from novice to expert reflected in an increase in lexical richness, semantic cohesion and constructive use of key terms from the literature they study?"
The study is based in a sample constituted by 10 males and 5 females.
Overall, the results found allow the conclusion that the students' lexical richness the feedback and reflections of peers increased, indicating that these students were evolving from novices to experts.
In my opinion this article does not have enough scientific value to be published in Education Sciences.
Answer:
First, the research is about: "In this study, students' textual peer feedback on video recordings of their teaching practice was analysed to determine the growth of their expertise in relation with blended curriculum design'. This was in the context of a project on video-supported collaborative learning.
Second, according to Wise and Schwarz, 2017). Computational approaches are a provocation in CSCL research. "It is the ability to look in fine-fine grained ways for complex patterns that might never be detected by hand" (p459). That is what we try to find with the lexical analysis and the KBDeX analysis in the context of bringing teacher students to form novice to-experts as teacher educators with a special focus on the video use by students and peer-feedback. "Collaborative learning is inherently defined as a process that occurs over time, yet traditional, quantitative research has ignored this character." (p 443). It is exactly this time dimension that we bring in the picture with the KBDeX analysis in the context of conceptual patterns that emerge in the abductive reasoning in the peer-feedback.
In our opinion, the study has value for the insights it gives in a more student active use of video; the use of a more innovative data-analysis method as a contribution to the challenges in the CSCL domain as mentioned by (Wise & Schwarz, 2017).
We have chosen a case study. A case study "may be central to scientific development via generalization as supplement or alternative to other methods." (Flyvbjerg, 2016). This method obtains information in real-life situations (Flyvbjerg, 2016) and allows to analyse patterns. With the KBDeX-analysis, we discovered these kinds of conceptual patterns in the use of 'professional jargon.' First, lexical richness increased steadily over time for all four subgroups. Second, relations in the word networks at the end of the course were stronger (indicated by thicker lines between the words) than in the word networks at the start.
Flyvbjerg, B. (2016). Five Misunderstandings About Case-Study Research. Qualitative Inquiry, 12(2). https://0-doi-org.brum.beds.ac.uk/10.1177/1077800405284363
Wise, A. F., & Schwarz, B. B. (2017). Visons of CSCL: eight provocations for the future of the field. International Journal of Computer Supported Collaborative Learning, 12(4), 423–467. https://0-doi-org.brum.beds.ac.uk/10.1007/s11412-017-9267-5
Reviewer The sample size is too small to make any kind of inference;
Answer:
As was mentioned in the limitations of the study, we are fully aware of that. The reviewer has in mind that the N consist of the 15. However, the students are not the unit of analysis, but the units of feedback that are given are the unit of analysis. "For the lexical analysis, we used the complete peer feedback dataset of the entire class group. We explored the students' peer feedback discourse by using the peer feedback and reflection data from each of the four student subgroups."
It can also be deduced from the regression ANOVA, and the MANOVA carried out in which the degrees of freedom are much larger. Besides that, the variance analyses test the hypothesis against the distribution of degrees of freedom, and therefore the N-size is taken into consideration. So, with a small N, effects must be large to become significant. When using large N-size small effects, not always practice relevant, might become significant. A larger N is important in case of generalization of the results; that is not the issue in this study.
At the beginning of the method is mentioned that we make use of a "pre-experimental one-group case study design." The qualitative analysis based on KBDeX patterns gives a rich insight into what happened in the different groups of students.
In the discussion, we are not really generalizing to a large population. Therefore, indeed a large study is needed with a more representative population. That was not our intention; we first wanted to explore if such a large population study is necessary. But even then, it might be difficult in ecological classroom settings to find a large experimental and control population considering the number of students following these VET-teacher courses as it is not exceptional in educational classroom studies.
Reviewer furthermore, the behaviour of students who know they are being filmed is not controlled, which may bias the results; as recognized by the authors there was no a control group which could allow an assessment of whether or not the students have actually progressed.
Answer:
The students were not filmed. The teacher-students are part-time students and already teaching in secondary VET-schools. They follow the part-time study to professionalize and become a graded teacher. The student-teachers video-recorded their own, authentic teaching practice in their own real secondary VET=teaching practice. So, it is not a matter of controlling their behaviour; it is the natural setting of their everyday practice. Their secondary VET students really do not allow bias in their teacher's behaviour. Also, the analysis and problems coming forward in the recordings are not one of a bias but everyday real practice. By the way, it is not the teaching behaviour that is analysed but the feedback given by peers on the video recorded everyday teaching practice of a peer.
Indeed, we did not use a control group because of the ecological course setting; we did not have the opportunity to form a control group. However, the study is part of more studies in different European countries within the ViSuAL-project, as is mentioned in the introduction. In those studies, similar effects are seen. They will be published, as well as an overall article in another journal. However, that article will not have the detailed and lexical analysis as the current article.
By taking the beginning, intermediate, and post-data time moments in the analysis, we can say if students' growth in their lexical richness and coherence over time as we could determine with the regression and MANOVA analysis. We agree that we cannot say if the growth is more or less in a more regular setting where no video and peer-feedback was used. Therefore, we indeed need a control group. The latter is important for further extended and larger studies. However, we can say that the use of the video-recorded practice of a peer and peer-feedbacked does not hold in and is at least enhanced in comparison with first meetings in the time of the for four moments in time during the progression of the course.
Reviewer Additionally, "there was only one teacher trainer involved and the influence of teaching style, experience and other personal characteristics was not investigated.
Answer:
This is correct. The study is not about the impact of teacher characteristics. Still, it concerns the analysis of students learning process concerning their use of 'jargon' from the professional as is coming forward in the literature to be studied in the lecturing. This is an indication of the growth of becoming an expert. Also, because only one teacher was involved, style, experience, personal characteristics, etc., were the same for all involved students. So, we did not find a reason to take this as different influencing factors in the study.
Of course, this would be different if in a larger scale study when more teachers are involved. However, this was not the case in the current study.
Reviewer Furthermore, the influence of collaborative learning by peer feedback was not investigated at an intensive level of collaborative learning as was intended." Consequently, there may be high subjectivity in peer feedback.
Answer:
We agree that we did not study an intensive level of collaborative learning. The intention was not to do that. The study was framed in the context of "of the Erasmus+ knowledge Alliance video supported collaborative learning (ViSuAL) project. The main objective of that project is to research pedagogy for using video in supporting collaborative learning. It was not the objective of the current study. The current study aimed to study the "support student teachers' development from' novice' to' starting expert' by using students' video recordings and peer feedback in a blended curriculum as was the only way to teach during the Covid19 period. And yes, we view the way the students gave peer-feedback online as a less intensive collaborative way of learning, but we still see it as "students' peer feedback discourse" coming forward in students' peer feedback and reflections.
We don't know if we understand the comment of 'high subjectivity' well. Of course, the feedback given by peers is based on personal history, experience, interpretations of the literature studied, and what they heard during the lectures. In that sense, it is remarkable that the 'subjectivity' in the sense of a kind of 'personal', 'lay' opinion as was mainly the case in the first period of feedback giving, gradually is enriched by the professional 'jargon' indication a positive answer to the question stated in the introduction "do peer feedback and reflection like 'discussion and dialogue' increase students' use of relevant knowledge they acquire from literature or lectures?
Also, in this stated question in the introduction, it might be clear that we did not intend to study an intensive level of collaboration or knowledge creation but a lower level of peer feedback and reflection in which different perspectives and experiences, and interpretations of students are exchanged and reflected. The latter are important and process in intensive collaboration, e.g., knowledge building as well.
We hope that the answers we gave are sufficient to brighten the picture. Because the answers are based on what is already written in the article, we did not see any leads to make any changes in the text based on the review comments. However, this does not mean that we did not change anything on the article, but it was more concrete for us to on the basis of the comments of the other reviewer comments. That makes that some of the methodological clarity is improved, which is in line with the comments of your review checklist.
Kind regards,
Marije, Erick and Frank.
Reviewer 2 Report
The paper is written clearly and consistently. The methodology used to obtain the results is detailed and the conclusions are supported by these results. Indicators to measure semantic cohesion and lexical richness; and its relationship with the Latent Dirichlet Allocation (LDA) are original and the conclusions are coherent and supported by an adequate statistical treatment.
However some minor corrections must be made:
- Section 1 introduction: Too long and uses repetitive language. The same can be said in a more clear and concise way.
- Starting section 1.2 with “However” is confusing.
- Section 2: At the beginning of the section, it should be explained briefly what KBDeX is and later expand it in point 2.7.
- Subsection 3.1: The interpretation of the subjects for subjective, 5,6,7 have intersections.
- Figures 13, 15, 17 and 19 are not named in any paragraph of the text.
Author Response
Please see the attachment.
Kind regards,
Marije, Erick and Frank
Author Response File: 
Author Response.pdf
Round 2
Reviewer 1 Report
Dear authors,
I agree with the improvements you have made to your article.
Kind regards
