Gamifying Massive Online Courses: Effects on the Social Networks and Course Completion Rates

Round 1

Reviewer 1 Report

The article is good. Congratulation!

Idea is interesting and actual.

 

Author Response

Point1: The article is good. Congratulation! Idea is interesting and actual.

Response: Thank you for your comments

Reviewer 2 Report

The paper provides a study concerning the influence of games on Social Networks and an online course. 

The topic of the paper is interesting and inline with the topic of the journal. However, the paper presents several limitations. Some of them are expressed at the end of the paper, other are related to social activity.

- It is not clear how social activity could be considered as impressive to be a result of this paper. Authors mix several social activities together, like followers, following, comments, centralities, etc. Each social activity has a social impact. What is the social impact of each considered feature in this experiment?

- How Centrality is computed?

- Why authors choose centrality as a feature?

- Why F1 can be considered a good measure to predict the probability of learning success? I think that authors should provide at least other metrics and a comparison to highlight that F1 is a good measure.

I think the paper is not good enough for a publication, and several improvements should be provided in order to be considered for another evaluation.

 

 

Author Response

Response in attached PDF file.

Author Response File: Author Response.pdf

Reviewer 3 Report

My comments refer to the minor revisions to the paper. Overall, the paper is well written, and results section is insightfully written (that was the part that I really enjoyed reading).

Here are my comments:

• Lines 88 - 92, if you claim that something is "contrastingly" you shouldn't use "also" (whish implies equivalency or, at least similarity to the previous sentence), please, check
• Line 214 - Deci
• Methodology section:
  • The specifics of the gamification layer should be clearly stated at the begining or in the "instruments" section, as well as the measured variables. What will be measured? What are exact differences between the basic course and gamified one? Ever since the research questions (lines 150 - 153), the reader waits for the methodological justification for the use of the "influence", and first such notion is written in the line 246. Please, consider this.
  • The methodology should be more clearly represented (and preferably, this section would be shorter).
  • Also, you examine the students of the Digital Skills for Teachers course - don't you find the sample biased? Those students are already more motivated to learn and apply digital skills (and gamification, as one of the possible systems) by their choice of that course (in other words, the subjects are interested in the very subject of the study) and that should be stated as a limitation of the study. Such results should not be generalized.
  • In the section about analytical tools, where you introduce PCA, you should discuss does the data satisfy PCA assumptions (for example, outliers - were there any, how were they treated)
• line 318 - how many were excluded from further analysis (if any)?
• Lines 325 - 326 - "Given the number of edges, it is even difficult to find if two nodes are connected." - clarify
• Line 332 - 4.48 times larger?
• Lines 477 - 479 given the limitation, would you like to reconsider the research question, and change the word influence to a more appropriate one - as a relationship (or a similar one)?

Author Response

[Track changes was activated in manuscript]

Reviewer3

Point1:    Lines 88 - 92, if you claim that something is "contrastingly" you shouldn't use "also" (which implies equivalency or, at least similarity to the previous sentence), please, check

Response: Checked.

 

Point2:     Line 214 - Deci

Response: Done.

 

Point3.1:    Methodology section:         The specifics of the gamification layer should be clearly stated at the begining or in the "instruments" section, as well as the measured variables. What will be measured? What are exact differences between the basic course and gamified one? Ever since the research questions (lines 150 - 153), the reader waits for the methodological justification for the use of the "influence", and first such notion is written in the line 246. Please, consider this.

Response: We included a sentence at the beginning of the methodology section, just under research questions, to present the specific components of the gamification layer. We also describe briefly the measured provided by the educational social networking and by the gamification layer in the Instruments subsection (paragraph 1).

 

Point3.2        The methodology should be more clearly represented (and preferably, this section would be shorter).

Response: Section 3.2 was reduced. The justification of the gamification elements (final part of subsection 3.3. Instrument) was also substantially summarized. To be honest, we tried several arrangements and we did not find anyone more satisfactory than the others.

 

Point4:        Also, you examine the students of the Digital Skills for Teachers course - don't you find the sample biased? Those students are already more motivated to learn and apply digital skills (and gamification, as one of the possible systems) by their choice of that course (in other words, the subjects are interested in the very subject of the study) and that should be stated as a limitation of the study. Such results should not be generalized.

Response: We agree. Thank you for your suggestion. We included it as another limitation to generalization. Section 5, last paragraph, now reads as follows: “Another limitation of our study relates to generalization. Since the setting is a course on Digital Skills for teachers, participants may be more motivated to learn and apply any digital skill just by their choice. Gamification is the very subject of the study, but it is also another digital skill in which participants are just interested and participate actively.”

 

Point5:         In the section about analytical tools, where you introduce PCA, you should discuss does the data satisfy PCA assumptions (for example, outliers - were there any, how were they treated)

Response: The discussion about data assumptions of PCA was included in section 3.6 as suggested.

 

Point6:    line 318 - how many were excluded from further analysis (if any)?

Response: 287 in the experimental group and 269 in the control group. This is included in the text now. Please note that nodes with 0 connections are not included only in the graphical representation (figures 2 & 3). They were considered for further analysis since they may still have other social/learning activity, and pass the course.

 

Point7:    Lines 325 - 326 - "Given the number of edges, it is even difficult to find if two nodes are connected." – clarify

Response: The sentence was rewritten as “The density of connections makes difficult to see if two particular nodes are connected.”

 

Point8:    Line 332 - 4.48 times larger?

Response: Fixed.

 

Point9:    Lines 477 - 479 given the limitation, would you like to reconsider the research question, and change the word influence to a more appropriate one - as a relationship (or a similar one)?

Response: The research question was restated as: RQ2. How does gamification relates to course completion in a large-scale online course? Several minor updates were also included in different parts of the paper to reflect this change consistently.

 

Round 2

Reviewer 2 Report

The status of the paper has not been improved according to the previous suggestions.

It is clear, for me, what is centrality and in particular how and when centrality is used. It is also clear that it is computed on the graph. I asked why, because there should be a clear motivation to study centrality of a graph and why these specific measures have been included. It is not a good reply to say that the centrality is used in the paper because it is used in SNA.

Furthermore, authors say do not use social activity even if they are clearly evaluating actions such as follow, like, that are social activities with a meaning.

Author Response

Point1: It is clear, for me, what is centrality and in particular how and when centrality is used. It is also clear that it is computed on the graph. I asked why, because there should be a clear motivation to study centrality of a graph and why these specific measures have been included. It is not a good reply to say that the centrality is used in the paper because it is used in SNA.

Response: We acknowledge that our previous argument was not convincing. We think that literature review provides a good background of the use of centrality metrics in e-learning. We may have probably pointed to that instead ‘standard practice’. Since previous research suggests that centrality is associated with final grades and position influences learning (section 2, paragraph 2) and we are including gamification elements to the learning platform, our intention is to analyze the impact of these elements on the social network of participants and also the possible effect in learning. Further, there not many alternatives to centrality measures in SNA. Although Community Detection (segmentation) is certainly and alternative, existing literature on their influence in learning is scarce. Indeed, Clustering Coefficient, which we used, can be considered a measure of integration in the local community. To try to address Reviewer’s comment, we have included a short paragraph in section 3.1 that tries to motivate the use of centrality measures. In subsection 3.5 we briefly point to clustering coefficient as a measure of segmentation.

 

Point2: Furthermore, authors say do not use social activity even if they are clearly evaluating actions such as follow, like, that are social activities with a meaning.

Response: Sorry, again, for insisting… The objective of the paper is not to analyze or assess the meaning of each individual social action. Indeed, we are measuring 7 different social variables. A discussion about the meaning and impact of each individual action will result, in our humble opinion, in a detour (probably a long one) from research objectives that will not improve the paper.

 

Thank you for your comments and suggestions.