Affecting Young Children’s Knowledge, Attitudes, and Behaviors for Ultraviolet Radiation Protection through the Internet of Things: A Quasi-Experimental Study

Round 1

Reviewer 1 Report

In this manuscript the authors are studying the linking between prolonged exposure to ultraviolet (UV) radiation and skin cancer, with emphasis to children being more vulnerable to UV harmful effects compared to adults. In this context, current study implemented light sensors in a STEM inquiry-based learning environment focusing on UV radiation and protection in primary education. After this study, the authors add empirical evidence suggesting the value of real-time data-driven approaches implementing IoT devices to positively influence students’ knowledge and behaviors related to socio-scientific problems affecting their health.

Overall, this manuscript is a well-organized manuscript, which could be accepted upon minor revision, following the comments below:

• The study of the state-of-the-art analysis and related work is well established; however, I would like the authors to better categorize it and split in sub-sections.
• To conclude to a safe explanation of results, a bigger sample of participants should be collected. Maybe the authors could try to furtherly disseminate their questionnaires and gather a larger number of results.
• The overall analysis should be based on Machine Learning trained models, to outcome more reliable results, and perhaps the authors could use Causal AI techniques to identify the correlation, causality, and reasoning behind some of the results. Are the authors sure about the integrity of their outcomes, based upon the SPSS results? Perhaps a comparative analysis between different analytical tools could be performed, to verify the integrity – applicability of derived insights.
• I would like to see a more detailed list of actions from the authors, with regards how to address, minimize, or even tackle the problematic situations, following a detailed State of the Art analysis
• Overall, the authors miss the use of advanced Artificial Intelligence/Machine Learning techniques to make their analysis more defendable.

Author Response

Point 1: In this manuscript the authors are studying the linking between prolonged exposure to ultraviolet (UV) radiation and skin cancer, with emphasis to children being more vulnerable to UV harmful effects compared to adults. In this context, current study implemented light sensors in a STEM inquiry-based learning environment focusing on UV radiation and protection in primary education. After this study, the authors add empirical evidence suggesting the value of real-time data-driven approaches implementing IoT devices to positively influence students’ knowledge and behaviors related to socio-scientific problems affecting their health.

Overall, this manuscript is a well-organized manuscript, which could be accepted upon minor revision, following the comments below:

The study of the state-of-the-art analysis and related work is well established; however, I would like the authors to better categorize it and split in sub-sections.

Response 1: We thank Reviewer 1 for the comments and minor revisions that are requested. The state-of-the-art analysis and related work are better categorized in the revised manuscript and they are split in sub-sections in the Introductory part, to improve readability of the paper.

-----------------------------------------------------

Point 2: To conclude to a safe explanation of results, a bigger sample of participants should be collected. Maybe the authors could try to furtherly disseminate their questionnaires and gather a larger number of results.

Response 2: We agree with Reviewer 1 that a bigger sample of participants would strengthen the study. However, it is extremely difficult to disseminate the questionnaires and repeat the intervention in additional primary schools as due to the COVID19 pandemic the entrance of researchers in schools is currently prohibited. Moreover, getting access to additional schools would require an application for approval by the Ethics Committee of the Centre of Educational Research and Evaluation of the country in reference. Finally, as the study examines UV radiation, measurements exceeding the recommended safe for our health UV index are better taken during the summer months. During the summer months temperature measurements in the country under examination frequently exceed 40 degrees Celsius and the value of using a UV index is easily demonstrated. This was the main reason the study was conducted during May and June 2021. Increased temperatures are not observed during the fall and winter seasons in the country in reference. Therefore, for the three reasons outlined above, an additional data collection phase cannot take place before the summer of 2022.

To compensate for our inability to collect data from a bigger sample, the abstract was revised to emphasize that  this was an exploratory, small-scale study to investigate the effect of a STEM environment implementing IoT devices on 6th graders’ knowledge, attitudes, and behaviors about UV radiation and protection.

Please note that we acknowledge Reviewer’s 1 criticism of the small sample size and we clearly identify the small sample of the study as a weakness in the Limitations section of the paper where we state:

“The study was exploratory, used a small sample, and was based solely on quantitative data. The use of a larger sample and the addition of qualitative data in the form of in-depth student interviews or observations contacted by parents or teachers concerning children’s attitudes and behaviors regarding UV protection would strengthen the study”.

-----------------------------------------------------

Point 3: The overall analysis should be based on Machine Learning trained models, to outcome more reliable results, and perhaps the authors could use Causal AI techniques to identify the correlation, causality, and reasoning behind some of the results. Are the authors sure about the integrity of their outcomes, based upon the SPSS results? Perhaps a comparative analysis between different analytical tools could be performed, to verify the integrity – applicability of derived insights.

I would like to see a more detailed list of actions from the authors, with regards how to address, minimize, or even tackle the problematic situations, following a detailed State of the Art analysis

Overall, the authors miss the use of advanced Artificial Intelligence/Machine Learning techniques to make their analysis more defendable.

Response 3:

Machine learning has been attracting tremendous attention lately due to its predictive power but evidence suggests it is directly proportional to the size of the available datasets (Zhang & Ling, 2018). Advanced machine learning techniques are useful for finding complex relationships and hidden patterns in data consisting of many interdependent variables. The data environment must provision large quantities of raw data for discovery-oriented analytics practices such as machine learning techniques to be used.  

Our dataset consists of a total of 31 participants as this was a small-scale exploratory study and the variables for which statistically significant results are documents using SPSS (and confirmed using R) are essentially only 4 (pre-test on UV radiation knowledge, post-test on UV radiation knowledge, delayed post-test on UV radiation knowledge, pre-test on behaviors and delayed post-test on behaviors. Advanced machine learning techniques are therefore not applicable to our dataset.

With respect to the suggestion to use causal Artificial Intelligence techniques, our aim in this study is not to identify correlation or causality. Our results suggest a positive effect of the IoT approach that was used on students’ knowledge and behaviour but we cannot support causality as our results could be attributed to a number of confounding variables that were not measured in this small-scale study.

To address Reviewer’s 1 criticism the following sentence was acknowledged as a limitation and it was added in the appropriate section of the manuscript:

“Due to the small sample of the study the use of advanced Artificial Intelligence/Machine Learning techniques in order to provide more reliable results was not feasible”.   

To verify the integrity of our outcomes, based upon the SPSS results, a comparative analysis using a different analytical tool, namely R, a programming language and environment for statistical computing and graphics, was performed for statistically significant results and provided the same results. This was noted in the “data analysis” part of the paper. In all cases where a confirmatory analysis using R was performed a sentence was added in the “results” section indicating that results were identical.

Results obtained through R are also provided below:

data:  pretest and posttest

t = -3.6415, df = 14, p-value = 0.002669

alternative hypothesis: true difference in means is not equal to 0

95 percent confidence interval:

-4.872886 -1.260448

sample estimates:

mean of the differences

              -3.066667

> t.test(pretest, delayedtest, paired = TRUE, alternative = "two.sided")

                Paired t-test

data:  pretest and delayedtest

t = -4.1523, df = 14, p-value = 0.0009772

alternative hypothesis: true difference in means is not equal to 0

95 percent confidence interval:

-5.055110 -1.611557

sample estimates:

mean of the differences

              -3.333333

> t.test(behaviorpre, `behavior_delayed post`, paired = TRUE, alternative = "two.sided")

                Paired t-test

data:  behaviorpre and behavior_delayed post

t = -2.468, df = 14, p-value = 0.02709

alternative hypothesis: true difference in means is not equal to 0

95 percent confidence interval:

-0.69278931 -0.04854402

sample estimates:

mean of the differences

             -0.3706667

Zhang, Y., Ling, C. A strategy to apply machine learning to small datasets in materials science. npj Comput Mater 4, 25 (2018). https://0-doi-org.brum.beds.ac.uk/10.1038/s41524-018-0081-z

Reviewer 2 Report

Author Response

Point 1: Unfortunately, from my point of view, this article cannot be accepted in this section because it does not have research done in computer science.

The authors do only a kind of survey regarding the use of the Internet of

Things (IoT) devices to collect and analyze real-time UV radiation data, the

authors giving only a few data without revising the research in the field

(software/hardware/devices).

Response 1: We thank Reviewer 2 for the comment. We agree that the study does not involve research done in computer science. The study lies in the intersection of Computer Science and Social Sciences. The main contribution of the paper lies in that it demonstrates an innovative application of the Internet of Things in the field of education in a way that shows social impact rather than having a significant contribution in the computer science field. However, its topic might be of interest to both the Computer Science community and Education community.

To respond to Reviewer’s 2 comment, additional information with respect to the software, hardware and devices that were used in the study are added in the revised manuscript.

Round 2

Reviewer 2 Report

Author Response

Point 1: The article has been slightly improved and could be further improved.

I maintain the conclusion that this article folds better in another field and not Computers     

Response 1: We thank Reviewer 2 for the comment.

To respond to Reviewer’s 2 comment, minor corrections were made throughout the manuscript to increase readability and three references were added to justify limitations commonly encountered in social science quasi-experimental studies.