A Comparison of Parent Reports, the Mental Synthesis Evaluation Checklist (MSEC) and the Autism Treatment Evaluation Checklist (ATEC), with the Childhood Autism Rating Scale (CARS)

Round 1

Reviewer 1 Report

Dear Authors

Thank you for allowing me to read your article "A comparison of parent-reports Mental Synthesis Evaluation Checklist (MSEC) and Autism Treatment Evaluation Checklist (ATEC) with the Childhood Autism Rating Scale (CARS-2)". The manuscript is very well structured and well written. However, I have some questions that require clarification in the paper.

In section 3.2 the results of the EFA of the MSEC are presented and in the same way in the following section 3.3 the results of the CFA. According to the manuals in use, EFA and CFA should be performed on different samples (Brown, 2006). However, if a CFA is performed first and the hypothesized structure is not confirmed, then an EFA can be performed on the same sample to venture a possible dimensionality different from that proposed in the CFA. The same is true for the rest of the instruments analyzed.

I hope my comments are useful.

Best regards

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 2 Report

It is important to validate the effects of novel treatments for children with ASD and parents’ assessment can make a huge contribution in that. Compared to the CARS-2 tests that require a trained clinician, MSEC and ATEC assessments can be performed by parents in a more dynamic way over multiple assessments spanning many years to generate more meaningful outcomes. However, these tests have remained obscure to most researchers and in the present study, the authors have performed a well-designed study combined with detailed statistical analyses to provide a strong case for MSEC and ATEC as new tools for the ASD field. In my opinion, it is a very important study and is very relevant currently. I don’t have any major issue with this article, except I would recommend the authors to get it read again and edit minor language typos – like for example changing “meet” to “meat”. 

English is fine, just a few minor edits are required. Thanks!

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Reviewer 3 Report

This study compares two parental-reported scales, the Mental Synthesis Evaluation Checklist (MSEC) and the Autism Treatment Evaluation Checklist (ATEC), with the Childhood Autism Rating Scale (CARS-2) diagnostic tool for autism. Although the topic is interesting, the presentation could use some improvement. I have some suggestions that I would like to share.

Introduction:

First of all, I think that the introduction is quite long and could benefit from being more concise. Aim to introduce the topic, highlight the key points, and state the research objectives within a shorter space. To be able to make the introduction concise and less lengthy I suggest the following information to be reconsidered:

Briefly introduce them by name and purpose. Avoid mentioning specific scoring or sub-scales in the introduction and the detailed descriptions of the assessments (MSEC, ATEC, CARS) might be reconsidered.

Extensive statistics from previous studies and the way it is presented at the moment might be reconsidered. Instead of mentioning specific correlation coefficients and sample sizes, simply state that previous research showed varying and inconclusive results.

Briefly mention their benefits like cost-effectiveness, but avoid lengthy explanations unless it's crucial for your study's context. The justification for using parent-reported assessments is very lengthy.

In the updated introduction I suggest focusing on the overall gap the study addresses. As an example, I suggest emphasizing the lack of understanding of the relationship between MSEC, ATEC, and CARS in measuring ASD severity and language development.

I also suggest highlighting the novelty of the used approach. I also suggest emphasizing how understanding this relationship will improve assessment practices and intervention development for autism.

Methodology

This section is also very lengthy and there is information and repetition of information that might be reduced. I also have some concerns regarding the Autism diagnosis and the used scale for the diagnosis.

First of all, autism is considered a single diagnosis. In contrast, individuals with autism are very heterogeneous groups with different levels of functioning and it is vital to report the level of severity of autism. On the other hand, the used scale CARS2 (Childhood Autism Rating Scale second version), has three sections, CARS-HF (High Functioning Individuals), CARS-ST (Standard Version), and CARS-QPC (Questionnaire for Parents or Caregivers). Therefore, the scale has its own parental report scale which is very important to be reported here (CARS2-QPC). The scale also has two separate parts in which both high and low levels of Autism are possible to be diagnosed (ACRS2-ST, and CARS2-HF). If the second version of CARS is used these data are in hand and should be reported and the results of the (MSEC, ATEC) should be compared with the CARS2-QPC reports as well.

In sum, my impression is that the CARS and CARS2 are used interchangeably in this study and are considered as a similar scale while it is not.  

I also think that while 143 participants is a decent sample size, it might be helpful to mention if this meets the power analysis requirements for this study.

Results

I suggest Instead of presenting all correlations between individual CARS items and ATEC subscales, focus on the strongest and weakest correlations for each subscale and briefly discuss their meaning.

Consider presenting factor loadings in a table or figure for easier comparison.

I suggest using consistent formatting for presenting statistical results (e.g., decimal places, p-values).

Consistent formatting of statistical results is crucial for clear and professional presentation. Here are some examples:

Decimal places:

Correlations: Typically reported with two decimal places (e.g., r = 0.78, p < 0.001).

Means and standard deviations: Round to one more decimal place than the original data (e.g., if data is in whole numbers, round to one decimal place; if data has two decimal places, round to three decimal places).

Confidence intervals: Report to the same number of decimal places as the corresponding statistic (e.g., if the mean is reported to two decimal places, report the confidence interval to two decimal places).

It is written:

The CARS total score correlated strongly with parent-reports: complex-language comprehension MSEC (r(141) = 0.60, p < 0.0001), expressive language (r(141) = 0.66, p < 0.0001), sociability (r(141) = 0.58, p < 0.0001), sensory awareness (r(141) = 0.71, p < 0.0001), health (r(141) = 0.53, p < 0.0001), and the Total ATEC (r(141) = 0.75, p < 0.0001).

My suggestion is:

The CARS total score correlated strongly with parent-reported complex-language comprehension (MSEC, r = 0.60, p < 0.001), expressive language (r = 0.66, p < 0.001), sociability (r = 0.58, p < 0.001), sensory awareness (r = 0.71, p < 0.001), health (r = 0.53, p < 0.001), and the Total ATEC score (r = 0.75, p < 0.001).

Discussion

I think this section needs to be reorganized in light of the suggestions for improvement in the above section. I think the authors need to consider that the study was correlational, so it cannot be used to determine whether the MSEC or ATEC causes changes in ASD severity.

The study did not include a control group, so it is possible that the observed changes were due to factors other than the interventions.

The study only followed participants for a short period (6 months), so it is not known whether the effects of the interventions would be sustained over time.

Overall, this study provides evidence that parent-reported outcome measures can be a valuable tool for assessing the effects of interventions for children with ASD. However, it is important to note that these measures should not be used as the sole measure of a child's progress. Clinician-administered measures and other objective measures should also be used to provide a comprehensive picture of a child's development.

The study has no limitations section. Still, I think that there are many serious limitations to the present manuscript concerning its methodology. Some very specific examples are “Sample size: While larger than some studies, 143 participants are still relatively small for generalizing findings to the entire ASD population.

Sample characteristics: The study focused on a broad age range (2-22 years) and did not control for other factors like ASD severity or specific diagnoses within the spectrum, potentially affecting results.

Cross-sectional design: This design only captures a snapshot in time and limits conclusions about cause-and-effect relationships between interventions and changes in MSEC/ATEC scores.

Limited intervention details: The study lacks information about specific interventions used, making it difficult to link observed changes to particular approaches.

Considering autism as one single diagnosis regardless of the severity of the symptoms and level of functioning are critical items in the diagnosis of autism.

Short follow-up period: Six months might not be enough to capture meaningful long-term effects of interventions.

With respect to the measures used in this study, the limitations are:

Measures:

MSEC and ATEC are parent-reported: While convenient, self-report measures can be susceptible to bias and limited objectivity.

ATEC Health subscale: The lack of unidimensionality in this subscale raises concerns about its reliability and validity for specific health factors.

It is important to note that the version of the CARS diagnostic scale being used (CARS or CARS-2) is not specified. Although the CARS-2 version is well-established, it is a time-consuming process and requires application training. Additionally, it is considered a level C scale and can be expensive to administer, which can limit its feasibility in certain settings.

In summary, the manuscript requires reorganization. The introduction and methodology sections contain detailed information that needs to be reduced, while some critical information has been omitted from the manuscript.

There were some typos and the text needs to be proofread.

Author Response

Please see the attachment

Author Response File: Author Response.pdf

Round 2

Reviewer 3 Report

This manuscript represents a significant improvement over the previous submission, as the authors have addressed the raised issues and produced a more focused and concise paper.