Elaboration of Screening Scales for Mental Development Problems Detection in Russian Preschool Children: Psychometric Approach

Round 1

Reviewer 1 Report

Abstract:

Please provide more quantitative indices in addition to the forecast accuracy.

Introduction:

How could the authors provide justifications about the reliability of using the 3-year-old indices? Please discuss.

Methods:

How was the sample size estimated in this study?

What is the possible bias?

What are the inclusion and exclusion criteria?

The reviewer invites the authors to provide the items of the survey as supplementary.

Results:

Please provide the descriptive statistics table indicating the samples analyzed in the study in different sub-categories.

When reporting the discriminative capability of the method, training, and test sets must be used, and the performance indices of the test sets, including sensitivity, specificity, PPV, and accuracy, are reported. Moreover, based on the STARD guideline, the CI 95% of the performance indices must be reported.

Discussion:

The authors must discuss the limitations of the study, including possible bias and future directions.

Any method as the state-of-the-art for comparison?

Author Response

Dear Reviewer, thank you very much for your attention to our work ans useful recommendations. Our answers you will find in the uploaded file. Kind regards

Answers to the Reviewer 1

Deeply honored Reviewer,

First of all let us express our sincere gratitude for your precise attention to our work and your useful comments and suggestions.

• R Abstract: Please provide more quantitative indices in addition to the forecast accuracy.

Answer: The indicators of sensitivity (92.3%) and specificity (93.4%) obtained in this study are preliminary, since the sample of 3-year-old children with DD diagnosis is still small. More definite quantitative characteristics of forecast accuracy will be obtained after the approbation and wider usage of the screening.

• R Introduction: How could the authors provide justifications about the reliability of using the 3-year-old indices? Please discuss.

Answer: It is not clear what we are asked about. If by indices you mean developed scales, then their reliability in terms of items internal consistency is justified using the Cronbach's alpha.

• R Methods:

1. a) How was the sample size estimated in this study?

Answer: The data was collected in 26 preschool institutions using the computer system “Longitude”, which is regularly used in these institutions for annual surveys. The study used all the results of diagnostics of 3-year-old children, including children without a diagnosis (group Norm) and with an assessed diagnosis (groups DD/ASD), in the period from 2015 to 2019.

We made changes in the manuscript on p. 3, the lines 98-103 as a response on this question.

1. b) What is the possible bias?

Answer: We addressed the issue in the discussion section (see p. 10, the lines 283-289). Also, we added the limitations section where we addressed possible bias of our research and future directions. Respectively the changes were made on p.12, lines 378-386.

1. c) What are the inclusion and exclusion criteria?

Answer: The results of the routine examinations of all children for the entire specified period (2015-2019) were obtained. No results were excluded from the analysis.

1. d) The reviewer invites the authors to provide the items of the survey as supplementary.

Answer: The table on the distribution of the correct responses to 27 items for the Norm and DD samples was added in the manuscript on p. 4, the line 152 and also is presented below:

Also we give the link to the full content of items, tasks and stimuli materials which are available on the project's website http://info11.testpsy.net/ (see footnotes below Table 2, p. 6, the lines 162-163).

• R Results:

1. Please provide the descriptive statistics table indicating the samples analyzed in the study in different sub-categories.

Answer: Table 4 shows descriptive statistics and the number of samples surveyed (see p. 7, the line 207). Or is this question addressed to descriptive statistics for the original 27 items? The table of distribution of correct answers is given above and in the text of the article (p. 4, the line 152).

1. When reporting the discriminative capability of the method, training, and test sets must be used, and the performance indices of the test sets, including sensitivity, specificity, PPV, and accuracy, are reported. Moreover, based on the STARD guideline, the CI 95% of the performance indices must be reported.

Answer: Following your recommendation we added Table 7 “Indicators of test norms effectiveness” (see p. 10, the line 277). Performance indices will be available as the data on 3-year-olds who have already been diagnosed with DD is accumulated. In this article we have focused on the implementation of the psychometric approach in the development of screening. As we point out in the discussion (p. 10, the lines 279-283) and in the limitation section (p. 12, the lines 378-386).

• R Discussion:

1. The authors must discuss the limitations of the study, including possible bias and future directions.

Answer: Thanks for the recommendation. The reference to the possible study bias was added in the discussion section (p. 10, the lines 283-289). Also, the limitations section was added (p. 12, the lines 378-386).

1. Any method as the state-of-the-art for comparison?

Answer: Surprisingly and unfortunately, we searched for modern screenings far and wide but couldn’t find anything new except for long existing ones, such as Denver, Griffiths, Vineland. We quoted Denver in our article. Would you recommend us to quote some of the others?

Additional: In order to improve English language and style we will apply to the MDPI English editing service as soon as there no more correction needed of the manuscript’s content.

Reviewer 2 Report

My review will be based on the statistical methodology that was followed as this is the point of major consideration. While the authors present an analysis conducted skillfully, I am afraid the choice of methods it is not supported by current literature. 

To begin with, SPSS and AMOS have been used, meaning that the authors consider the data approximately normally distributed. Is that the case? IS the response scale five or more points on ordinal scale? Are the responses symmetrical? This is not clear in the manuscript. Otherwise please consider running a categorical factor analysis in MPlus, Stata or R.

Assuming that the common factor model is appropriate, the principal components method for factor extraction is definitely not. The method is not used anymore in the context of factor analysis due to its lack of acknowledging measurement error. The authors need to choose a better method based on the distribution of the data and on the existence of missing values. Also, why varimax rotation? It is not clear to me why one expects the different dimensions not to be related?

To the major commonly made  EFA mistakes, I would also like to draw attention to the CFA correlated error terms to ensure identification of the model. Unless if this correlation was a-priori expected or can be properly and clearly interpreted, adding correlated error terms is not recommended as it is data driven in a theory driven model, and it only shows a bad model. 

My advice to the authors is to seek advise for the analysis from a specialist psychometrician. 

Author Response

Dear Reviewer, thank you very much for your attention to our work and useful recommendations. Our answers you will find in the uploaded file. Kind regards

Answers to the Reviewer 2

Deeply honored Reviewer,

First of all let us express our sincere gratitude for your precise attention to our work and your useful comments and suggestions.

• R: To begin with, SPSS and AMOS have been used, meaning that the authors consider the data approximately normally distributed. Is that the case?

Answer: SPPS and AMOS allow you to work with the data which is not distributed normally due to the availability of the options that enable you to work with such data. For example categorical principal components analysis (CATPCA) or Asymptotically Distribution Free (Method of Estimate) in AMOS.

• R: Is the response scale five or more points on ordinal scale?

Answer: Thanks for your question. We clarified that all the items are binary (see p.3, the line 144).

• R: Are the responses symmetrical?

Answer: No, but the asymmetry is not so great (table 1 was included in the article on p. 4, the line 152) to assume that parametric factor analysis gives significant errors (see Kline R. 2011, p. 63). ). N548 has the largest asymmetry (skew=3.069), while the others have significantly less. However, taking into account your recommendations, we applied categorical principal components analysis.

• R: Otherwise please consider running a categorical factor analysis.

Answer: Thank you for the recommendation. We applied a categorical factor analysis and got not identical, but very similar results. The results of the analysis are included in the article on pp. 4-5, the lines 153-163.

R: Assuming that the common factor model is appropriate, the principal components method for factor extraction is definitely not. The method is not used anymore in the context of factor analysis due to its lack of acknowledging measurement error.

Answer: Discussions about the applicability of PCA as one of the methods of factor analysis are still ongoing (Thompson, 2004, Fabrigar, Wegener, MacCallum, & Strahan, 1999, Mulaik, 1992). Yes, of course, such FA methods as the Maximum Likelihood Method are much more theoretically justified, since they are aimed at the most accurate reproduction of the original correlations. But, taking into account the huge number of sources of errors in psychological measurements, the use of a simpler PCA is justified. In addition it can be added that when the number of measured variables is large - also with respect to the supposed number of latent variables (as is the case in our study) PCA and FA lead to the similar results (Thompson, 2004; Fabrigar et al., 1999).

• R: The authors need to choose a better method based on the distribution of the data and on the existence of missing values.

Answer: First of all, we don't have missing values, since each item is binary and the response can be either the child can perform the task or he/she cannot perform the task. Given the questionable symmetry of the distributions, we used your recommendation and applied CATPCA. The results are shown in Table 2 (on pp.  4-5, the lines 153-163), and although they differ from the previous ones, the difference is insignificant. Probably this is due to the large sample size. You will also find full printouts of the SYNTAX and OUTPUT: CATPCA at the end of our responses.

• R: Also, why varimax rotation? It is not clear to me why one expects the different dimensions not to be related?

Answer: Of course, the scales correlate, as indicated in Table 5 on p. 8, the line 219, and on Figure 1, p. 9. Nevertheless, We prefer orthogonal rotation options, as they have a more transparent interpretation. In addition, psychometrics has empirically developed the rule of "factor loading not less than 0.4" for including item into factor to obtain a reliable scale, after an orthogonal rotation (Kline, 2000; Cooper, 2010). In our case, the most optimal method was the Quartimax rotation with Kaiser Normalization (see Table 2 on pp. 4-5, the lines 153-163; also see full printouts at the end of our responses in this file).

• R: To the major commonly made  EFA mistakes, I would also like to draw attention to the CFA correlated error terms to ensure identification of the model. Unless if this correlation was a-priori expected or can be properly and clearly interpreted, adding correlated error terms is not recommended as it is data driven in a theory driven model, and it only shows a bad model. 

Answer: Of course, such surprises require either a priori assumptions or distinct a posteriori interpretations. We used CFA primarily to show that the 4-factor model is better than the 1-factor model. Regarding the assumption of correlations between errors for Model 4 there are many situations—particularly with respect to social psychological research where these parameters can make strong substantive sense and therefore should be included in the model (Joreskog, & Sorbom, 1993). In our case, correlations between errors are explained by the similarity of the corresponding task pairs. Correlate errors for tasks that are similar in one way or another: n629 – n288 (knows the seasons), n195 – n248 (draws a cross), n291 – n362 (can count), and so on.

If you are referring to the model shown in Figure 1, on p. 9, where 1 link is added between 2 errors, then we consider this acceptable if there are not very many such links. We added the interpretation in the discussion section on the p.11, the lines 313-317 as following: The weak link between errors (residuals) can be explained by the fact that when a child has a delay in a maturation of the frontal cortex, which is reflected in an underdevelopment of executive functions, the main compensatory mechanism for children with DD is the mastering of verbal abilities and first of all an expansion of general awareness.

Additional: In order to improve English language and style we will apply to the MDPI English editing service as soon as there no more correction needed of the manuscript’s content.

SYNTAX:

CATPCA VARIABLES=n187 n189 n238 n248 n260 n266 n270 n288 n289 n291 n295 n302 n325 n355 n361 n362

    n385 n548 n628 n629 n640 n655 n847 n890 n818 n195 n182

  /ANALYSIS=n187(WEIGHT=1,LEVEL=ORDI) n189(WEIGHT=1,LEVEL=ORDI) n238(WEIGHT=1,LEVEL=ORDI)

    n248(WEIGHT=1,LEVEL=ORDI) n260(WEIGHT=1,LEVEL=ORDI) n266(WEIGHT=1,LEVEL=ORDI)

    n270(WEIGHT=1,LEVEL=ORDI) n288(WEIGHT=1,LEVEL=ORDI) n289(WEIGHT=1,LEVEL=ORDI)

    n291(WEIGHT=1,LEVEL=ORDI) n295(WEIGHT=1,LEVEL=ORDI) n302(WEIGHT=1,LEVEL=ORDI)

    n325(WEIGHT=1,LEVEL=ORDI) n355(WEIGHT=1,LEVEL=ORDI) n361(WEIGHT=1,LEVEL=ORDI)

    n362(WEIGHT=1,LEVEL=ORDI) n385(WEIGHT=1,LEVEL=ORDI) n548(WEIGHT=1,LEVEL=ORDI)

    n628(WEIGHT=1,LEVEL=ORDI) n629(WEIGHT=1,LEVEL=ORDI) n640(WEIGHT=1,LEVEL=ORDI)

    n655(WEIGHT=1,LEVEL=ORDI) n847(WEIGHT=1,LEVEL=ORDI) n890(WEIGHT=1,LEVEL=ORDI)

    n818(WEIGHT=1,LEVEL=ORDI) n195(WEIGHT=1,LEVEL=ORDI) n182(WEIGHT=1,LEVEL=ORDI)

  /DIMENSION=4

  /NORMALIZATION=VPRINCIPAL

  /MAXITER=100

  /CRITITER=.00001

  /ROTATION=QUARTIMAX KAISER

  /RESAMPLE=NONE

  /PRINT=LOADING(SORT)

  /PLOT=OBJECT(20).

CATPCA - Principal Components Analysis for Categorical Data

Component Loadings

Round 2

Reviewer 1 Report

The reviewer would like to thank the authors to consider the entire comments. The paper is suitable for publication.