Next Article in Journal
A Note on the Nonparametric Estimation of the Conditional Mode by Wavelet Methods
Previous Article in Journal
Local Processing of Massive Databases with R: A National Analysis of a Brazilian Social Programme
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Stats
Volume 3
Issue 4
10.3390/stats3040029
stats-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Psychometric Properties of the Adult Self-Report: Data from over 11,000 American Adults

by
Michelle Guerrero
1,*,
Matt Hoffmann
2 and
Laura Pulkki-Råback
3,4
1
Healthy Active Living and Obesity Research Group, Children’s Hospital of Eastern Ontario Research Institute, Ottawa, ON K1H 8L1, Canada
2
Department of Kinesiology, California State University, Fullerton, CA 92831, USA
3
Department of Psychology and Logopedics, Faculty of Medicine, University of Helsinki, 00100 Helsinki, Finland
4
Department of Clinical Medicine, Division of Child Psychiatry, University of Turku, 20014 Turku, Finland
*
Author to whom correspondence should be addressed.
Stats 2020, 3(4), 465-474; https://doi.org/10.3390/stats3040029
Submission received: 8 September 2020 / Revised: 8 October 2020 / Accepted: 25 October 2020 / Published: 29 October 2020
(This article belongs to the Special Issue Statistics in Epidemiology)
Versions Notes

Abstract

:
The first purpose of this study was to examine the factor structure of the Adult Self-Report (ASR) via traditional confirmatory factor analysis (CFA) and contemporary exploratory structural equation modeling (ESEM). The second purpose was to examine the measurement invariance of the ASR subscales across age groups. We used baseline data from the Adolescent Brain Cognitive Development study. ASR data from 11,773 participants were used to conduct the CFA and ESEM analyses and data from 11,678 participants were used to conduct measurement invariance testing. Fit indices supported both the CFA and ESEM solutions, with the ESEM solution yielding better fit indices. However, several items in the ESEM solution did not sufficiently load on their intended factors and/or cross-loaded on unintended factors. Results from the measurement invariance analysis suggested that the ASR subscales are robust and fully invariant across subgroups of adults formed on the basis of age (18–35 years vs. 36–59 years). Future research should seek to both CFA and ESEM to provide a more comprehensive assessment of the ASR.

1. Introduction

The Achenbach System of Empirically Based Assessment (ASEBA) is a comprehensive evidence-based assessment system designed to measure adaptive and maladaptive functioning across individuals from age 1½ to over 90 years [1]. Quantitative data from ASEBA-inventories can be conceptualized into functioning scales, syndrome scales, substance use scales, Diagnostic and Statistical Manual of Mental Disorders (DSM)-oriented scales, and critical item scales (items used primarily by clinicians). ASEBA-inventories have been translated in over 110 languages and referenced and used in more than 8000 publications. ASEBA inventories are widely used in various settings, including education, research, clinical, and healthcare (mental health and medical).
The Adult Self-Report (ASR) is one component of the ASEBA [2]. This self-report inventory comprises eight syndrome scales: anxious/depressed, withdrawn, somatic complaints, thought problems, attention problems, aggressive behavior, rule-breaking behavior, and intrusiveness. The syndrome scales can be examined independently, or aggregated into two broad-band scales, internalizing (i.e., anxious/depressed, withdrawn, and somatic complaints) and externalizing (i.e., aggressive behavior, rule-breaking behavior, and intrusiveness).
One gap within the ASR literature is that few studies have examined its factor structure [3,4]. The eight syndrome scales were initially supported via confirmatory factor analysis (CFA) during the development of the ASR [2]. Ivanova et al. [5] more recently used CFA to test the generalizability of the eight-syndrome model across 29 societies, such as Brazil, China, Russia, and the UK. Results showed that the correlated eight syndrome model converged for all 29 samples. Specifically, the Root Mean Square Error of Approximation (RMSEA) values demonstrated good model fit for all societies (<0.06). The Comparative Fit Index (CFI) and Tucker Lewis Index (TLI) values indicated inadequate model fit for 13 of the 29 societies, with values falling below the recommended 0.90 threshold. Ivanova et al.’s work provides an up-to-date overview of the ASR’s generalizability. However, their study did not include American adults. More research using data from the United States is needed to support and extend our knowledge of the ASR’s psychometric properties. This is especially true as newer, more contemporary analytical techniques are advanced to test the psychometric properties of instruments.
While CFA has traditionally been used to assess the factorial validity of an instrument, there are criticisms of this method. Mainly, cross-loadings on non-targeted factors are assumed to be zero in CFA, which is considered overly restrictive in social sciences and idealistic [6,7]. When cross-loadings are constrained to zero, CFA will result in inflated factor correlations, which can ultimately undermine validity evidence (e.g., discriminant, predictive) of a multidimensional instrument [8]. Fortunately, more contemporary methods have been proposed in recent years to remedy the limitations of established analytic techniques such as CFA.
Exploratory structural equation modeling (ESEM) is one alternative to CFA [8]. ESEM is a flexible technique that integrates exploratory factor analysis, CFA, and structural equation modeling. The primary strength of ESEM is that it allows for minor cross-loadings, while simultaneously preserving the advantages of CFA (e.g., invariance testing) [8]. Allowing cross-loadings does not undermine constructs by adding ‘noise’ but rather allows them to be estimated using all of the relevant information [9]. ESEM studies comparing CFA and ESEM models across various domains have often reported more favorable results of the ESEM model [7,10,11].
Measurement invariance of the ASR subscales is another disparity in the literature. Studies that have examined whether ASR subscales function equivalently across groups have been primarily conducted with dyadic data (i.e., romantic couples) [4,12]. This research has generally shown that the ASR subscales are conceptualized differently (i.e., noninvariant) across self- and partner-reports as well as across males and females. Only one study has examined age measurement invariance, testing equivalency on one of the eight ASR syndrome scales (i.e., thought problems) [13]. Results showed that the thought problems subscale was invariant across three age groups (12–18 years, 19–27 years, and 28–59 years). Taken together, these findings can have significant implications for research and applied work.
Analyzing the factorial structure and measurement invariance of the ASR may extend the instrument’s validity and provide further support for its psychometric properties in the American population. Therefore, the first purpose of the current study was to compare CFA and ESEM approaches. The second purpose was to investigate the invariance of the syndrome scales of the ASR across age groups. To the best of our knowledge, this is the first study to use ESEM with the ASR and one of the few to test for measurement invariance across age groups.

2. Materials and Methods

2.1. Data Source

We used data from the Adolescent Brain Cognitive Development (ABCD) study, a broadly representative ongoing 10-year longitudinal study examining psychological and neurobiological changes from preadolescence to early adulthood. Baseline assessments (2016 and 2018) included recruiting 11,875 children (9–10 years) and their parents/guardians from 21 sites across the United States. Baseline data revealed that the sample was relatively diverse on several socio-demographic factors: 59% of parents had a bachelor’s or postgraduate degree; household income was <$50,000 for 30% of the families and between $50,000–99,999 for 26% of the families; 52% of families identified as White, 15% as Black, 20% as Hispanic, 2% as Asian, and 11% as other (e.g., biracial). Complete recruitment details can be found elsewhere [14]. Study procedures received ethics board approval. Prior to participation, parents/guardians provided written informed consent and children provided assent.

2.2. ASR

Participants answered the 99 items corresponding to the ASR’s eight syndrome scales [2]: anxious/depressed (18 items), withdrawn (9 items), somatic complaints (12 items), thought problems (10 items), attention problems (15 items), aggressive behavior (15 items), rule-breaking behavior (14 items), and intrusiveness (6 items). Items are rated on a 3-point scale: 0 = not true, 1 = somewhat or sometimes true, or 2 = very true or often true. Participants were instructed to answer each item based on the preceding 6 months.

2.3. Statistical Analysis

For the main analysis, we conducted CFA and ESEM using the latent variable modeling program Mplus Version 8.3. Data for these analyses were based on responses from 11,773 parents. A total of 102 participants were removed from the analyses because of incomplete data on ASR items (n = 5) or they exceeded the age threshold (i.e., >60 years; n = 97). We used the weighted least squares mean and variance-adjusted (WLSMV) estimator, which is suitable for ordered categorical (ordinal) variables wherein the assumption of normality is not assumed [15]. We used target rotation for the ESEM solution and set all unintended factor loadings (i.e., cross-loadings) to be close to zero. We used the following fit indices to evaluate the model fit of both the CFA and ESEM solutions: CFI, TLI, and RMSEA. Model fit was deemed acceptable if CFI and TLI values were ≥0.90 and RMSEA was ≤0.08 [16,17]. Model fit was considered good if CFI and TLI values were ≥0.95 and RMSEA was ≤0.06 [18]. We also reported the Chi-Square statistic (χ2), but because of its sensitivity to large sample sizes [19], we did not use it to gauge model fit. Factor loadings for both the CFA and ESEM solutions were required to be ≥0.30 for items to be considered salient and used in defining constructs. Loadings >0.71 were considered excellent, >0.63 very good, >0.55 good, >0.45 fair, and ≥0.30 poor [20,21]. Whereas cross-loadings were automatically zero for the CFA solution, they were required to be <0.30 for the ESEM solution. Had both the CFA and ESEM solutions met all criteria for an adequate fitting model, our intention was to compare the two solutions by examining the model fit statistics for both models. However, as described in detail in the Results section, the ESEM solution demonstrated issues with some of the factor loadings. Consequently, we deemed the CFA solution to be the more favorable model.
Because we deemed the CFA solution as the more favorable model, we tested for measurement invariance by examining changes in model fit statistics using CFA. Data for the invariance testing were based on responses from 11,678 participants (Mage = 39.76, SD = 6.48). A total of 95 participants did not self-report their age and were therefore excluded from the analysis. We tested for configural, metric, and scalar invariance to determine whether the underlying factor structure of the ASR was the same across age groups (i.e., 18–35 years [n = 3094] vs. 36–59 years [n = 8584]). These age categories were selected in order to align with the normed scales for syndromes outlined in the ASEBA manual [2]. Configural invariance assesses the fit of the factor structure when there are no invariance constraints imposed across groups. Metric invariance assesses the invariance of factor loadings across groups. Scalar invariance assesses the invariance of both factor loadings and item intercepts across groups. We assessed metric invariance after configural invariance was established, and scalar invariance after metric invariance was established. Thus, this process involved testing a series of increasingly constrained models. There is support for a more constrained model when the CFI decreases by less than 0.010 and the RMSEA increases by less than 0.015 [22]. There is strong measurement invariance when all three tests demonstrate invariance across groups [23].
Last, we used standardized factor loadings to calculate composite reliability scores for the latent factors [24]. We assessed reliability using a composite reliability index. Given the persistent use of alpha coefficients as a measure of reliability, alpha coefficients were also calculated, though it should be noted that alpha has been increasingly criticized as a measure of reliability [25,26].

3. Results

The results of the CFA indicated that the 8-factor model fit acceptably well based on CFI (0.919) and TLI (0.917) values and very well based on the RMSEA (0.026; 90% CI [0.026–0.026]), χ2 (4724) = 42,010.26, p < 0.001. All standardized factor loadings for the CFA were significant (p < 0.001), ranging from 0.36 to 0.90 (see Table 1).
The 8-factor model using ESEM had very good model fit based on the CFI (0.970), TLI (0.965), and RMSEA (0.017; 90% CI [0.017–0.017]), χ2 (4087) = 17,706.41, p < 0.001. However, not all items loaded cleanly on their intended factors. The anxious/depressed factor had one intended item that loaded <0.30, and three intended items that cross-loaded ≥0.30 on unintended factors. The withdrawn factor had two intended items that loaded <0.30, and one intended item that cross-loaded >0.30 on an unintended factor. The somatic complaints factor had one intended item that loaded <0.30, yet no intended items that cross-loaded ≥0.30 on unintended factors. The thought problems factor was particularly problematic; seven of ten intended items loaded <0.30, and three intended items cross-loaded ≥0.30 on unintended factors. The attention problems factor had three intended items that loaded <0.30, and two intended items that cross-loaded >0.30 on an unintended factor. The aggressive behavior factor had three intended items that loaded <0.30, and one intended item that cross-loaded >0.30 on an unintended factor. The rule-breaking behavior factor was quite problematic; six of fourteen intended items loaded <0.30, and four intended items cross-loaded ≥0.30 on unintended factors. All intended items for the intrusiveness factor loaded >0.30, and one intended item cross-loaded >0.30 on an unintended factor. Important to note is that in some cases, secondary loadings were larger than the primary loadings. Factor loadings (including cross-loadings) for the ESEM solution are reported in Table 1.
The results of measurement invariance testing using CFA demonstrated evidence of acceptable model fit. All changes in fit indices pertaining to configural, metric, and scalar invariance were well within acceptable ranges [22]. Consequently, we found support for strong measurement invariance across age groups (i.e., 18–35 years vs. 36–59 years). These results are summarized in Table 2.
Descriptive statistics, as well as latent factor correlations (polychoric) and internal consistencies for the CFA and ESEM solutions, are reported in Table 3. Reliability scores for the CFA solution were good (0.83 to 0.95 for composite reliability and 0.82 to 0.95 for alpha coefficient) and scores for the ESEM solution ranged from poor to good (0.59 to 0.88).

4. Discussion

Scholars have argued that the classic CFA models may not be appropriate for many well-established psychological measures. ESEM is a new and evolving methodological approach that has proven to overcome the typical shortcomings of CFA. The purpose of the current study was to compare the factor structure of the ASR using both CFA and ESEM and to test for age measurement invariance. Results of the current study showed that both measurement models met conventional model fit criteria, with the ESEM solution showing an improvement in fit over CFA but demonstrating poor factor loadings on intended factors and substantial cross-loading on unintended factors. Furthermore, invariance testing revealed that the ASR is robust and invariant across two age groups.
The ESEM solution showed an improvement in model fit statistics over the CFA solution. This finding is not surprising given that ESEM yields better fit in the presence of nontrivial cross-loadings [27,28]. However, several (n = 26) items in the ESEM solution did not sufficiently load on their intended factors (<0.30) and/or cross-loaded on unintended factors (≥0.30). There were nine instances wherein the cross-loading on the unintended factor was greater than the factor loading on the intended factor. Two factors that were particularly problematic were thought problems and rule-breaking behavior, with more than half of the items (13 of 24) being problematic. Moreover, factor loadings on intended factors were substantially lower in the ESEM solution compared to the CFA solution. The strength of factor loadings on intended factors in the ESEM model generally ranged from ‘poor’ to ‘fair’ and very few were considered ‘excellent’, whereas the strength of factor loadings in the CFA solution typically hovered around ‘good’ and ‘excellent’. Overall, results of the ESEM solution revealed that: (a) factor loadings on intended factors were generally weak; (b) factor loadings did not sufficiently load on intended factors; (c) there were substantial cross-loadings on unintended factors; (d) in some cases, secondary loadings were larger than the primary loadings.
Many studies examining the factorial validity of psychological measures seek to compare results from traditional CFA to the newer method of ESEM. In these studies, the conclusion that the ESEM solution was superior to the CFA solution is more common than not [11]. However, in some cases [29], and certainly in the current study, ESEM may not always provide the best overall solution—a notion acknowledged within the statistical literature [6]. Booth and Hughes [29] used ESEM as an alternative to CFA and found little evidence supporting the application of ESEM over CFA. They acknowledged that there are several nuances that must be considered when selecting between the two analytic strategies. For example, they discuss how in ESEM the inclusion of all possible parameters ensures that no substantive parameters are omitted, but that this approach may also be interpreted as overly inclusive and contradictory to the principle of parsimony. In their study, additional parameters included in the ESEM solution added very little to the definition of latent constructs. Highlighting the advantages and disadvantages of both CFA and ESEM, Booth and Hughes illustrate how CFA and ESEM can be used together to improve measurement of constructs.
The results of measurement invariance analysis suggested that the ASR is robust and fully invariant across subgroups of adults formed on the basis of age (18–35 years vs. 36–59 years). That is, the factor structure was the same regardless of the adults’ age (configural invariance), the strength of the factor loadings were similar across age (metric invariance), and no significant age differences were noted in the intercepts (full scalar invariance). This finding is consistent with previous research that showed the thought problems subscale was invariant across three age cohorts (12–18 years, 19–27 years, and 28–59 years) [13]. Results of the current study are encouraging and confirm that the ASR’s psychometric properties are robust across adulthood.
The results of this study should be considered in light of some limitations. Results provide evidence of the ASR’s factorial validity only. Subsequent research should consider examining other types of construct validity, such as criterion or concurrent. Furthermore, we were unable to test for gender invariance because data on the participants’ gender were unavailable. Existing research with dyadic data (romantic couples) has shown that the ASR subscales are noninvariant across females and males [12]. Specifically, five subscales (withdrawn, attention problems, aggressive behavior, rule-breaking behavior, and intrusiveness) met full metric invariance, two subscales (anxious/depressed and somatic complaints) met partial metric invariance, and all subscales met partial scalar invariance [12]. Future research should seek to explore the measurement invariance of the ASR across other factors (e.g., socio-demographic status) as well as across multiple time points as more ABCD data is released.
The results of the current study should be carefully considered in future studies using the ASR. Results of the ESEM solution uncovered important findings regarding factor loadings on intended and unintended factors, especially for the thought problems and rule-breaking behavior syndromes. The ESEM results therefore may raise concern for some regarding the sole reliance on CFA to examine the factor structure of the ASR. It is hoped that future researchers use CFA and ESEM in harmony rather than in isolation. Previous works examining the factor structure of psychological measures have successfully illustrated how ESEM and CFA can be used to complement each other [28,29]. Future research should also explore the psychometric properties of the ASR using other modeling approaches, such as partial least squares SEM, bifactor models that incorporate a general factor encompassing all syndromes [30] and Bayesian structural equation modeling (BSEM) [31]. The latter approach is similar to ESEM in that it allows cross-loadings. A key difference between ESEM and BSEM is that prior knowledge (referred to as priors) can inform the statistical model via statements regarding model parameters (e.g., mean, path coefficient). Thus, the model is tested against a set of known parameters rather than against a null hypothesis. An obvious challenge for researchers seeking to use BSEM is the choice of the priors, since using improper priors can have a negative impact on estimates [31]. Another advantage of BSEM is that it performs well with small sample sizes.

5. Conclusions

Findings showed that both the eight-syndrome CFA and ESEM solutions met model fit criteria.
As anticipated and in line with previous research, the ESEM solution revealed an improvement in fit statistics and lower factor correlations than the CFA solution; however, examination of the item factor loadings revealed that several items had poor factor loadings on intended factors and/or significant cross-loadings on unintended factors. Future research should seek to use both CFA and ESEM to provide a more comprehensive assessment of the ASR. Measurement invariance testing revealed that the ASR subscales are equivalent for younger and older adults. To the best of our knowledge, this study was the first to explore the factor structure of the ASR using ESEM and the first to test measurement invariance of the individual items from all ASR subscales across age groups.

Author Contributions

Conceptualization, M.G. and L.P.-R.; methodology, M.G. and L.P.-R.; software, M.H.; validation, M.H.; formal analysis, M.G. and M.H.; writing—original draft preparation, M.G. and M.H.; writing—review and editing, M.H. and L.P.-R.; visualization, M.H.; supervision, L.P.-R. All authors have read and agreed to the published version of the manuscript.

Funding

The Jenny and Antti Wihuri Foundation and The Yrjo Jahnsson Foundation, Finland to L.P.-R.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Achenbach, T.M. As Others See Us Clinical and Research Implications of Cross-Informant Correlations for Psychopathology. Curr. Dir. Psychol. Sci. 2006, 15, 94–98. [Google Scholar] [CrossRef]
  2. Achenbach, T.M.; Rescorla, L.A. Manual for the ASEBA Adult Forms & Profiles; University of Vermont Research Center for Children, Youth, & Families: Burlington, VT, USA, 2003. [Google Scholar]
  3. Michelini, G.; Barch, D.M.; Tian, Y.; Watson, D.; Klein, D.N.; Kotov, R. Delineating and Validating Higher-Order Dimensions of Psychopathology in the Adolescent Brain Cognitive Development (ABCD) Study. Transl. Psychiatry 2019, 9, 21–25. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. Claxton, S.E.; Deluca, H.K.; Van Dulmen, M.H.M. Testing Psychometric Properties in Dyadic Data Using Confirmatory Factor Analysis: Current Practices and Recommendations. Test. Psychom. Methodol. Appl. Psychol. 2015, 22, 181–198. [Google Scholar]
  5. Ivanova, M.Y.; Achenbach, T.M.; Rescorla, L.A.; Turner, L.V.; Ahmeti-Pronaj, A.; Au, A.; Maese, C.A.; Bellina, M.; Caldas, J.C.; Chen, Y.C.; et al. Syndromes of Self-Reported Psychopathology for Ages 18–59 in 29 Societies. J. Psychopathol. Behav. Assess. 2015, 37, 171–183. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  6. Marsh, H.W.; Morin, A.J.S.; Parker, P.D.; Kaur, G. Exploratory Structural Equation Modeling: An Integration of the Best Features of Exploratory and Confirmatory Factor Analysis. Annu. Rev. Clin. Psychol. 2014, 10, 85–110. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  7. Marsh, H.W.; Liem, G.A.D.; Martin, A.J.; Morin, A.J.S.; Nagengast, B. Methodological Measurement Fruitfulness of Exploratory Structural Equation Modeling (ESEM): New Approaches to Key Substantive Issues in Motivation and Engagement. J. Psychoeduc. Assess. 2011, 29, 322–346. [Google Scholar] [CrossRef]
  8. Asparouhov, T.; Muthén, B. Exploratory Structural Equation Modeling. Struct. Equ. Model. 2009, 16, 397–438. [Google Scholar] [CrossRef]
  9. Asparouhov, T.; Muthén, B.; Morin, A.J.S. Bayesian Structural Equation Modeling with Cross-Loadings and Residual Covariances. J. Manag. 2015, 41, 1561–1577. [Google Scholar] [CrossRef] [Green Version]
  10. Hoffmann, M.D.; Loughead, T.M. Preliminary Development of a Questionnaire to Assess Peer Athlete Mentoring Functions: The Athlete Mentoring Questionnaire (AMQ). Meas. Phys. Educ. Exerc. Sci. 2019, 23, 10–25. [Google Scholar] [CrossRef]
  11. Joshanloo, M.; Jose, P.E.; Kielpikowski, M. The Value of Exploratory Structural Equation Modeling in Identifying Factor Overlap in the Mental Health Continuum-Short Form (MHC-SF): A Study with a New Zealand Sample. J. Happiness Stud. 2017, 18, 1061–1074. [Google Scholar] [CrossRef]
  12. DeLuca, H.K.; Sorgente, A.; Van Dulmen, M.H.M. Dyadic Invariance of the Adult Self-Report and Adult Behavior Checklist: Evidence from Young Adult Romantic Couples. Psychol. Assess. 2019, 31, 192–209. [Google Scholar] [CrossRef]
  13. Abdellaoui, A.; De Moor, M.H.M.; Geels, L.M.; Van Beek, J.H.D.A.; Willemsen, G.; Boomsma, D.I. Thought Problems from Adolescence to Adulthood: Measurement Invariance and Longitudinal Heritability. Behav. Genet. 2012, 42, 19–29. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  14. Garavan, H.; Bartsch, H.; Conway, K.; Decastro, A.; Goldstein, R.Z.; Heeringa, S.; Jernigan, T.; Potter, A.; Thompson, W.; Zahs, D. Recruiting the ABCD Sample: Design Considerations and Procedures. Dev. Cogn. Neurosci. 2018, 32, 16–22. [Google Scholar] [CrossRef]
  15. Li, C.H. Confirmatory Factor Analysis with Ordinal Data: Comparing Robust Maximum Likelihood and Diagonally Weighted Least Squares. Behav. Res. Methods 2016, 48, 936–949. [Google Scholar] [CrossRef] [Green Version]
  16. Brown, T. Confirmatory Factor Analysis for Applied Research, 2nd ed.; Guilford Press: New York, NY, USA, 2015. [Google Scholar]
  17. Browne, M.W.; Cudeck, R. Alternative Ways of Assessing Model Fit. Sociol. Methods Res. 1992, 21, 230–258. [Google Scholar] [CrossRef]
  18. Hu, L.T.; Bentler, P.M. Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria versus New Alternatives. Struct. Equ. Model. 1999, 6, 1–55. [Google Scholar] [CrossRef]
  19. Kline, R.B. Principles and Practice of Structural Equation Modeling, 3rd ed.; Guilford Press: New York, NY, USA, 2011. [Google Scholar]
  20. Comrey, A.L.; Lee, H.B. A First Course in Factor Analysis, 2nd ed.; Erlbaum: Hillsdale, NJ, USA, 1992. [Google Scholar]
  21. Tabachnick, B.G.; Fidell, L.S. Using Multivariate Statistics, 6th ed.; Pearson: Boston, MA, USA, 2012. [Google Scholar]
  22. Chen, F.F. Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance. Struct. Equ. Model. 2007, 14, 464–504. [Google Scholar] [CrossRef]
  23. Marsh, H.W.; Parker, P.D.; Morin, A.J.S. Invariance Testing across Samples and Time: Cohort-Sequence Analysis of Perceived Body Composition. In An Introduction to Intermediate and Advanced Statistical Analyses for Sport and Exercise Scientists; Ntoumanis, N., Myers, N., Eds.; John Wiley & Sons: Hoboken, NJ, USA, 2016; pp. 101–130. [Google Scholar]
  24. Raykov, T. Estimation of Composite Reliability for Congeneric Measures. Appl. Psychol. Meas. 1997, 21, 173–184. [Google Scholar] [CrossRef]
  25. McNeish, D. Thanks Coefficient Alpha, We’ll Take It From Here. Psychol. Methods 2017. [Google Scholar] [CrossRef]
  26. Martin, J.J.; Beasley, V.L.; Guerrero, M.D. Sport Psychology Research: Proper Standards and Limitations. In APA Handbooks in Psychology Series: APA Handbook of Sport and Exercise Psychology; Ansel, M., Petrie, T., Steinfeldt, J., Eds.; American Psychological Association: Washington, DC, USA, 2019; Volume 1, pp. 17–40. [Google Scholar]
  27. Marsh, H.W.; Nagengast, B.; Morin, A.J.S.; Parada, R.H.; Craven, R.G.; Hamilton, L.R. Construct Validity of the Multidimensional Structure of Bullying and Victimization: An Application of Exploratory Structural Equation Modeling. J. Educ. Psychol. 2011, 103, 701–732. [Google Scholar] [CrossRef] [Green Version]
  28. Joshanloo, M. Factorial/Discriminant Validity and Longitudinal Measurement Invariance of MHC-SF in Korean Young Adults. Curr. Psychol. 2020, 39, 51–57. [Google Scholar] [CrossRef]
  29. Booth, T.; Hughes, D.J. Exploratory Structural Equation Modeling of Personality Data. Assessment 2014, 21, 260–271. [Google Scholar] [CrossRef] [Green Version]
  30. Chen, F.F.; Hayes, A.; Carver, C.S.; Laurenceau, J.P.; Zhang, Z. Modeling General and Specific Variance in Multifaceted Constructs: A Comparison of the Bifactor Model to Other Approaches. J. Pers. 2012, 80, 219–251. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  31. Muthén, B.; Asparouhov, T. Bayesian Structural Equation Modeling: A More Flexible Representation of Substantive Theory. Psychol. Methods 2012, 17, 313–335. [Google Scholar] [CrossRef] [PubMed]
Table
Table 1. Standardized factor loadings and standard errors for CFA and ESEM models.
Table 1. Standardized factor loadings and standard errors for CFA and ESEM models.
ESEM
ItemsF1F2F3F4F5F6F7F8CFAESEM S.E.CFA S.E.
F1. Anxious/depressed
 12. Lonely0.560.190.11−0.040.030.010.13−0.010.750.020.01
 13. Confused0.290.060.230.090.330.03−0.030.010.780.020.01
 14. Cries0.460.000.19−0.07−0.090.170.190.010.690.020.01
 22. Worry future0.51−0.050.05−0.070.000.070.21−0.030.580.020.01
 31. Fears doing bad0.420.140.020.120.010.050.130.100.670.020.02
 33. Unloved0.570.350.16−0.10−0.110.040.140.080.790.020.01
 34. Out to get him/her0.370.260.140.08−0.090.130.080.160.740.030.02
 35. Worthless0.680.190.04−0.000.11−0.020.030.030.830.020.01
 45. Nervous0.66−0.120.060.160.080.16−0.220.070.760.020.01
 47. Lacks self-conf.0.640.15−0.05−0.020.29−0.02−0.13−0.060.740.020.01
 50. Fearful0.75−0.160.050.170.070.11−0.270.040.750.020.01
 52. Too guilty0.63−0.090.010.060.130.08−0.080.100.700.020.01
 71. Self-conscious0.540.09−0.060.060.160.11−0.17−0.100.630.020.01
 91. Thinks suicide0.560.100.080.21−0.10−0.040.22−0.020.750.040.02
 103. Sad0.510.070.150.050.060.160.12−0.190.850.010.01
 107. Can’t succeed0.430.140.02−0.080.27−0.010.14−0.040.700.020.01
 112. Worries0.68−0.180.070.01−0.020.24−0.01−0.050.730.010.01
 113. Worries opp. sex0.410.100.010.010.000.030.320.030.650.020.01
F2. Withdrawn
 25. Not getting along0.000.500.070.010.050.28−0.040.110.660.020.01
 30. Poor opp. sex0.300.280.04−0.02−0.020.050.33−0.020.710.020.01
 42. Rather be alone0.070.40−0.010.280.120.15−0.11−0.240.600.020.01
 48. Not liked0.140.550.060.020.120.17−0.060.120.770.020.01
 60. Enjoys little0.280.340.110.060.170.140.05−0.160.860.020.01
 65. Won’t talk0.070.420.040.180.090.200.09−0.140.760.020.02
 67. No friends0.190.57−0.050.100.220.10−0.09−0.070.760.020.01
 69. Secretive0.080.310.050.250.010.160.13−0.160.650.020.01
 111. Withdrawn0.070.43−0.030.200.130.120.00−0.200.610.020.01
F3. Somatic complaints
 51. Dizzy0.16−0.020.540.110.100.00−0.130.030.700.020.01
 54. Feels tired0.15−0.020.320.150.310.11−0.03−0.230.840.020.01
 56a. Aches−0.06−0.070.660.100.090.030.00−0.020.640.010.01
 56b. Headaches0.04−0.090.65−0.010.010.07−0.07−0.060.560.010.01
 56c. Nausea0.04−0.010.87−0.030.01−0.080.010.080.750.010.01
 56d. Eye problems−0.060.050.480.030.010.030.08−0.000.500.020.02
 56e. Skin problems−0.020.020.410.040.090.02−0.060.030.450.020.02
 56f. Stomach aches0.08−0.040.76−0.040.01−0.04−0.020.080.670.020.01
 56g. Vomits−0.060.130.87−0.05−0.09−0.060.050.100.690.020.02
 56h. Heart pounds0.25−0.070.500.140.030.01−0.080.040.730.020.01
 56i. Numbness−0.02−0.040.600.100.080.060.01−0.020.680.020.01
 100. Sleep problems0.16−0.020.290.120.090.15−0.04−0.110.660.020.01
F4. Thought problems
 9. Can’t get mind off0.40−0.060.070.180.100.17−0.010.050.770.020.01
 18. Harms self0.520.070.150.15−0.06−0.090.320.060.830.060.04
 36. Gets hurts−0.01−0.010.280.190.270.02−0.100.100.550.020.02
 40. Hears things0.040.260.320.44−0.08−0.010.18−0.060.790.050.03
 46. Twitches0.160.010.280.290.110.06−0.020.050.720.020.01
 63. Pref. older people−0.050.240.170.130.030.160.13−0.080.550.020.01
 66. Repeats acts0.070.110.110.270.040.180.140.050.700.020.01
 70. Sees things0.000.250.300.31−0.090.090.170.020.730.040.03
 84. Strange behavior−0.050.10−0.030.740.060.000.150.150.710.020.01
 85. Strange ideas0.050.07−0.040.740.07−0.030.120.110.740.020.01
F5. Attention problems
 1. Forgetful −0.070.010.19−0.030.630.05−0.040.030.630.010.01
 8. Can’t concentrate0.05−0.020.110.110.550.06−0.060.060.680.010.01
 11. Dependent0.390.060.02−0.060.31−0.030.040.160.660.020.01
 17. Daydreams0.17−0.010.050.240.22−0.060.070.110.540.020.01
 53. Can’t plan0.260.020.030.020.46−0.020.27−0.050.810.010.01
 59. Fails to finish0.020.10−0.010.030.71−0.030.090.020.740.010.01
 61. Poor work0.110.230.03−0.020.55−0.010.120.000.770.020.02
 64. Can’t prioritize0.070.06−0.07−0.030.74−0.030.170.030.750.010.01
 78. Decisions0.330.04−0.06−0.010.530.00−0.07−0.010.690.010.01
 101. Avoids work−0.060.090.030.110.240.070.30−0.060.530.030.02
 102. Lacks energy0.110.000.300.090.380.13−0.04−0.280.750.010.01
 105. Disorganized−0.110.010.030.040.76−0.070.140.140.670.020.01
 108. Loses things−0.130.010.170.000.660.080.030.060.680.010.01
 119. Poor at details−0.050.08−0.04−0.020.59−0.010.130.070.550.020.01
 121. Late−0.08−0.020.04−0.080.480.020.140.100.430.020.01
F6. Aggressive behavior
 3. Argues0.070.06−0.05−0.190.010.68−0.020.180.580.020.01
 5. Blames0.280.21−0.08−0.160.180.18−0.070.280.530.030.02
 16. Mean−0.090.280.01−0.040.000.470.050.210.580.020.02
 28. Badly with family0.150.340.06−0.070.030.270.080.040.640.020.01
 37. Fights−0.040.220.120.00−0.030.480.100.220.700.040.03
 55. Elation-depression0.30−0.010.180.16−0.020.360.20−0.140.900.020.01
 57. Attacks−0.120.160.180.10−0.050.550.120.120.740.070.04
 68. Screams0.020.030.05−0.22−0.020.770.070.040.660.020.01
 81. Behavior changes0.030.03−0.020.240.030.290.150.010.560.020.01
 86. Stubborn0.090.04−0.010.170.010.64−0.07−0.080.720.020.01
 87. Mood changes0.15−0.040.060.20−0.050.590.17−0.160.830.020.01
 95. Temper −0.03−0.00−0.03−0.01−0.030.840.000.050.690.020.01
 97. Threatens−0.030.170.120.18−0.070.410.190.220.780.050.03
 116. Upset0.26−0.04−0.02−0.060.030.73−0.01−0.040.820.020.01
 118. Impatient0.07−0.10−0.100.030.190.64−0.010.030.680.020.01
F7. Rule-breaking behav.
 6. Uses drugs0.10−0.01−0.060.23−0.07−0.050.340.170.380.030.03
 20. Damages things0.040.210.110.09−0.060.340.130.140.730.050.03
 23. Breaks rules−0.020.16−0.050.170.160.050.240.290.590.020.02
 26. Lacks guilt−0.100.17−0.010.050.010.140.240.090.370.020.02
 39. Bad companions−0.080.110.110.160.070.150.320.190.670.040.03
 41. Impulsive0.020.030.030.140.230.340.130.130.820.020.01
 43. Lies, cheats0.040.17−0.010.080.100.110.280.220.630.030.02
 76. Irresponsible0.130.04−0.010.210.250.060.320.190.840.020.02
 82. Steals0.120.06−0.040.250.040.030.320.220.640.040.03
 90. Gets drunk0.08−0.09−0.120.210.060.040.210.140.360.030.02
 92. Trouble with law0.030.05−0.070.320.020.080.430.180.690.030.02
 114. Doesn’t pay debts0.03−0.180.08−0.090.230.110.77−0.220.710.020.01
 117. Money troubles0.02−0.220.02−0.070.350.140.70−0.160.740.020.01
 122. Can’t keep job0.040.200.030.010.200.140.33−0.040.720.030.02
F8. Intrusiveness
 7. Brags0.040.050.010.080.030.050.040.510.570.020.02
 19. Demands attention0.18−0.060.030.000.07−0.010.160.590.700.020.02
 74. Shows off−0.05−0.070.050.200.080.010.120.540.600.020.02
 93. Talks too much0.03−0.310.110.000.130.250.070.480.690.020.01
 94. Teases−0.06−0.01−0.050.150.160.23−0.050.470.700.020.02
 104. Loud−0.07−0.240.070.020.060.410.120.440.760.020.02
Note. S.E. = standard error. Factor loadings that are larger than 0.30 are shown in boldface. All CFA factor loadings are significant (p < 0.001). All ESEM values loading on their intended factors are significant (p < 0.01). Item numbers correspond with those in the ASEBA manual [2].
Table
Table 2. Measurement invariance testing across age groups (18–35 years vs. 36–59 years).
Table 2. Measurement invariance testing across age groups (18–35 years vs. 36–59 years).
CFITLIRMSEACMΔCFIΔTLIΔRMSEA
1a. Configural0.9260.9240.024----
1b. Metric0.9290.9270.0231a+0.003+0.003−0.001
1c. Scalar0.9310.9310.0231b+0.002+0.0040.000
Note. CM = Comparison model.
Table
Table 3. Descriptive statistics, and latent factor correlations and composite reliability for the CFA and ESEM models.
Table 3. Descriptive statistics, and latent factor correlations and composite reliability for the CFA and ESEM models.
VariablesF1F2F3F4F5F6F7F8
F1. Anxious/depressed-0.350.470.370.540.510.320.01 *
F2. Withdrawn0.80-0.300.290.200.370.380.17
F3. Somatic complaints0.720.64-0.390.380.490.320.04
F4. Thought problems0.760.780.70-0.340.430.350.20
F5. Attention problems0.790.690.680.72-0.480.300.13
F6. Aggressive behavior0.780.770.650.750.69-0.390.27
F7. Rule-breaking behavior0.660.700.560.730.750.75-0.23
F8. Intrusiveness0.370.350.350.540.480.590.56-
M5.041.562.941.444.643.361.191.00
SE0.050.020.030.020.040.030.020.01
Composite reliability (CFA)0.950.900.900.910.920.930.910.83
Alpha coefficient (CFA)0.950.890.900.890.920.930.900.82
Composite reliability (ESEM)0.880.660.870.590.850.860.660.68
Note. CFA and ESEM correlations are presented below and above the diagonal, respectively. * Correlation is non-significant (p > 0.05); all remaining ESEM correlations are significant (p < 0.01). All CFA correlations are significant (p < 0.001).
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Guerrero, M.; Hoffmann, M.; Pulkki-Råback, L. Psychometric Properties of the Adult Self-Report: Data from over 11,000 American Adults. Stats 2020, 3, 465-474. https://0-doi-org.brum.beds.ac.uk/10.3390/stats3040029

AMA Style

Guerrero M, Hoffmann M, Pulkki-Råback L. Psychometric Properties of the Adult Self-Report: Data from over 11,000 American Adults. Stats. 2020; 3(4):465-474. https://0-doi-org.brum.beds.ac.uk/10.3390/stats3040029

Chicago/Turabian Style

Guerrero, Michelle, Matt Hoffmann, and Laura Pulkki-Råback. 2020. "Psychometric Properties of the Adult Self-Report: Data from over 11,000 American Adults" Stats 3, no. 4: 465-474. https://0-doi-org.brum.beds.ac.uk/10.3390/stats3040029

Article Metrics

Back to TopTop