Next Article in Journal
Framework for the Development of Data-Driven Mamdani-Type Fuzzy Clinical Decision Support Systems
Next Article in Special Issue
Codex (Cognitive Disorders Examination) Decision Tree Modified for the Detection of Dementia and MCI
Previous Article in Journal
Four Virtual-Reality Simulators for Diagnostic Abdominal Ultrasound Training in Radiology
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

MACE for Diagnosis of Dementia and MCI: Examining Cut-Offs and Predictive Values

Cognitive Function Clinic, Walton Centre for Neurology and Neurosurgery, Liverpool L9 7LJ, UK
Submission received: 28 March 2019 / Revised: 23 April 2019 / Accepted: 1 May 2019 / Published: 6 May 2019
(This article belongs to the Special Issue Diagnosis of Dementia and Cognitive Impairment)

Abstract

:
The definition of test cut-offs is a critical determinant of many paired and unitary measures of diagnostic or screening test accuracy, such as sensitivity and specificity, positive and negative predictive values, and correct classification accuracy. Revision of test cut-offs from those defined in index studies is frowned upon as a potential source of bias, seemingly accepting any biases present in the index study, for example related to sample bias. Data from a large pragmatic test accuracy study examining the Mini-Addenbrooke’s Cognitive Examination (MACE) were interrogated to determine optimal test cut-offs for the diagnosis of dementia and mild cognitive impairment (MCI) using either the maximal Youden index or the maximal correct classification accuracy. Receiver operating characteristic (ROC) and precision recall (PR) curves for dementia and MCI were also plotted, and MACE predictive values across a range of disease prevalences were calculated. Optimal cut-offs were found to be a point lower than those defined in the index study. MACE had good metrics for the area under the ROC curve and for the effect size (Cohen’s d) for both dementia and MCI diagnosis, but PR curves suggested the superiority for MCI diagnosis. MACE had high negative predictive value at all prevalences, suggesting that a MACE test score above either cut-off excludes dementia and MCI in any setting.

1. Introduction

The Mini-Addenbrooke’s Cognitive Examination (MACE) is a shortened version of the Addenbrooke’s Cognitive Examination-Revised (ACE-R) and ACE-III developed by Mokken scaling analysis of these longer instruments [1]. MACE comprises tests of attention, memory (7-item name and address), verbal fluency, clock drawing and memory recall (score range 0–30, impaired to normal), and takes between 5–10 min to administer.
In the index MACE study, two cut-off points were identified in the cohort examined (n = 242; Alzheimer’s disease 28, behavioural variant frontotemporal dementia 23, primary progressive aphasia 82, corticobasal syndrome 21, controls 78): ≤25/30 had high sensitivity (0.85) and high specificity (0.87); and ≤21/30 had high specificity (1.00), and hence an abnormal score was almost certain to have come from a dementia patient. MACE was found to be more sensitive than the Mini-Mental State Examination (MMSE) and less likely to have ceiling effects [1].
The general applicability of these MACE cut-offs for the diagnosis of dementia and mild cognitive impairment (MCI) has not been widely examined. A Spanish translation administered to a cohort of mixed dementia patients and controls (n = 175) with relatively low educational experience found that a cut-off between 16/30 and 17/30 had optimal sensitivity (0.867) and specificity (0.870) for dementia diagnosis [2].
MACE was adopted in the author’s practice, based in a dedicated Cognitive Function Clinic located at a regional neuroscience centre in the northwest United Kingdom in June 2014 [3], and has been routinely used since then. Access to a large dataset has provided the opportunity to examine a variety of parameters not hitherto examined. The aims of this study were:
  • To determine optimal MACE cut-off points for the diagnosis of dementia and MCI, with the anticipation that cut-offs optimising sensitivity would also minimise false negative rate (FNR) and optimise negative predictive value (NPV) whilst cut-offs optimising specificity would minimise false positive rate (FPR) and optimise positive predictive value (PPV);
  • To plot receiver operating characteristic (ROC) and precision recall (PR) curves for dementia and MCI, and to calculate areas under the ROC curves and Q* index (a measure of diagnostic value);
  • To calculate MACE effect sizes (Cohen’s d) for diagnosis of dementia and MCI;
  • To calculate MACE predictive values across a range of disease prevalences.

2. Methods

Consecutive new patient referrals administered the MACE were included, seen over the period June 2014–December 2018 (inclusive), including data reported in previous studies [3,4,5]. Other than those with a pre-existing diagnosis of dementia, there were no exclusion criteria. As previously detailed [3,4,5], criterion diagnosis of dementia or mild cognitive impairment was by judgement of an experienced clinician using standard diagnostic criteria (DSM-IV; Petersen); in those without evidence of cognitive impairment, a diagnosis of subjective memory complaint (SMC) was made. MACE scores were not used to make criterion diagnoses to avoid review bias. Subjects gave informed consent, and the study protocol was approved by the institute’s committee on human research.
MACE scores were plotted against diagnosis, and the Pearson 2 skewness coefficient (Sk2) was used to assess skew, where:
Sk2 = 3(mean – median)/standard deviation,
with values lying between –1 and +1 deemed acceptable for the elimination of floor or ceiling effects [6].
Various measures were calculated for all MACE cut-offs within the range deemed clinically sensible. Paired measures were: sensitivity (Sens, or recall) and specificity (Spec); positive and negative predictive values (PPV or precision, and NPV); positive and negative likelihood ratios (LR+ and LR–); and positive and negative clinical utility indexes (CUI+ and CUI–). Unitary or global measures were: correct classification accuracy (Acc); Youden index (Y, where Y = Sens + Spec – 1) [7]; predictive summary index (PSI, where PSI = PPV = NPV – 1) [8]; and identification index (II, where II = 2(Acc – 1) [9].
Two methods to optimise test cut-offs were examined [10,11], namely, maximising either Acc or Youden index.
In addition, various “number needed to” metrics were calculated: number needed to diagnose (NND, where NND = 1/Y); number needed to predict (NNP, where NNP = 1/PSI) [8]; and number needed to misdiagnose (NNM, where NNM = 1/(1 – Acc)) [12]. Also calculated were the “likelihood to be diagnosed or misdiagnosed” ratio (LDM, where LDM = NNM/NND or NNM/NNP) [13,14], and the “summary utility index” (SUI, where SUI = (CUI+ + CUI–)) [15].The multiplicative inverse of the latter—the “number needed for screening utility” (NNSU = 1/SUI) [15]—was compared to the “number needed to screen” (NNS) metric, the multiplicative inverse of the “identification index” (II) [9], both rounded to the next highest integer value since they represent numbers of patients.
ROC curves were plotted (false positive rate versus sensitivity) and areas under the curve (AUC ROC) were calculated and categorised according to the scale of Metz [16]. The Q* index, a measure of diagnostic value [17], was determined as the point in ROC space where the anti-diagonal intersected the ROC curve (i.e., where Sens = Spec). Precision recall (PR) curves [18] were also plotted, as these have been recommended over ROC curves when analysing highly skewed datasets [19]. The F measure, or F1 score—the harmonic mean of precision and sensitivity—was also calculated as a global measure of accuracy [20].
Effect sizes (Cohen’s d) were calculated as the difference of the means of diagnostic groups divided by the weighted pooled standard deviations of the groups [21]. Cohen’s d values were categorised according to Sawilowsky’s extension of Cohen’s rules of thumb [22].
Predictive values across a range of disease prevalence (Prev) were calculated from observed sensitivity and specificity at the maximum Youden index for dementia and MCI, specifically at prevalence rates of 5%, 10%, 20% and 40% using the standard formulae:
PPV = Sens × Prev/(Sens × Prev) + [(1 – Spec) × (1 – Prev)],
NPV = Spec × (1 – Prev)/[Spec × (1 – Prev)] + [(1 – Sens) × Prev].

3. Results

A total of 755 patients were assessed with MACE (F:M = 352:403, 47% female; median age 60 years), of whom 114 were diagnosed with dementia (prevalence = 0.15) and 222 with MCI (prevalence = 0.29). The distribution of MACE scores by diagnosis, as seen in Figure 1, showed the anticipated unimodal negative skew (to the right, i.e., higher test scores = better performance). The Pearson 2 skewness coefficient (Sk2) was −0.48, suggesting that the population sampled was not from a normal distribution, although, as the value was between −1 and +1, the presence of floor or ceiling effects was probably excluded.
For the diagnosis of dementia (Table 1 and Table 2), looking at all MACE cut-off values, the optimal cut-off determined by maximal Youden index was ≤20/30 (sensitivity 0.91, specificity 0.71). By maximal correct classification accuracy the optimal cut-off was ≤14/30 (sensitivity 0.59, specificity 0.92). This latter cut-off also had the maximal values of PSI and LDM (Table 3), the latter also found at ≤15/30, which was also the cut-off for the maximal value of SUI and F measure.
For the diagnosis of MCI (Table 4 and Table 5), looking at all MACE cut-off values, the optimal cut-off determined by maximal Youden index was ≤24/30 (sensitivity 0.90, specificity 0.57). By maximal correct classification accuracy the optimal cut-off was ≤19/30 (sensitivity 0.47, specificity 0.88). Both cut-offs coincided with the maximal values of LDM (Table 6), whereas maximum PSI and F measure were at the same cut-off as maximal Youden index (cf. diagnosis of dementia), whilst maximal SUI was at ≤22/30 and ≤21/30.
Comparison of the values for NNS and NNSU (Table 3 and Table 6) showed negative values of NNS (which are clinically meaningless) when identification index (II) had negative values, but this was not the case with NNSU, since by definition SUI cannot be <0.
A ROC curve was plotted for the diagnosis of dementia and MCI (Figure 2). The AUC ROC was 0.89 (95% confidence interval (CI) 0.86–0.92) for the diagnosis of dementia and 0.81 (95% CI 0.77–0.84) for the diagnosis of MCI, hence the AUC ROC values were categorised as good [16] for the diagnosis of both dementia vs. no dementia and MCI vs. SMC. The Q* index for dementia was 0.8 (comparable to the value of 0.76 found in a previous study examining the first 135 patients in this cohort [3]) and for MCI it was 0.73. PR curves (Figure 3) did not show the desired approximation to the top-right corner of the graph, reflecting the relatively poor PPV of MACE, but did show greater area under the curve for MCI versus dementia.
Effect sizes (Cohen’s d) were 1.74 for dementia and 1.13 for MCI, hence very large and large, respectively [22].
PPV and NPV for MACE calculated at prevalence rates of 5%, 10%, 20% and 40% using the sensitivity and specificity figures at the maximum Youden index showed high NPV (≥0.9) at all prevalences examined, but with less-impressive figures for PPV, optimal at higher disease prevalences (Table 7).

4. Discussion

This study of a large cohort of patients examined with the Mini-Addenbrooke’s Cognitive Examination suggested that the optimal test cut-offs differ slightly from those suggested in the index study, being a point lower for both high-sensitivity (≤24/30 vs. ≤25/30) and high-specificity (≤20/30 vs. ≤21/30) cut-offs. Even allowing for the objections raised to changing test cut-offs because of risk of bias [23], these findings suggest a possible need for cut-off revision when using MACE in general cognitive clinics, as well as for patient educational level [2].
ROC curves suggested that MACE had adequate accuracy for the diagnosis of dementia and MCI—a finding corroborated by the measure of effect size (Cohen’s d). The Q* index for dementia (0.80) was comparable to that found for other cognitive screening instruments [24], but the Q* index for MCI was lower (0.73). However, PR curves suggested better MACE performance (i.e., distinguishable classification performance) for the diagnosis of MCI than for dementia, consistent with findings in previous comparative studies of MACE with the Montreal Cognitive Assessment (MoCA)—a test designed specifically for MCI diagnosis (equivalent MACE performance) [25,26], and Free-Cog (superior MACE performance) [5]. The use of PR curves [18] in diagnostic test accuracy studies is worth emphasizing (to the author’s knowledge this is the first such use for dementia test accuracy studies), since these avoid some of the “optimism” of ROC curves (resulting from their combining test accuracy over a range of thresholds which may be both clinically relevant and clinically nonsensical) [27]. PR curves are more informative than ROC curves for skewed datasets [19]. Area under the PR curve may be calculated, although this is not straightforward [28], and visual interpretation may be adequate to denote better classification performance (as for ROC curves).
The comparison of NNS [9] and NNSU [15] in this study was also instructive, demonstrating some of the difficulties in working with reciprocals (multiplicative inverses). By definition, the identification index (II) ranges from −1 to +1, and hence NNS ranges from −∞ to +∞ [9]. As II approaches 0, values of NNS are inflated (e.g., Table 3, cut-off ≤24/30; Table 6, cut-off ≤27/30), and when II is negative then NNS also has a negative value. The latter finding is problematic from the clinical standpoint: “number needed to” metrics were originally designed to appeal intuitively at the individual level, so negative values (representing non-individuals?) are meaningless. The construction of SUI is such that by definition its range is from 0 to 2, and hence NNSU ranges from ∞ (no screening value) to 0.5 (perfect screening utility) [15], avoiding the problems encountered with II and NNS.
The high NPV (≥0.9) at all disease prevalences examined (0.05 to 0.4) suggests that a MACE test score above either cut-off excludes dementia and MCI in any setting. PPV was less impressive, but improved with increasing disease prevalence, suggesting a case-finding role for MACE in dedicated cognitive and memory clinics.
In summary, in addition to being quick, easy to use and score and acceptable to patients [1,3,4], when using appropriate cut-offs MACE is a sensitive test for the identification of cognitive impairment and for excluding dementia and MCI with scores below the cut-offs.

Funding

This research received no external funding.

Acknowledgments

Nil.

Conflicts of Interest

The author declares no conflict of interest.

References

  1. Hsieh, S.; McGrory, S.; Leslie, F.; Dawson, K.; Ahmed, S.; Butler, C.R.; Rowe, J.B.; Mioshi, E.; Hodges, J.R.; et al. The Mini-Addenbrooke’s Cognitive Examination: A new assessment tool for dementia. Dement. Geriatr. Cogn. Disord. 2015, 39, 1–11. [Google Scholar] [CrossRef] [PubMed]
  2. Matias-Guiu, J.A.; Fernandez-Bobadilla, R. Validation of the Spanish-language version of Mini-Addenbrooke’s Cognitive Examination as a dementia screening tool [in Spanish]. Neurologia 2016, 31, 646–648. [Google Scholar] [CrossRef] [PubMed]
  3. Larner, A.J. Mini-Addenbrooke’s Cognitive Examination: A pragmatic diagnostic accuracy study. Int. J. Geriatr. Psychiatry 2015, 30, 547–548. [Google Scholar] [CrossRef] [PubMed]
  4. Williamson, J.C.; Larner, A.J. MACE for diagnosis of dementia and MCI: 3-year pragmatic diagnostic test accuracy study. Dement. Geriatr. Cogn. Disord. 2018, 45, 300–307. [Google Scholar] [CrossRef] [PubMed]
  5. Larner, A.J. Free-Cog: Pragmatic test accuracy study. Dement. Geriatr. Cogn. Disord. 2019, 48. accepted. [Google Scholar]
  6. Doane, D.P.; Seward, L.E. Measuring skewness: A forgotten statistic? J. Stat. Educ. 2011, 19, 1–18. [Google Scholar] [CrossRef]
  7. Youden, W.J. Index for rating diagnostic tests. Cancer 1950, 3, 32–35. [Google Scholar] [CrossRef]
  8. Linn, S.; Grunau, P.D. New patient-oriented summary measure of net total gain in certainty for dichotomous diagnostic tests. Epidemiol. Perspect. Innov. 2006, 3, 11. [Google Scholar] [CrossRef] [PubMed]
  9. Mitchell, A.J. Index test. In Encyclopedia of Medical Decision Making; Kattan, M.W., Ed.; Sage: Los Angeles, CA, USA, 2009; pp. 613–617. [Google Scholar]
  10. Larner, A.J. Optimizing the cutoffs of cognitive screening instruments in pragmatic diagnostic accuracy studies: Maximising accuracy or Youden index? Dement. Geriatr. Cogn. Disord. 2015, 39, 167–175. [Google Scholar] [CrossRef] [PubMed]
  11. O’Caoimh, R.; Gao, Y.; Svendovski, A.; Gallagher, P.; Eustace, J.; Molloy, D.W. Comparing approaches to optimize cut-off scores for short cognitive screening instruments in mild cognitive impairment and dementia. J. Alzheimers Dis. 2017, 57, 123–133. [Google Scholar] [CrossRef] [PubMed]
  12. Habibzadeh, F.; Yadollahie, M. Number needed to misdiagnose: A measure of diagnostic test effectiveness. Epidemiology 2013, 24, 170. [Google Scholar] [CrossRef] [PubMed]
  13. Larner, A.J. Number needed to diagnose, predict, or misdiagnose: Useful metrics for non-canonical signs of cognitive status? Dement. Geriatr. Cogn. Dis. Extra 2018, 8, 321–327. [Google Scholar] [CrossRef] [PubMed]
  14. Larner, A.J. Evaluating cognitive screening instruments with the “likelihood to be diagnosed or misdiagnosed” measure. Int. J. Clin. Pract. 2019, 73, e13265. [Google Scholar] [CrossRef]
  15. Larner, A.J. Manual of Screeners for Dementia: Pragmatic Test Accuracy Studies; Springer: London, UK, 2019; in press. [Google Scholar]
  16. Metz, C.E. Basic principles of ROC analysis. Semin. Nucl. Med. 1978, 8, 283–298. [Google Scholar] [CrossRef]
  17. Walter, S.D. Properties of the summary receiver operating characteristic (SROC) curve for diagnostic test data. Stat. Med. 2002, 21, 1237–1256. [Google Scholar] [CrossRef]
  18. Davis, J.; Goadrich, M. The relationship between Precision-Recall and ROC curves. In ICML ’06: Proceedings of the 23rd International Conference on Machine Learning; ACM: New York, NY, USA, 2006; pp. 233–240. [Google Scholar]
  19. Saito, T.; Rehmsmeier, M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE 2015, 10, e0118432. [Google Scholar] [CrossRef]
  20. Powers, D.M.W. Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation. J. Mach. Learn. Technol. 2011, 2, 37–63. [Google Scholar]
  21. Cohen, J. Statistical Power Analysis for the Behavioral Sciences, 2nd ed.; Lawrence Erlbaum: Hillsdale, NJ, USA, 1988. [Google Scholar]
  22. Sawilowsky, S.S. New effect sizes rules of thumb. J. Mod. Appl. Stat. Methods 2009, 8, 597–599. [Google Scholar] [CrossRef]
  23. Davis, D.H.; Creavin, S.T.; Noel-Storr, A.; Quinn, T.J.; Smailagic, N.; Hyde, C.; Brayne, C.; McShane, R.; Cullum, S.; et al. Neuropsychological tests for the diagnosis of Alzheimer’s disease dementia and other dementias: A generic protocol for cross-sectional and delayed-verification studies. Cochrane Database Syst. Rev. 2013, 3, CD010460. [Google Scholar] [CrossRef]
  24. Larner, A.J. The Q* index: A useful global measure of dementia screening test accuracy? Dement. Geriatr. Cogn. Dis. Extra 2015, 5, 265–270. [Google Scholar] [CrossRef]
  25. Nasreddine, Z.S.; Phillips, N.A.; Bédirian, V.; Charbonneau, S.; Whitehead, V.; Collin, I.; Cummings, J.E.; Chertkow, H.; et al. The Montreal Cognitive Assessment, MoCA: A brief screening tool for mild cognitive impairment. J. Am. Geriatr. Soc. 2005, 53, 695–699. [Google Scholar] [CrossRef] [PubMed]
  26. Larner, A.J. MACE versus MoCA: equivalence or superiority? Pragmatic diagnostic test accuracy study. Int. Psychogeriatr. 2017, 29, 931–937. [Google Scholar] [CrossRef] [PubMed]
  27. Ozenne, B.; Subtil, F.; Maucort-Boulch, D. The precision-recall curve overcame the optimism of the receiver operating characteristic curve in rare diseases. J. Clin. Epidemiol. 2015, 68, 855–859. [Google Scholar] [CrossRef] [PubMed]
  28. Keilwagen, J.; Grosse, I.; Grau, J. Area under precision-recall curves for weighted and unweighted data. PLoS ONE 2014, 9, e92209. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Mini-Addenbrooke’s Cognitive Examination (MACE) scores versus patient diagnosis (n = 755). MCI: mild cognitive impairment; SMC: subjective memory complaint.
Figure 1. Mini-Addenbrooke’s Cognitive Examination (MACE) scores versus patient diagnosis (n = 755). MCI: mild cognitive impairment; SMC: subjective memory complaint.
Diagnostics 09 00051 g001
Figure 2. MACE receiver operating characteristic (ROC) plots for the diagnosis of dementia (upper; area under the curve (AUC) = 0.89) and MCI (lower; AUC = 0.81) with chance diagonal (y = x) and anti-diagonal (y = 1 − x) lines.
Figure 2. MACE receiver operating characteristic (ROC) plots for the diagnosis of dementia (upper; area under the curve (AUC) = 0.89) and MCI (lower; AUC = 0.81) with chance diagonal (y = x) and anti-diagonal (y = 1 − x) lines.
Diagnostics 09 00051 g002
Figure 3. MACE precision recall (PR) curves for the diagnosis of dementia (lower) and MCI (upper); note the reversal of position versus ROC curves.
Figure 3. MACE precision recall (PR) curves for the diagnosis of dementia (lower) and MCI (upper); note the reversal of position versus ROC curves.
Diagnostics 09 00051 g003
Table 1. Diagnosis of dementia: paired measures of discrimination at various MACE cut-offs.
Table 1. Diagnosis of dementia: paired measures of discrimination at various MACE cut-offs.
Cut-offSensitivity (= Recall); SpecificityFalse Positive Rate (FPR); False Negative Rate (FNR)PPV (= Precision); NPVLR+; LR−CUI+; CUI−
≤29/301.00; 0.020.98; 00.15; 1.001.02; 00.15; 0.02
≤28/300.99; 0.060.94; 0.010.16; 0.971.05; 0.160.16; 0.05
≤27/300.99; 0.130.87; 0.010.17; 0.991.14; 0.070.17; 0.13
≤26/300.99; 0.220.78; 0.010.18; 0.991.27; 0.040.18; 0.22
≤25/30 *0.99; 0.320.68; 0.010.20; 0.991.45; 0.030.20; 0.32
≤24/300.98; 0.410.59; 0.020.23; 0.991.66; 0.040.23; 0.41
≤23/300.98; 0.490.51; 0.020.25; 0.991.91; 0.040.25; 0.49
≤22/300.97; 0.560.44; 0.030.28; 0.992.23; 0.050.27; 0.55
≤21/30 *0.95; 0.640.36; 0.050.32; 0.992.64; 0.080.30; 0.63
≤20/300.91; 0.710.29; 0.090.36; 0.983.11; 0.120.33; 0.70
≤19/300.86; 0.760.24; 0.140.38; 0.973.51; 0.190.33; 0.74
≤18/300.80; 0.800.20; 0.200.42; 0.964.03; 0.250.34; 0.77
≤17/300.74; 0.820.18; 0.260.42; 0.954.11; 0.320.31; 0.78
≤16/300.72; 0.860.14; 0.280.48; 0.955.24; 0.330.35; 0.82
≤15/300.66; 0.900.10; 0.340.53; 0.946.29; 0.380.35; 0.85
≤14/300.59; 0.920.08; 0.410.56; 0.937.11; 0.450.33; 0.86
≤13/300.48; 0.930.07; 0.520.57; 0.917.36; 0.550.27; 0.85
CUI+ and CUI−: positive and negative clinical utility indices; LR+ and LR−: positive and negative likelihood ratios; NPV: negative predictive value; PPV: positive predictive value. * Test cut-off established in index study (Hsieh et al. 2015 [1]).
Table 2. Diagnosis of dementia: unitary measures of discrimination at various MACE cut-offs.
Table 2. Diagnosis of dementia: unitary measures of discrimination at various MACE cut-offs.
Cut-offAcc; InaccY
(= Sens + Spec − 1)
PSI
(= PPV + NPV − 1)
II
(= 2(Acc − 1))
DOR
(= LR+/LR−)
SUI
(= CUI+ + CUI−)
F Measure (F1 Score)
≤29/300.17; 0.830.020.15−0.660.150.27
≤28/300.20; 0.800.050.13−0.616.720.210.27
≤27/300.26; 0.740.120.16−0.4817.30.300.29
≤26/300.37; 0.630.210.17−0.3331.90.400.31
≤25/30 *0.42; 0.580.310.19−0.1752.00.520.34
≤24/300.49; 0.510.390.22−0.0138.70.640.37
≤23/300.56; 0.440.470.240.1252.80.740.40
≤22/300.63; 0.370.530.270.2547.70.820.44
≤21/30 *0.69; 0.310.590.310.3732.20.930.48
≤20/300.74; 0.260.6190.340.4825.11.030.51
≤19/300.77; 0.230.6140.350.5418.91.070.53
≤18/300.80; 0.200.600.380.6016.01.110.55
≤17/300.81; 0.190.560.370.6212.81.090.54
≤16/300.84; 0.160.580.430.6816.11.170.58
≤15/300.859; 0.140.560.460.7216.51.200.59
≤14/300.867; 0.130.510.490.7415.81.190.57
≤13/300.866; 0.130.410.490.7313.31.120.52
Acc: correct classification accuracy; DOR: diagnostic odds ratio; II: identification index; Inacc: inaccuracy; PSI: predictive summary index; Sens: sensitivity; Spec: specificity; SUI: summary utility index; Y: Youden index. * Test cut-off established in index study (Hsieh et al. 2015 [1]).
Table 3. Diagnosis of dementia: “number needed to” measures (rounded to next highest integer value) and “likelihood to be diagnosed or misdiagnosed” at various MACE cut-offs.
Table 3. Diagnosis of dementia: “number needed to” measures (rounded to next highest integer value) and “likelihood to be diagnosed or misdiagnosed” at various MACE cut-offs.
Cut-offNND (= 1/Y)NNP (= 1/PSI)NNM (= 1/Inacc)NNS (= 1/II)NNSU (= 1/SUI)LDM (= NNM/NND; NNM/NNP)
≤29/305072−270.04; 0.29
≤28/302082−250.10; 0.25
≤27/30972−340.22; 0.29
≤26/30562−430.40; 0.33
≤25/30 *462−720.50; 0.33
≤24/30352−10820.67; 0.40
≤23/30353921.00; 0.60
≤22/30243421.50; 0.75
≤21/30 *244322.00; 1.00
≤20/30234312.00; 1.33
≤19/30235212.50; 1.67
≤18/30235212.50; 1.67
≤17/30236213.00; 2.00
≤16/30237213.50; 2.33
≤15/30238214.00; 2.67
≤14/30238214.00; 2.67
≤13/30338212.67; 2.67
LDM: likelihood to diagnose or misdiagnose; NND: number needed to diagnose; NNM: number needed to misdiagnose; NNP: number needed to predict; NNS: number needed to screen; NNSU: number needed for screening utility. * Test cut-off established in index study (Hsieh et al. 2015 [1]).
Table 4. Diagnosis of MCI: paired measures of discrimination at various MACE cut-offs.
Table 4. Diagnosis of MCI: paired measures of discrimination at various MACE cut-offs.
Cut-offSensitivity (= Recall); SpecificityFalse Positive (FPR); False Negative (FNR)PPV (= Precision); NPVLR+; LR−CUI+; CUI−
≤29/301.00; 0.030.97; 00.35; 1.001.03; 00.35; 0.03
≤28/301.00; 0.090.91, 00.37; 1.001.09; 00.37; 0.09
≤27/300.99; 0.200.80; 0.010.40; 0.991.25; 0.020.40; 0.20
≤26/300.98; 0.320.68; 0.020.43; 0.961.45; 0.070.42; 0.31
≤25/30 *0.95; 0.460.54; 0.050.48; 0.951.76; 0.100.46; 0.44
≤24/300.90; 0.570.43; 0.100.53; 0.922.11; 0.170.48; 0.52
≤23/300.82; 0.640.36; 0.180.55; 0.872.29; 0.290.45; 0.56
≤22/300.73; 0.720.28; 0.270.58; 0.842.63; 0.370.42; 0.60
≤21/30 *0.64; 0.790.21; 0.360.61; 0.802.99; 0.460.39; 0.63
≤20/300.54; 0.840.16; 0.460.63; 0.773.33; 0.550.34; 0.65
≤19/300.47; 0.880.12; 0.530.67; 0.763.81; 0.600.31; 0.67
≤18/300.37; 0.890.11; 0.630.65; 0.733.56; 0.700.24; 0.65
LR+; LR− = positive and negative likelihood ratio; CUI+; CUI− = positive and negative clinical utility index. * = test cut-off established in index study (Hsieh et al. 2015 [1]).
Table 5. Diagnosis of MCI: unitary measures of discrimination at various MACE cut-offs.
Table 5. Diagnosis of MCI: unitary measures of discrimination at various MACE cut-offs.
Cut-offAcc; InaccY (= Sens + Spec − 1)PSI (= PPV + NPV − 1)II (= 2(Acc − 1))DOR (= LR+/LR−)SUI (= CUI+ + CUI−)F Measure (F1 Score)
≤29/300.37; 0.630.030.35−0.260.380.52
≤28/300.40; 0.600.090.37−0.200.460.54
≤27/300.48; 0.520.190.39−0.0555.40.600.57
≤26/300.55; 0.450.300.390.1020.90.730.60
≤25/30 *0.63; 0.370.410.430.2617.90.900.64
≤24/300.69; 0.310.470.450.3712.21.000.67
≤23/300.70; 0.300.460.420.418.001.010.66
≤22/300.73; 0.270.450.420.457.131.020.65
≤21/30*0.73; 0.270.430.410.476.451.020.62
≤20/300.73; 0.270.380.400.476.070.990.58
≤19/300.74; 0.260.350.430.476.330.980.55
≤18/300.71; 0.290.260.380.435.090.890.48
Acc: correct classification accuracy; DOR: diagnostic odds ratio; II: identification index; Inacc: inaccuracy; PSI: predictive summary index; SUI: summary utility index; Y: Youden index. * Test cut-off established in index study (Hsieh et al. 2015 [1]).
Table 6. Diagnosis of MCI: “number needed to” measures (rounded to next highest integer value) and “likelihood to be diagnosed or misdiagnosed” at various MACE cut-offs.
Table 6. Diagnosis of MCI: “number needed to” measures (rounded to next highest integer value) and “likelihood to be diagnosed or misdiagnosed” at various MACE cut-offs.
Cut-offNND (= 1/Y)NNP (= 1/PSI)NNM (= 1/Inaccuracy)NNS (= 1/II)NNSU (= 1/SUI)LDM (= NNM/NND; NNM/NNP)
≤29/303432−430.06; 0.67
≤28/301232−630.17; 0.67
≤27/30632−2120.33; 0.67
≤26/304331020.75; 1.00
≤25/30 *333421.00; 1.00
≤24/30334311.33; 1.33
≤23/30334311.33; 1.33
≤22/30334311.33; 1.33
≤21/30 *334311.33; 1.33
≤20/30334321.33; 1.33
≤19/30334321.33; 1.33
≤18/30434321.00; 1.33
LDM: likelihood to diagnose or misdiagnose; NND: number needed to diagnose; NNM: number needed to misdiagnose; NNP: number needed to predict; NNS: number needed to screen; NNSU: number needed for screening utility. * Test cut-off established in index study (Hsieh et al. 2015 [1]).
Table 7. MACE predictive values at differing disease prevalence of dementia and MCI (0.05–0.4).
Table 7. MACE predictive values at differing disease prevalence of dementia and MCI (0.05–0.4).
Prevalence
0.050.10.15 (Observed)0.20.29 (Observed)0.4
PPV dementia (cut-off ≤20/30)0.140.260.360.44-0.68
NPV dementia (cut-off ≤20/30)0.990.990.980.97-0.92
PPV MCI (cut-off ≤24/30)0.100.19-0.340.530.58
NPV MCI (cut-off ≤24/30)0.990.98-0.960.920.90

Share and Cite

MDPI and ACS Style

Larner, A.J. MACE for Diagnosis of Dementia and MCI: Examining Cut-Offs and Predictive Values. Diagnostics 2019, 9, 51. https://0-doi-org.brum.beds.ac.uk/10.3390/diagnostics9020051

AMA Style

Larner AJ. MACE for Diagnosis of Dementia and MCI: Examining Cut-Offs and Predictive Values. Diagnostics. 2019; 9(2):51. https://0-doi-org.brum.beds.ac.uk/10.3390/diagnostics9020051

Chicago/Turabian Style

Larner, Andrew J. 2019. "MACE for Diagnosis of Dementia and MCI: Examining Cut-Offs and Predictive Values" Diagnostics 9, no. 2: 51. https://0-doi-org.brum.beds.ac.uk/10.3390/diagnostics9020051

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop