Next Article in Journal
Leptospirosis and Coinfection: Should We Be Concerned?
Previous Article in Journal
Impact of the SARS-CoV-2 Coronavirus Pandemic on Physical Activity, Mental Health and Quality of Life in Professional Athletes—A Systematic Review
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
IJERPH
Volume 18
Issue 17
10.3390/ijerph18179405
ijerph-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Adaptation and Psychometric Validation of the EMPOWER-SUSTAIN Usability Questionnaire (E-SUQ) among Patients with Metabolic Syndrome in Primary Care

by
Nurul Hidayatullaila Sahar
1,
Siti Fatimah Badlishah-Sham
1 and
Anis Safura Ramli
1,2,*
1
Department of Primary Care Medicine, Faculty of Medicine, Universiti Teknologi MARA, Jalan Prima Selayang 7, Batu Caves 68100, Selangor, Malaysia
2
Institute of Pathology, Laboratory & Forensic Medicine (I-PPerForM), Universiti Teknologi MARA, Jalan Hospital, Sungai Buloh 47000, Selangor, Malaysia
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2021, 18(17), 9405; https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph18179405
Submission received: 11 July 2021 / Revised: 27 August 2021 / Accepted: 29 August 2021 / Published: 6 September 2021
Browse Figure
Review Reports Versions Notes

Abstract

:
Self-management support is one of the most important components of the Chronic Care Model (CCM). The EMPOWER-SUSTAIN Global Cardiovascular Risks Self-Management Booklet© was developed for patients with Metabolic Syndrome (MetS), inspired by the CCM. Assessing usability of a self-management tool is important in chronic disease management. However, there was no available instrument to assess usability of a self-management booklet, as most instruments were developed to assess usability of mobile application. Therefore, this study aimed to adapt Skala Kebolehgunaan Aplikasi Mudah Alih (SKAMA) into the EMPOWER-SUSTAIN Usability Questionnaire (E-SUQ) and to determine its validity and reliability in assessing usability of a self-management booklet. A cross-sectional validation study was conducted among patients with MetS attending a university primary care clinic in Selangor, Malaysia. Content validation, adaptation and face validation of E-SUQ were performed according to recommended guidelines. It underwent two rounds of content validation as major revision was required for item 5. Subsequently, the revised E-SUQ was face-validated by 10 participants. Psychometric evaluation was conducted using principal component analysis with varimax rotation to determine the underlying structure of E-SUQ. Internal consistency reliability was assessed using Cronbach’s α coefficient and the test-retest reliability was assessed using intraclass correlation coefficient (ICC (2,k)). A total of 205 patients participated in the study. The item-level content-validity-index (I-CVI) for item 5 improved from 0.57 to 1.0 after the second round of content validation. The final S-CVI/Ave value for ESUQ was >0.90. The item-level face-validity-index (I-FVI) ranged between 0.9 and 1.0. Kaiser-Meyer-Olkin value of 0.871 and Bartlett’s test of sphericity p-value of <0.05 indicated good sample adequacy for factor analysis. Two factors with eigenvalues of >1 were extracted according to the Kaiser’s Criteria. The two extracted factors explained 60.6% of the cumulative percentage of variance. The elbow of the scree plot occurred between the second and third component, suggesting two factors to be retained. The two factors were consistent with “Positive” and “Negative” tone model. The overall Cronbach’s α coefficient was 0.77, indicating good internal reliability. The overall ICC was 0.85, indicating good reproducibility. The E-SUQ is shown to be valid, reliable and stable to measure the usability of a self-management booklet among patients with MetS in a university primary care clinic in Malaysia.

1. Introduction

The increasing prevalence of cardiovascular (CV) risk factors which include Metabolic Syndrome (MetS) represents a public health issue of growing concern [1]. MetS is defined by a clustering of CV risk factors which include central obesity, dyslipidaemia, elevated blood glucose and elevated blood pressure [2]. Individuals with MetS were twice as likely to develop CV events, namely, heart attack and stroke [3]. The prevalence of MetS globally has reached epidemic proportion [4]. In Malaysia, 43.4% of Malaysian adults have MetS [5]. The Chronic Care Model (CCM) is a guide to transform the management of chronic conditions including MetS to improve outcomes in primary care [6]. It is a conceptual model for restructuring the healthcare system to ensure productive interactions between an informed, activated patient and a proactive, prepared practice team [7]. The CCM focuses on optimizing 6 interrelated key elements which include healthcare organization, delivery system design, patient self-management support, clinical information system, decision support and use of community resources [8].
Self-management support has been recognized as one of the most important components of the CCM [9]. Evidence has shown that self-management support by healthcare professionals and utilization of self-management tools were effective in improving knowledge, motivation, self-management behaviours and health outcomes in patients with MetS [10,11,12,13,14]. In Malaysia, a self-management support tool named the EMPOWER-PAR Global CV Risks Self-Management Booklet© was developed in the EMPOWER-PAR study [15,16]. This booklet has recently been revised to become the EMPOWER-SUSTAIN Global CV Risks Self-Management Booklet©. It is given to patients so that they can be empowered to actively participate in the management of their own health. Therefore, it is important to assess the usability of this self-management booklet among the target population to ensure its usefulness.
Usability is defined as “the extent to which a product can be used by a specific user for a specific goal in a specific context or environment, and provides a satisfying experience” [17]. Two types of study that can be used to assess usability are qualitative or quantitative studies. One of the quantitative methods to assess usability is by using a questionnaire. The System Usability Scale (SUS) is one of the most widely used questionnaires to assess the usability of mobile apps [18]. It consists of 10 items which can be answered on a five-point Likert scale ranging from “strongly disagree” to “strongly agree”. SUS has been translated into various languages, i.e., Indonesian, Spanish, French, Dutch, Portuguese, Slovenian, Persian, German, and more recently into the Malay language, which is called the “Skala Kebolehgunaan Aplikasi Mudah Alih” (SKAMA) (Appendix A) [19]. However, to the best of our knowledge, there is no available questionnaire that assesses the usability of a self-management booklet. Thus, the first objective of this study was to adapt SKAMA into the EMPOWER-SUSTAIN Usability Questionnaire (E-SUQ) to assess usability of the EMPOWER-SUSTAIN Global CV Risks Self-Management Booklet© exploring the extent to which this book can be used by patients with MetS in primary care and their personal experience. The second objective was to validate the questionnaire among patients with MetS in primary care.

2. Materials and Methods

2.1. Study Design and Population

This was a cross-sectional validation study that was conducted in three phases. The first phase consisted of adaptation and content validation of the E-SUQ. This was followed by face validation in the second phase, and field testing and psychometric evaluation of the questionnaire in the final phase. The study population included patients with MetS attending a university primary care clinic in Selangor, Malaysia. It was conducted based on the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) guideline and principles of good practice for validation of a questionnaire [20,21]. The conduct of the study is shown in the flowchart presented in Figure 1.
The inclusion criteria were as follows: (a) aged 18–80 years old; (b) attended the university primary care clinic for at least one year; (c) given the EMPOWER-SUSTAIN Global CV Risks Self-Management Booklet© for ≥6 months; (d) had blood investigations (Fasting Plasma Glucose (FPG), Fasting Serum Lipid (FSL) and HbA1c) done in the last one year; (e) fulfilled at least 3 out of 5 diagnostic criteria for MetS based on the 2008 Joint Interim Statement definition [2] (i.e., waist circumference South Asian cut points: male ≥ 90 cm, female ≥ 80 cm; systolic blood pressure (BP) ≥ 130 and/or diastolic BP ≥ 85 mmHg or on treatment for hypertension (HTN); FPG ≥ 5.6 mmol/L or on treatment for elevated glucose; triglycerides (TG) ≥ 1.7 mmol/L or on treatment for dyslipidaemia; high-density lipoprotein-cholesterol (HDL-C): male < 1.0 mmol/L, female < 1.3 mmol/L or on treatment for dyslipidaemia); (f) were able to read and understand the Malay language and (g) willing to participate in the study.
The following patients were excluded from the study: (a) diagnosed with circulatory disorders requiring secondary care over the past year (e.g., acute coronary syndrome, stroke, transient ischemic attacks, peripheral vascular disease); (b) on renal dialysis; (c) presented with severe HTN (systolic BP > 180 mmHg and/or diastolic BP > 110 mmHg); (d) on radiotherapy/chemotherapy or palliative care; (e) had any form of mental disorder or cognitive impairment that would affect the ability to answer the questionnaire, for example, dementia or mental retardation; (f) pregnant; and (g) unable to give informed consent.

2.2. Study Tool

The E-SUQ in this study was adapted from SKAMA to assess the level of usability of the EMPOWER-SUSTAIN Global CV Risks Self-Management Booklet©. SKAMA is a 10-item questionnaire in the Malay language which assesses the usability of mobile apps, in which the response score is calculated using the 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). The item score on the positively phrased statements is subtracted by 1 (x − 1) and the item score on the negatively phrased statements is calculated by subtracting the score from 5 (5 − x) [22]. The overall score is computed as the summation of all item scores multiplied by 2.5, which gives the overall score that ranged from 0 (extremely poor usability) to 100 (excellent usability). The score value of > 68 is recommended by the original author to indicate the cut-off point for good usability of an app [23]. Permission to adapt and validate SKAMA into E-SUQ was obtained from the researchers.

2.3. Phase 1: Adaptation and Content Validation

The SKAMA underwent a process of adaptation into E-SUQ. Each item in the questionnaire which assessed the usability of mobile application was adapted to assess the usability of the self-management booklet. The word “mobile application” was substituted with “self-management booklet” in the Malay version.
Content validation was conducted through an online survey by seven family medicine specialists who are experts in questionnaire validation. They are clinical experts with a special interest in patient empowerment and chronic disease management. According to the literature, content validation should include at least five experts to have sufficient control over chance agreement [24]. The original 10-item SKAMA was critically reviewed by the panels for clarity and relevance to the conceptual framework. The items were rated on a scale from 1 to 4 (1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, 4 = highly relevant). The item level content validity index (I-CVI) was computed for each item by dichotomizing the 4-point scale. Items with a score of either 1 or 2 were recategorized into “not relevant” with 0 point. Items with a score of either 3 or 4 were recategorized into “relevant” with 1 point. The values (0 or 1) for each item were added up and then the total value was divided by the total number of experts [25]. An I-CVI value of at least 0.83 determined that the items were relevant and to be retained in the questionnaire [26]. Two rounds of content validity were conducted in this study because a major revision was required for item 5. After the second round, the scale level content validity index based on the average method (S-CVI/Ave) was computed to evaluate the relevance of the revised questionnaire. S-CVI/Ave is the average of the I-CVI scores for all items on the scale or the average of proportion relevance judged by all experts [25]. S-CVI/Ave value of >0.90 indicates that the item should be retained [27].

2.4. Phase 2: Face Validation

The face validation process was done through a face-to-face interview on a sample of 10 patients who met the inclusion and exclusion criteria. They were asked to comment on the questionnaire’s instructions, contents, terminology, comprehensibility and overall structure. Correction and fine-tuning of the E-SUQ were conducted by the research team based on the patient’s feedback. Face validity index (FVI) was used to evaluate the items in the form of clarity and comprehensibility of language and instructions used in the questionnaire [28]. The participants were requested to rate the comprehensibility of each item to the conceptual framework on a scale from 1 to 4 (1 = not understandable, 2 = somewhat understandable, 3 = understandable, 4 = very understandable). The item level-FVI (I-FVI) was computed for each item by dichotomizing the 4-point scale, with items scoring either 1 or 2 being recoded as 0 and items with a score of either 3 or 4 being recoded as 1 [29]. The values (0 or 1) for each item were added up and then the total value was divided by the total number of experts. Marzuki et al. suggested that if the number of raters is 10, the acceptable cut-off score of FVI is at least 0.83 [19]. This process produced the refined E-SUQ, which was ready to undergo the psychometric evaluation.

2.5. Phase 3: Field Testing and Psychometric Evaluation

In the final phase, the E-SUQ went through field testing for psychometric evaluation. The same inclusion and exclusion criteria were applied to the recruited participants. However, participants who were involved in Phase 2 were not included in Phase 3 of this study.

2.5.1. Sample Size Determination

The sample size for the psychometric evaluation was calculated using the sample-to-variable ratio (SVR) of 20:1 [30]. The E-SUQ contains 10 items; therefore, a minimum sample size of 200 patients was required. After taking into consideration a 10% non-responder and non-eligibility rate, the study aimed to approach 220 patients.

2.5.2. Sampling Method, Patient Recruitment and Data Collection

The EMPOWER-SUSTAIN Global CV Risks Self-Management Booklet© was distributed to patients with MetS at the university primary care clinic from October 2019 to March 2020, to ensure that the patients had the booklet for at least 6 months prior to the usability data collection.
Patients were recruited over four months from December 2020 to March 2021 when they attended the clinic for their follow-up appointment. The data collection was conducted by a trained research assistant to maintain a consistent method of collecting the data. Patients who were identified to have the booklet were approached in the nurse assessment room on the day of their follow-up appointment. They were briefed about the study and were invited to participate. A patient information sheet was given to participants to provide further details regarding the study, and their right for confidentiality was explained. Those who agreed to participate were screened for eligibility according to the inclusion and exclusion criteria. The patient’s eligibility was screened by reviewing the electronic medical record. A written informed consent was obtained from participants who agreed and were eligible to participate in the study.
The usability data were collected using E-SUQ, which was distributed to participants to be self-administered. Clear verbal instructions were provided on how to complete the questionnaire without assistance. The participants were advised to seek clarification from the researcher for any questions. Participants took approximately 15 minutes to complete the questionnaire. Following completion, participants were asked to return the questionnaire to the researcher to be checked for completeness.

2.5.3. Data Collection for Test-Retest

Thirty participants were recruited for the test-retest of E-SUQ. A date was given for them to come back to the clinic within 2–4 weeks to answer the same questionnaire. Participants were called one day before to remind them about the appointment.

2.6. Statistical Analysis

Data entry and statistical analysis were performed using the latest IBM SPSS Statistics Program Version 25. During the data entry, the responses for negative statements (item 2, 4, 6, 8 and 10) were reversed. Data quality was examined using the percentage of missing data and mean score of E-SUQ with standard deviation (±SD) was calculated using the formula recommended by the SUS author [22]. Descriptive analysis was presented as frequency and percentage for categorical data. Normally distributed continuous data were expressed as mean with ±SD and non-normally distributed data were expressed as median with interquartile range (IQR).
Psychometric elements of E-SUQ were examined in three parts. Firstly, the factorability of the 10 items was examined to determine the suitability of the data to undergo factor analysis. The sampling adequacy was assessed using the Kaiser-Meyer-Olkin (KMO), whereas the appropriateness of data was conducted using Bartlett’s test of sphericity. The data are considered to be suitable for factor analysis when the KMO value is >0.50 [31] and if the p-value of Bartlett’s test of sphericity is <0.05 [32].
Secondly, factor extraction using principal component analysis (PCA) was conducted to identify the dimensionality of the 10 items of E-SUQ. The number of factors to retain was determined using the following tests: the rule of eigenvalue >1 according to Kaiser’s Criteria, >50% cumulative percentage of variance and the scree plot [33]. The retained factors were then rotated using varimax rotation with the factor loading significance set at >0.4 [34].
Thirdly, the internal consistency and test-retest reliability analyses were conducted to determine the reliability of E-SUQ. A Cronbach α coefficient and corrected item-total correlation (CITC) were used to measure internal consistency. A minimum value of Cronbach’s α coefficient of 0.7 [35] and a minimum CITC range (r) of 0.3 were considered as reliable [36]. The test-retest reliability was examined using the intra-class correlation coefficient (ICC) to assess the temporal stability of the item. The ICC estimates and their 95% confidence intervals were calculated based on the mean rating (k = 30), absolute agreement and two-way mixed-effect model (ICC [2,k]). The ICC values <0.5 indicate poor reproducibility, between 0.5 and 0.75 indicate moderate reproducibility, between 0.75 and 0.9 indicate good reproducibility, and values >0.90 indicate excellent reproducibility [37].

3. Results

3.1. Phase 1: Content Validation of the E-SUQ

Table 1 shows the calculated I-CVI for the 10 items of E-SUQ. The I-CVI were found to be between 0.85 and 1.0 after two rounds of content validity analysis. In the first round, the I-CVI score for item 5 was 0.57, in which three experts found that the statement was not clear and relevant to the construct being measured. According to the three experts who gave low scores for item 5, they found that the sentence was unclear as there was no example given for the content of self-management booklet such as My Profile, My Cardiovascular Risks, My Treatment Targets, My Check-Up, My Weight Management, My Smoking Habit, My Self-Management, and My Medication sections. Therefore, item 5 was revised and examples were added to make the item clearer and more relevant. The three expert panels who scored low for item 5 were invited to rate it again in the second round of content validation. The I-CVI for item 5 after the second round was 1.0. The calculated S-CVI/Ave for the 10 items after the second round was 0.98. Therefore, all 10 items were retained.

3.2. Phase 2: Face Validation of the E-SUQ

The face validation was performed on 10 participants. Five participants gave suggestions to improve the wording of item 7 for better understanding and clarity. They suggested that item 7, which began with “I would imagine that” (“Saya membayangkan”), was changed to “I think that” (“Saya berpendapat”). Therefore, item 7 of the questionnaire was revised as suggested by the participants. This amendment is consistent with the other items, which encourage participants to express their personal views through opening statements such as “ I think that” or “I find that”. I-FVI was calculated and the results are shown in Table 2. The calculated I-FVI ranged between 0.9 and 1.0 for each item, which indicates an acceptable I-FVI level. The refined ESUQ was ready to undergo psychometric evaluation (Appendix B).

3.3. Phase 3: Psychometric Evaluation of E-SUQ

3.3.1. Recruitment for Field Testing

The EMPOWER-SUSTAIN Global CV Risks Self-Management Booklet© was distributed to 677 patients with MetS from October 2019 to March 2020. Patient recruitment for the usability data collection was done 6 months later in December 2020 to March 2021. In total, 210 patients were approached and invited to participate in the study when they attended the clinic for their follow-up appointment. Out of these, 205 patients (97.6%) fulfilled the eligibility criteria and agreed to participate. A total of 205 participants were recruited and completed the E-SUQ.

3.3.2. Demographic and Clinical Characteristics of Participants

Table 3 summarizes the demographic and clinical characteristics of the participants in this study. The mean age was 63.5 (SD ± 7.5). The age of the participants ranged between 42 and 79 years old. Out of 205 participants, the majority were males (53.2%), Malay (88.3%), married (85.4%), and had secondary and tertiary educational level (93.7%), retiree (60.5%), came from the B40 income group (52.2%), obese (62%), never smoke (72.7%) and reported to have good health status (87.8%).

3.3.3. Psychometric Properties

A total of 205 responses were included in the psychometric evaluation. There was no missing response to the questionnaire items, indicating good quality of the data. The means score of E-SUQ was 77.3 (SD ± 13.8). In the correlation matrix, all of the items had a correlation of ≥0.30 with at least one other item, indicating reasonable factorability. The sample size for this study was adequate, as evidenced by the high KMO value of 0.871 and Bartlett’s test of sphericity p-value of <0.05. These two values emphasized the appropriateness for factor analysis.
All items of the E-SUQ underwent PCA with subsequent varimax rotations. The analysis extracted two factors with eigenvalues >1 according to the Kaiser’s Criteria. The two extracted factors explained 60.6% of the cumulative percentage of variance. The elbow of the scree plot occurred between the second and third component, suggesting two factors to be retained.
Table 4 shows the loading values based on the PCA and varimax rotation analysis. The loading values were above the minimum cut-off point value of 0.40, ranging between 0.484 and 0.804. The highest and lowest communalities were for item 7 with a value of 0.690 and item 2 with a value of 0.383, respectively. This means that item 7 accounted for 69.0% of its variability. All the 10 items of E-SUQ were valid and can be used to measure usability of the self-management booklet. Items number 1, 3, 5, 7 and 9 are positively phrased statements loaded onto Factor 1. Items number 2, 4, 6, 8 and 10 are negatively phrased statements loaded onto Factor 2. Therefore, Factor 1 is labeled as “Positive Tone” and Factor 2 is labeled as “Negative Tone”. These findings indicate that E-SUQ is characterised by the Positive/Negative Tone model.
In addition, the reliability analysis was conducted to assess the internal consistency of the items in E-SUQ. Table 5 shows the results of the analysis. The calculated CITC was between 0.30 and 0.52. The overall Cronbach’s α coefficient for E-SUQ was 0.77. The Cronbach’s α coefficient for Factor 1 was 0.88 and for Factor 2 was 0.75. All the values met the minimum cut-off of 0.7, indicating that E-SUQ was reliable.
Table 6 shows the test-retest reliability results among 30 participants. The demographic characteristics of the 30 participants were similar to the other 175 participants who were not involved in the test-retest reliability study (Appendix C). The overall ICC value was 0.85, indicating that ESUQ was stable over time. The individual item values ranged between 0.58 and 0.88, indicating good reproducibility.

4. Discussion

To the best of our knowledge, there is no published study on the validation of a usability questionnaire to assess a self-management booklet. Our study is the first to demonstrate that E-SUQ is a valid and reliable tool to assess the usability of the EMPOWER-SUSTAIN Global CV Risks Self-Management Booklet©. E-SUQ also underwent a thorough adaptation and validation process based on the recommended guidelines [20].
The data set of our study is of good quality as there were no missing values. The mean score of E-SUQ among patients with MetS was 77.3 (SD ± 13.8). Direct comparison with other studies assessing usability of the self-management booklet could not be made as there was no published literature in this area. Our findings can be compared with usability of mobile apps as there is numerous published evidence in this area. The mean score in our study is comparable to the mean score of SKAMA (72.9 (SD ± 11.5)) which was used to assess a colorectal community education mobile app in Malaysia [38]. Other validation studies that used the original SUS had reported higher mean scores. These include studies among patients with type 2 diabetes mellitus in Utah which showed a mean score of 80.5 (SD ± 11.5) [39] and among patients living with chronic pain in Norway which showed mean score of 85.7 (SD ± 12.9) [40]. In contrast, a usability study of a cognitive behavioural therapy app using SUS among mental healthcare providers in six European countries yielded a lower mean score of 67.9 (SD ± 18.8) [41]. This might be due to comprehension problems as there was no translation of SUS to the local language in these European countries.
Content validation was conducted twice in our study due to the need to improve clarity and relevance for item 5. Several studies supported multiple iterations in the process of content validation to improve the clarity and relevance of the items representing the underlying construct [26,42,43]. With regards to face validity, ESUQ was found to be clear and comprehensible with I-FVI ranged between 0.9 and 1.0. Our finding is comparable to the I-FVI of SKAMA which ranged between 0.8 and 1.0 [19].
Factor analysis in this study revealed a two-factor solution in which items 1, 3, 5, 7 and 9 loaded onto Factor 1 (“Positive Tone”) and items 2, 4, 6, 8 and 10 loaded onto Factor 2 (“Negative Tone”). This is known as the tone model, which has been described in various other studies [41,44,45,46]. However, two studies did not support the tone model as it was of limited theoretical interest in relation to usability [44,45]. They recommended to interpret SUS as a unidimensional measure of perceived usability [44,45]. A more recent study using confirmatory factor analysis (CFA) supported a two-factor model of “Usability” and “Learnability” as the best fitted model compared to the unidimensional or the tone model [41].
The overall Cronbach’s α coefficient found in our study was 0.77, which indicates that E-SUQ is reliable to assess the usability of a self-management booklet. This finding is comparable to SKAMA, which has a Cronbach α of 0.85 [19]. Likewise, our finding is also comparable to the reliability of the SUS in other studies which showed Cronbach’s α of 0.91 [41] and 0.88 [46].
In terms of test-retest reliability, our study demonstrates an overall value of 0.85, indicating good reproducibility. This finding is comparable to the Persian version of SUS, where the value of overall ICC was 0.96 [47]. In contrast, the Portuguese version of SUS showed a weak overall ICC value of 0.36 [48].

4.1. Strength and Limitation

The strengths of this study include having a good quality data set as there was no missing value and the mean score showed good usability at 77.3 (SD ± 13.8). Another strength of this study was the high response rate of 97.6%. Our study is the first to demonstrate that E-SUQ is a valid and reliable tool to assess the usability of a self-management booklet. To date, there is no other published questionnaire available that can be regarded as an “anchor tool” or “gold standard”. Therefore, convergent validity could not be performed. There were also limitations in this study. Firstly, the population was from a university primary care clinic with Malay ethnicity (88.3%) constituting the vast majority of participants. As a result, the outcome of this validation study may not be generalizable to other primary care clinics in Malaysia with multiracial populations such as Chinese and Indian. Secondly, patients with MetS who were given the EMPOWER-SUSTAIN Global CV Risks Self-Management Booklet© were chosen conveniently. This may introduce sampling bias. To minimize this, all patients who were identified to have the booklet on their appointment day were invited to participate in the study. Thirdly, CFA could not be conducted to confirm the dimensionality of the item as it requires a larger sample size of at least 300. This was not feasible considering the limited time frame given to complete this study. The COVID-19 pandemic also had an impact on the number of patients coming to our clinic, resulting in a longer time taken to collect the data. Lastly, the validated E-SUQ questionnaire can only be utilized by those who can read and understand the Malay language.

4.2. Implication for Further Research and Clinical Practice

This study has proven that E-SUQ is a valid and reliable questionnaire to assess the level of usability of the EMPOWER-SUSTAIN Global CV Risks Self-Management Booklet© among patients with MetS attending a university primary care clinic. Further validation study involving patients from other primary care clinics in Malaysia is recommended to improve its generalizability. ESUQ should also be translated into Mandarin and Tamil so that the level of usability can be assessed among other major ethnic groups in Malaysia such as Chinese and Indian. As there were many controversies with the factor structure of the original SUS, future studies using CFA should be performed to confirm the dimensionality of E-SUQ. Further research to determine the factors associated with usability of the booklet is also recommended to support its widespread use among patients with MetS.

5. Conclusions

E-SUQ is a valid and reliable instrument to assess the level of usability of a self-management booklet among patients with MetS in primary care. The psychometric properties of the final 10-item E-SUQ show that it has good construct validity, high internal consistency and good test-retest reproducibility. Establishing usability of a self-management booklet will support its widespread use which, in turn, would enhance patient’s empowerment and improve their health outcomes.

Author Contributions

Conceptualization, S.F.B.-S. and A.S.R.; data curation, N.H.S.; formal analysis, N.H.S. and S.F.B.-S.; funding acquisition, A.S.R.; methodology, S.F.B.-S. and A.S.R.; supervision, S.F.B.-S. and A.S.R.; writing—original draft, N.H.S.; writing—review and editing, S.F.B.-S. and A.S.R. All authors have read and agreed to the published version of the manuscript.

Funding

This study was funded by the Ministry of Higher Education (MOHE), Malaysia: Prototype Research Grant Scheme (PRGS) No. PRGS/1/2018/SKK05/UiTM/01/2 or 600-IRMI/PRGS 5/3 (003/2019). The funding body did not play any role in the design of the study; in data collection, analysis or interpretation; or in the writing of the manuscript.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Research Ethics Committee of Universiti Teknologi MARA ((600-TNCPI (5/1/6)/REC/61/19], 3 December 2019). Permission to use the Malay version of SUS (SKAMA) was obtained from the researcher who translated and validated SUS into SKAMA.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Data are kept at the Department of Primary Care Medicine, Universiti Teknologi MARA, in Selangor, Malaysia. Data will be shared upon request and they are subjected to the data protection regulations.

Acknowledgments

We would like to thank Muhamad Fadhil Mohamad Marzuki for granting us the permission to adapt the SKAMA into the E-SUQ. We also would like to thank all the experts at the Primary Care Medicine Department, UiTM, for their contribution to this study. We would like to extend our gratitude to all the participants who were involved in this study.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A. SKAMA

  • Saya rasa saya ingin kerap menggunakan aplikasi mudah alih ini.
  • Saya rasa aplikasi mudah alih ini sesuatu yang rumit.
  • Saya rasa aplikasi mudah alih ini mudah untuk digunakan.
  • Saya memerlukan bantuan daripada orang lain untuk menggunakan aplikasi mudah alih ini.
  • Saya mendapati maklumat dalam aplikasi mudah alih ini saling bersepadu (berhubungkait) dengan baik.
  • Saya rasa terdapat banyak kandungan di dalam aplikasi mudah alih ini yang tidak konsisten antara satu sama lain.
  • Saya membayangkan bahawa kebanyakan orang akan cepat belajar menggunakan aplikasi mudah alih ini.
  • Saya mendapati aplikasi mudah alih ini leceh untuk digunakan.
  • Saya berasa yakin menggunakan aplikasi mudah alih ini.
  • Saya perlu belajar terlalu banyak perkara sebelum boleh menggunakan aplikasi mudah alih ini.

Appendix B. E-SUQ

  • Saya rasa saya akan menggunakan buku pengawasan kendiri ini dengan kerap.
  • Saya rasa buku pengawasan kendiri ini adalah buku yang rumit.
  • Saya rasa buku pengawasan kendiri ini mudah untuk digunakan.
  • Saya memerlukan bantuan daripada orang lain untuk menggunakan buku pengawasan kendiri ini.
  • Saya mendapati isi kandungan buku pengawasan kendiri ini berhubungkait antara satu sama lain (Contoh: Faktor risiko kardiovaskular → Keputusan ujian darah → Pengurusan berat badan → Tabiat merokok → Kawalan tekanan darah/gula → Senarai ubatan)
  • Saya mendapati kandungan dalam buku pengawasan kendiri ini tidak konsisten antara satu sama lain.
  • Saya berpendapat bahawa cara penggunaan buku pengawasan kendiri ini mudah dipelajari.
  • Saya mendapati buku pengawasan kendiri ini leceh untuk digunakan.
  • Saya berasa yakin menggunakan buku pengawasan kendiri ini.
  • Saya perlu belajar banyak perkara sebelum boleh menggunakan buku pengawasan kendiri ini.

Appendix C

Table
Table A1. Characteristics of Participants Involved in the Test-Retest Reliability (n = 30) vs. Those Not Involved (n = 175).
Table A1. Characteristics of Participants Involved in the Test-Retest Reliability (n = 30) vs. Those Not Involved (n = 175).
Characteristics of Participantsn = 30n = 175Overall Mean Score of E-SUQ
Age in years, Mean (±SD)61.5 (±6.6)63.9 (±7.6)
Gender, n (%)
Male17 (56.7)92 (52.6)
Female13 (43.3)83 (47.4)
Ethnicity, n (%)
Malay22 (73.3)159 (90.9)
Chinese6 (20.0)9 (5.1)
Indian2 (6.7)5 (2.9)
Others0 (0.0)2 (1.1)
Marital Status, n (%)
Single0 (0.0) 4 (2.3)
Married26 (86.7)149 (85.1)
Widow/Widower3 (10.0)19 (10.9)
Divorced1 (3.3)3 (1.7)
Educational Level, n (%)
No formal education0 (0.0)1 (0.6)
Primary2 (6.7)10 (5.7)
Secondary12 (40.0)81 (46.3)
Tertiary16 (53.3)83 (47.4)
Occupation, n (%)
Unemployed7 (23.3)33 (18.9)
Employed6 (20.0)35 (20.0)
Retiree17 (56.7)107 (61.1)
Household income per month *, n (%)
B40 (<RM 4360)16 (53.3)91 (52.0)
M40 (RM 4360–9619)11 (36.7)60 (34.3)
T20 (>RM 9619)3 (10.0)24 (13.7)
Body Mass Index, kg/m², n (%)
Underweight (<18.5)0 (0.0)0 (0.0)
Normal (18.5–22.9)1 (3.3)11 (6.3)
Overweight (23.0–27.4)26 (86.7)40 (22.9)
Obese (≥27.5)3 (10.0)124 (70.9)
Smoking status, n (%)
Never smoke21 (70.0)128 (73.1)
Active smoker4 (13.3)19 (10.9)
Ex-smoker5 (16.7)28 (16.0)
Self-report health status, n (%)
Excellent0 (0.0)2 (1.1)
Very Good2 (6.7)8 (4.6)
Good27 (90.0)153 (87.4)
Fair1 (3.3)12 (6.9)
Mean score of E-SUQ, Mean (±SD)77.1 (SD ± 21.2)77.5 (SD ± 14.1)77.3 (SD ± 13.8)
* Based on Report of Household Income and Basic Amenities Survey 2016 by Department of Statistics, Malaysia.

References

  1. World Health Organization. Global Status Report on Noncommunicable Diseases 2014; WHO: Geneva, Switzerland, 2014; Available online: http://www.who.int/nmh/publications/ncd-status-report-2014/en/ (accessed on 29 June 2020).
  2. Alberti, K.G.M.M.; Eckel, R.H.; Grundy, S.M.; Zimmet, P.Z.; Cleeman, J.I.; Donato, K.A.; Fruchart, J.C.; James, W.P.; Loria, C.M.; Smith, S.C., Jr. Harmonizing the metabolic syndrome: A joint interim statement of the international diabetes federation task force on epidemiology and prevention; National heart, lung, and blood institute; American heart association; World heart federation; International. Am. Hear Assoc. J. 2009, 120, 1640–1645. [Google Scholar]
  3. Gami, A.S.; Witt, B.J.; Howard, D.E.; Erwin, P.J.; Gami, L.A.; Somers, V.K.; Montori, V.M. Metabolic Syndrome and Risk of Incident Cardiovascular Events and Death. A Systematic Review and Meta-Analysis of Longitudinal Studies. J. Am. Coll. Cardiol. 2007, 49, 403–414. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. Cornier, M.-A.; Dabelea, D.; Hernandez, T.L.; Lindstrom, R.C.; Steig, A.J.; Stob, N.R.; Van Pelt, R.; Wang, H.; Eckel, R.H. The metabolic syndrome. Endocr. Rev. 2008, 29, 777–822. [Google Scholar] [CrossRef]
  5. Ramli, A.S.; Daher, A.M.; Nor-Ashikin, M.N.; Mat-Nasir, N.; Ng, K.K.; Miskan, M.; Ambigga, K.S.; Ariffin, F.; Mazapuspavina, M.Y.; Abdul-Razak, S.; et al. JIS definition identified more malaysian adults with metabolic syndrome compared to the NCEP-ATP III and IDF criteria. BioMed Res. Int. 2013, 2013, 1–10. [Google Scholar] [CrossRef]
  6. Bodenheimer, T.; Wagner, E.H.; Grumbach, K. Improving primary care for patients with chronic illness. J. Am. Med. Assoc. 2002, 288, 1775–1779. [Google Scholar] [CrossRef]
  7. Baptista, D.R.; Wiens, A.; Pontarolo, R.; Regis, L.; Reis, W.C.T.; Correr, C.J. The chronic care model for type 2 diabetes: A systematic review. Diabetol. Metab. Syndr. 2016, 8, 1–7. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  8. Bodenheimer, T.; Willard-grace, R. Life style medicine. Lifestyle Med. 2016, 89–96. [Google Scholar] [CrossRef]
  9. Wagner, E.H.; Austin, B.T.; Davis, C.; Hindmarsh, M.; Schaefer, J.; Bonomi, A. Improving chronic illness care: Translating evidence into action. Health Aff. 2001, 20, 64–78. [Google Scholar] [CrossRef] [Green Version]
  10. Blackford, K.; Jancey, J.; Lee, A.H.; James, A.P.; Waddell, T.; Howat, P. Home-based lifestyle intervention for rural adults improves metabolic syndrome parameters and cardiovascular risk factors: A randomised controlled trial. Prev. Med. 2016, 89, 15–22. [Google Scholar] [CrossRef] [Green Version]
  11. Jahangiry, L.; Montazeri, A.; Najafi, M.; Yaseri, M.; Farhangi, M.A. An interactive web-based intervention on nutritional status, physical activity and health-related quality of life in patient with metabolic syndrome: A randomized-controlled trial (The Red Ruby Study). Nutr. Diabetes 2017, 7, e240. [Google Scholar] [CrossRef] [Green Version]
  12. Shin, N.M.; Choi, J.W.; Cho, I.; Park, B.J. Self-Management Program for Heart Healthy Behavior among Middle- and Old-Aged Korean Women at Risk for Metabolic Syndrome. J. Cardiovasc. Nurs. 2017, 32, 8–16. [Google Scholar] [CrossRef]
  13. Suwankruhasn, N.; Pothiban, L.; Panuthai, S.; Boonchuang, P. Effects of a Self-management Support Program for Thai People Diagnosed with Metabolic Syndrome. Pacific Rim Int. J. Nurs. Res. 2013, 17, 371–383. [Google Scholar]
  14. Tran, V.D.; Lee, A.H.; Jancey, J.; James, A.P.; Howat, P.; Mai, L.T.P. Physical activity and nutrition behaviour outcomes of a cluster-randomized controlled trial for adults with metabolic syndrome in Vietnam. Trials 2017, 18, 1–10. [Google Scholar] [CrossRef] [Green Version]
  15. Ramli, A.S.; Lakshmanan, S.; Haniff, J.; Selvarajah, S.; Tong, S.F.; Bujang, M.A.; Abdul-Razak, S.; Shafie, A.A.; Tg-Abu-Bakar-Sidik, T.M.I.; Bujang, M.A.; et al. Study protocol of EMPOWER Participatory Action Research (EMPOWER-PAR): A pragmatic cluster randomised controlled trial of multifaceted chronic disease management strategies to improve diabetes and hypertension outcomes in primary care. BMC Fam. Pract. 2014, 15, 1–16. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  16. Ramli, A.S.; Selvarajah, S.; Daud, M.H.; Haniff, J.; Abdul-Razak, S.; Tg-Abu-Bakar-Sidik, T.M.I.; Bujang, M.A.; Chew, B.H.; Rahman, T.; Tong, S.F.; et al. Effectiveness of the EMPOWER-PAR Intervention in Improving Clinical Outcomes of Type 2 Diabetes Mellitus in Primary Care: A Pragmatic Cluster Randomised Controlled Trial. BMC Fam. Pract. 2016, 17, 1–18. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  17. Bevan, N. International standards for HCl and usability. Int. J. Hum. Comput. Stud. 2001, 55, 533–552. [Google Scholar] [CrossRef] [Green Version]
  18. Bangor, A.; Kortum, P.T.; Miller, J.T. An empirical evaluation of the system usability scale. Int. J. Hum. Comput. Interact. 2008, 24, 574–594. [Google Scholar] [CrossRef]
  19. Mohamad-Marzuki, M.F.; Yaacob, N.A.; Yaacob, N.M. Translation, cross-cultural adaptation, and validation of the Malay version of the system usability scale questionnaire for the assessment of mobile apps. J. Med. Internet Res. 2018, 20, 1–7. [Google Scholar] [CrossRef]
  20. Mokkink, L.B.; Terwee, C.B.; Knol, D.L.; Stratford, P.W.; Alonso, J.; Patrick, D.L.; Bouter, L.M.; de Vet, H.C. Protocol of the COSMIN study: COnsensus-based Standards for the selection of health Measurement INstruments. BMC Med. Res. Methodol. 2006, 6, 1–7. [Google Scholar] [CrossRef] [Green Version]
  21. Wild, D.; Grove, A.; Eremenco, S.; McElroy, S.; Verjee-Lorenz, A.; Erikson, P. Principles of Good Practice for the Translation and Cultural Adaptation Process for Patient-Reported Outcomes (PRO) Measures: Report of the ISPOR Task Force for Translation and Cultural Adaptation. Value Heal. 2005, 8, 94–104. [Google Scholar] [CrossRef] [Green Version]
  22. Brooke, J. SUS—A quick and dirty usability scale. Usability Eval. Ind. 1996, 189, 4–7. [Google Scholar]
  23. Sauro, J. SUStisfied? Little-Known System Usability Scale Facts. User Exp. 2011, pp. 14–15. Available online: https://uxpamagazine.org/sustified/ (accessed on 2 July 2021).
  24. Zamanzadeh, V.; Ghahramanian, A.; Rassouli, M.; Abbaszadeh, A.; Alavi-Majd, H.; Nikanfar, A.R. Design and Implementation Content Validity Study: Development of an instrument for measuring Patient-Centered Communication. J. Caring Sci. 2015, 4, 165–178. [Google Scholar] [CrossRef]
  25. Polit, D.F.; Beck, C.T. The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Res. Nurs. Health 2006, 29, 489–497. [Google Scholar] [CrossRef] [Green Version]
  26. Lynn, M.R. Determination and quantification of content validity. Nurs. Res. 1986, 35, 382–385. [Google Scholar] [CrossRef] [PubMed]
  27. Waltz, C.F.; Strictland, O.L.; Lentz, E.R. Measurement in Nursing and Health Research, 4th ed.; Springer Publishing Company: New York, NY, USA, 2010. [Google Scholar]
  28. Cook, D.A.; Beckman, T.J. Current concepts in validity and reliability for psychometric instruments: Theory and application. Am. J. Med. 2006, 119, 166.e7–166.e16. [Google Scholar] [CrossRef] [PubMed]
  29. Yusoff, M.S.B. ABC of Response Process Validation and Face Validity Index Calculation. Educ. Med. J. 2019, 11, 55–61. [Google Scholar] [CrossRef]
  30. Goldberg, L.R.; Velicer, W.F. Principles of exploratory factor analysis. Differ. Norm. Abnorm. Personal. 2006, 2, 209–237. [Google Scholar] [CrossRef] [Green Version]
  31. Kaiser, H.F. Educational and Psychological Measurement. Ser. Rev. 1982, 8, 83–85. Available online: http://0-www-tandfonline-com.brum.beds.ac.uk/doi/abs/10.1080/00987913.1982.10763392 (accessed on 29 June 2021).
  32. Bentler, P.M. Comparative Fit Indexes in Structural Models. Am. Psychol. Assoc. 1990, 107, 238–246. [Google Scholar] [CrossRef]
  33. Thompson, B.; Daniel, L.G. Factor Analytic Evidence for the Construct Validity of Scores: A Historical Overview and Some Guidelines. Educ. Psychol. Meas. 1996, 56, 197–208. [Google Scholar] [CrossRef]
  34. Field, A. Discovering Statistics Using SPSS ISM (London, England) Introducing Statistical Methods Series; SAGE Publication: London, UK, 2009; pp. 654–655. [Google Scholar]
  35. Eisinga, R.; Grotenhuis, M.T.; Pelzer, B. The reliability of a two-item scale: Pearson, Cronbach, or Spearman-Brown? Int. J. Public Health 2013, 58, 637–642. [Google Scholar] [CrossRef]
  36. Cristobal, E.; Flavián, C.; Guinalíu, M. Perceived e-service quality (PeSQ): Measurement validation and effects on consumer satisfaction and web site loyalty. Manag. Serv. Theory Pract. 2007, 17, 317–340. [Google Scholar] [CrossRef]
  37. Koo, T.K.; Li, M.Y. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J. Chiropr. Med. 2016, 15, 155–163. [Google Scholar] [CrossRef] [Green Version]
  38. Mohamad-Marzuki, M.F.; Yaacob, N.A.; Yaacob, N.M.; Abu-Hassan, M.R.; Ahmad, S.B. Usable Mobile App for Community Education on Colorectal Cancer: Development Process and Usability Study. JMIR Hum. Factors 2019, 6, 12103. [Google Scholar] [CrossRef]
  39. Georgsson, M.; Staggers, N. Quantifying usability: An evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics. J. Am. Med. Inform. Assoc. 2016, 23, 5–11. [Google Scholar] [CrossRef]
  40. Bostrom, K.; Borosund, E.; Varsi, C.; Eide, H.; Nordang, E.F.; Schreurs, K.M.G.; Waxenberg, L.B.; Weiss, K.E.; Morrison, E.J.; Smastuen, M.C.; et al. Digital self-management in support of patients living with chronic pain: Feasibility pilot study. JMIR Form. Res. 2020, 4, e23893. [Google Scholar] [CrossRef]
  41. Mol, M.; Van-Schaik, A.; Dozeman, E.; Ruwaard, J.; Vis, C.; Ebert, D.D.; Etzelmueller, A.; Mathiasen, K.; Moles, B.; Mora, T.; et al. Dimensionality of the system usability scale among professionals using internet-based interventions for depression: A confirmatory factor analysis. BMC Psychiatry 2020, 20, 1–10. [Google Scholar] [CrossRef]
  42. Haynes, S.N.; Richard, D.C.S.; Kubany, E.S. Content Validity in Psychological Assessment: A Functional Approach to Concepts and Methods. Psychol. Assess. 1995, 7, 238–247. [Google Scholar] [CrossRef]
  43. Polit, D.F.; Beck, C.T.; Owen, S.V. Focus on Research Methods Is the CVI an Acceptable Indicator of Content Validity? Appraisal and Recommendations. Res. Nurs. Heal. 2007, 30, 459–467. [Google Scholar] [CrossRef]
  44. Lewis, J.R.; Sauro, J. Revisiting the Factor Structure of the System Usability Scale. J. Usability Stud. 2017, 12, 183–192. [Google Scholar]
  45. Lewis, J.R.; Brown, J.; Mayes, D.K. Psychometric Evaluation of the EMO and the SUS in the Context of a Large-Sample Unmoderated Usability Study. Int. J. Hum. Comput. Interact. 2015, 31, 545–553. [Google Scholar] [CrossRef]
  46. Kortum, P.; Sorber, M. Measuring the Usability of Mobile Applications for Phones and Tablets. Int. J. Hum. Comput. Interact. 2015, 31, 518–529. [Google Scholar] [CrossRef]
  47. Dianat, I.; Ghanbari, Z.; Asgharijafarabadi, M. Psychometric Properties of the Persian Language Version of the System Usability Scale. Health Promot. Perspect. 2014, 4, 82–89. [Google Scholar] [CrossRef]
  48. Isabel, A.; Filipa, A.; Queirós, A.; Silva, A. European Portuguese validation of the System Usability Scale (SUS). Procedia Comput. Sci. 2015, 67, 293–300. [Google Scholar] [CrossRef] [Green Version]
Ijerph 18 09405 g001 550
Figure 1. Flowchart of the study.
Figure 1. Flowchart of the study.
Ijerph 18 09405 g001
Table
Table 1. Final round of content validity index.
Table 1. Final round of content validity index.
ItemE a 1E2E3E4E5E6E7Experts in AgreementI-CVI b
Q1334443471.0
Q2444444471.0
Q3444444471.0
Q4444444471.0
Q5334443371.0
Q6444442460.85
Q7333444471.0
Q8444444471.0
Q9444443471.0
Q10444444471.0
Content validity index average (S-CVI/Ave)0.98
a E: Expert; b I-CVI: Item content validity index.
Table
Table 2. Face validity index.
Table 2. Face validity index.
ItemR a 1R2R3R4R5R6R7R8R9R10Raters in AgreementI-FVI b
Q14344443434101.0
Q24444444344101.0
Q34444444444101.0
Q44444444444101.0
Q54444444444101.0
Q6344444424490.9
Q73444444444101.0
Q84444444334101.0
Q94444434444101.0
Q104334424443100.9
a R: Rater b I-FVI: Item face validity index.
Table
Table 3. Demographics and clinical characteristics of participants (n = 205).
Table 3. Demographics and clinical characteristics of participants (n = 205).
Characteristics of ParticipantsFrequency, n (%)Mean (±SD)
Age (years) 63.5 (±7.5)
Gender
Male109 (53.2)
Female96 (46.8)
Ethnicity
Malay181 (88.3)
Chinese15 (17.3)
Indian7 (3.4)
Others2 (1.0)
Marital Status
Single4 (2.0)
Married175 (85.4)
Widow/Widower22 (10.7)
Divorced4 (2.0)
Educational Level
No formal education1 (0.5)
Primary12 (5.9)
Secondary93 (45.4)
Tertiary99 (48.3)
Occupation
Unemployed40 (19.5)
Employed41 (20.0)
Retiree124 (60.5)
Household income per month *
B40 (<RM 4360)107 (52.2)
M40 (RM 4360–9619)71 (34.6)
T20 (>RM 9619)27 (13.2)
Body Mass Index, kg/m²
Underweight (<18.5)0 (0.0)
Normal (18.5–22.9)12 (5.9)
Overweight (23.0–27.4)66 (32.1)
Obese (≥27.5)127 (62.0)
Smoking status
Never smoke149 (72.7)
Active smoker23 (11.2)
Ex-smoker33 (16.1)
Self-report health status
Excellent2 (1.0)
Very Good10 (4.9)
Good180 (87.8)
Fair13 (6.3)
* Based on Report of Household Income and Basic Amenities Survey 2016 by Department of Statistics, Malaysia.
Table
Table 4. Factor loading.
Table 4. Factor loading.
ItemQuestionFactor 1
12
ESUQ1I think that I would like to use this self-management booklet frequently.0.804
ESUQ2I found the self-management booklet unnecessarily complex. 0.484
ESUQ3I thought the self-management booklet was easy to use.0.796
ESUQ4I think I would need the support of a technical person to be able to use the self-management booklet. 0.662
ESUQ5I found that the contents of the self-management booklet were related to each other (Example: Cardiovascular risk factors → Blood test results → Weight management → Smoking habit → Home blood pressure monitoring/Self blood sugar monitoring → List of medication).0.803
ESUQ6I found that there was inconsistency in the content of self-management booklet. 0.734
ESUQ7I think the self-management booklet is easy to learn.0797
ESUQ8I found the self-management booklet is difficult to use. 0.695
ESUQ9I felt confident to use the self-management booklet.0.794
ESUQ10I needed to learn a lot of things before I could get going with the self-management booklet. 0.779
Table
Table 5. Internal consistency.
Table 5. Internal consistency.
ItemScale Mean if Item DeletedScale Variance if Item DeletedCorrected Item-Total CorrelationSquared Multiple CorrelationCronbach’s Alpha if Item Deleted
ESUQ 1 39.37 8.38 0.49 0.80 0.75
ESUQ 2 39.30 8.91 0.30 0.43 0.77
ESUQ 339.578.050.510.620.75
ESUQ 4 39.57 8.25 0.52 0.60 0.75
ESUQ 5 39.37 8.31 0.52 0.83 0.75
ESUQ 6 39.50 8.47 0.35 0.57 0.77
ESUQ 7 39.53 8.33 0.41 0.53 0.76
ESUQ 8 39.47 8.05 0.47 0.70 0.75
ESUQ 9 39.40 8.25 0.46 0.52 0.75
ESUQ 10 39.43 8.67 0.40 0.55 0.76
Overall Cronbach’s alpha value0.77
Table
Table 6. Test-retest reliability intra-class correlation coefficient (n = 30).
Table 6. Test-retest reliability intra-class correlation coefficient (n = 30).
ItemQuestionICC [2,k] (95% CI)
ESUQ 1I think that I would like to use this self-management booklet frequently.0.80 (0.57–0.90)
ESUQ 2I found the self-management booklet unnecessarily complex.0.66 (0.29–0.84)
ESUQ 3I thought the self-management booklet was easy to use.0.58 (0.11–0.80)
ESUQ 4I think I would need the support of a technical person to be able to use the self-management booklet.0.73 (0.43–0.87)
ESUQ 5I found that the contents of the self-management booklet were related to each other (Example: Cardiovascular risk factors → Blood test results → Weight management → Smoking habit → Home blood pressure monitoring/Self blood sugar monitoring → List of medication)0.76 (0.49–0.88)
ESUQ 6I found that there was inconsistency in the content of the self-management booklet.0.71 (0.40–0.86)
ESUQ 7I think the self-management booklet is easy to learn.0.88 (0.75–0.94)
ESUQ 8I found the self-management booklet is difficult to use.0.80 (0.58–0.90)
ESUQ 9I felt confident to use the self-management booklet.0.57 (0.09–0.80)
ESUQ 10I needed to learn a lot of things before I could get going with the self-management booklet.0.67 (0.31–0.84)
Overall value0.85
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Sahar, N.H.; Badlishah-Sham, S.F.; Ramli, A.S. Adaptation and Psychometric Validation of the EMPOWER-SUSTAIN Usability Questionnaire (E-SUQ) among Patients with Metabolic Syndrome in Primary Care. Int. J. Environ. Res. Public Health 2021, 18, 9405. https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph18179405

AMA Style

Sahar NH, Badlishah-Sham SF, Ramli AS. Adaptation and Psychometric Validation of the EMPOWER-SUSTAIN Usability Questionnaire (E-SUQ) among Patients with Metabolic Syndrome in Primary Care. International Journal of Environmental Research and Public Health. 2021; 18(17):9405. https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph18179405

Chicago/Turabian Style

Sahar, Nurul Hidayatullaila, Siti Fatimah Badlishah-Sham, and Anis Safura Ramli. 2021. "Adaptation and Psychometric Validation of the EMPOWER-SUSTAIN Usability Questionnaire (E-SUQ) among Patients with Metabolic Syndrome in Primary Care" International Journal of Environmental Research and Public Health 18, no. 17: 9405. https://0-doi-org.brum.beds.ac.uk/10.3390/ijerph18179405

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop