2. Materials and Methods
2.1. Study Sites and Participants
2.2. Procedure and Materials
2.3. Data Collection
- How applicable and comprehensible was the content of the Org-HLR tool?
- What were the benefits of undertaking the assessment?
- What were the key strengths of the Org-HLR tool and assessment process?
- What were the key limitations of the Org-HLR tool and assessment process?
- How can the Org-HLR tool and assessment process be improved?
2.4. Data Analysis
2.5. Ethics Approval
3.1. Initial Orientation to the Org-HLR Assessment Process and Selected Approach across Participating Sites
3.2. Applicability of the Org-HLR Tool Content
3.3. Strengths of the Org-HLR Tool and Assessment Process
3.4. Benefits of Undertaking the Org-HLR Assessment Process
3.5. Limitations of the Org-HLR Tool and Assessment Process
3.6. Refinements to the Org-HLR Tool and Assessment Process
Considerations for Future Users
Conflicts of Interest
- Nutbeam, D. Health Promotion Glossary, World Health Organization. 1998. Available online: http://www.who.int/healthpromotion/about/HPR%20Glossary%201998.pdf?ua=1 (accessed on 5 August 2014).
- Herndon, J.B.; Chaney, M.; Carden, D. Health literacy and emergency department out-comes: A systematic review. Ann. Emerg. Med. 2011, 57, 334–345. [Google Scholar] [CrossRef] [PubMed]
- Wolf, M.S.; Gazmararian, J.A.; Baker, D.W. Health literacy and functional health status among older adults. Arch. Intern. Med. 2005, 165, 1946–1952. [Google Scholar] [CrossRef] [PubMed]
- Tokuda, Y.; Doba, N.; Butler, J.P.; Paasche-Orlow, M.K. Health literacy and physical and psychological wellbeing in Japanese adults. Patient Educ. Couns. 2009, 75, 411–417. [Google Scholar] [CrossRef] [PubMed]
- Jessup, R.L.; Osborne, R.H.; Beauchamp, A.; Bourne, A.; Buchbinder, R. Health literacy of recently hospitalised patients: A cross-sectional survey using the Health Literacy Questionnaire (HLQ). BMC Health Serv. Res. 2017, 17, 52. [Google Scholar] [CrossRef] [PubMed]
- Institute of Medicine. Health Literacy: A Prescription to End Confusion, Institute of Medicine, retrieved May 23 2014. Available online: http://www.nationalacademies.org/hmd/Reports/2004/Health-Literacy-A-Prescription-to-End-Confusion.aspx (accessed on 23 May 2014).
- Ishikawa, H.; Yano, E. Patient health literacy and participation in the health-care process. Health Expect. 2008, 11, 113–122. [Google Scholar] [CrossRef] [PubMed]
- Van der Heide, I.; Uiters, E.; Rademakers, J.; Struijs, J.N.; Schuit, A.J.; Baan, C.A. Associations among health literacy, diabetes knowledge, and self-management behavior in adults with diabetes: Results of a dutch cross-sectional study. J. Health Commun. 2014, 19 (Suppl. 2), 115–131. [Google Scholar] [CrossRef] [PubMed]
- Pleasant, A.; Cabe, M.A.; Martin, L.; Rikard, R.V. A Prescription is Not Enough: Improving Public Health with Health Literacy; Institute of Medicine: Washington, DC, USA, 2013. [Google Scholar]
- Koh, H.K.; Baur, C.; Brach, C.; Harris, L.M.; Rowden, J.N. Toward a systems approach to health literacy research. J. Health Commun. 2013, 18, 1–5. [Google Scholar] [CrossRef] [PubMed]
- DeWalt, D.; McNeill, J. Integrating Health Literacy with Health Care Performance Measurement; Institute of Medicine: Washington, DC, USA, 2013. [Google Scholar]
- Brach, C.; Keller, D.; Hernandez, L.M.; Baur, C.; Parker, R.; Dreyer, B.; Schyve, P.; Lemerise, A.; Schillinger, D. Ten Attributes of Health Literate Health Care Organisations: Discussion Paper; Institute of Medicine: Washington, DC, USA, 2012. [Google Scholar]
- World Health Organization Regional Office for Europe. Health Literacy: The Solid Facts; World Health Organization Regional Office for Europe: Copenhagen, Denmark, 2013. [Google Scholar]
- Trezona, A.; Dodson, S.; Osborne, R.H. Development of the Organisational Health Literacy Responsiveness (Org-HLR) Framework in Collaboration with Health and Social Services Professionals. BMC Health Serv. Res. 2017, 17, 513. [Google Scholar] [CrossRef] [PubMed]
- Ritchie, L.; Dale, B.G. Self-assessment using the business excellence model: A study of practice and process. Int. J. Prod. Econ. 2000, 66, 241–254. [Google Scholar] [CrossRef]
- Miller, R.; Bedney, B.; Guenther-Grey, C. Assessing Organizational Capacity to Deliver HIV Prevention Services Collaboratively: Tales from the Field. Health Educ. Behav. 2003, 30, 582–600. [Google Scholar] [CrossRef] [PubMed]
- Ford, M.W. Baldrige Assessment and Organizational Learning: The Need for Change Management. Qual. Manag. J. 2001, 8, 9–25. [Google Scholar] [CrossRef]
- Fountain, M. The target assessment model as an international standard for self-assessment. Total Qual. Manag. 1998, 9, S95–S99. [Google Scholar] [CrossRef]
- Trezona, A.; Dodson, S.; Osborne, R.H. Development of the Organisational Health Literacy Responsiveness (Org-HLR) self-assessment tool and process. BMC Health Serv. Res. 2018, 18, 694. [Google Scholar] [CrossRef] [PubMed]
- Department of Health and Human Services, (2015) Primary Care Partnerships, Department of Health and Human Services, retrieved 1 April 2017. Available online: https://www2.health.vic.gov.au/primary-and-community-health/primary-care/primary-care-partnerships (accessed on 1 April 2017).
- Thomas, D.R. A General Inductive Approach for Analyzing Qualitative Evaluation Data. Am. J. Eval. 2006, 27, 237–246. [Google Scholar] [CrossRef]
- Ford, M.W.; Evans, J.R. Models for Organizational Self-Assessment. Business Horizons. 2002 2 January 2017. Available online: https://www.researchgate.net/publication/4884983_Models_for_Organizational_Self-Assessment?enrichId=rgreq-1f7a6e2f3443860e63c540825c5a4e74-XXX&enrichSource=Y292ZXJQYWdlOzQ4ODQ5ODM7QVM6MTg5MTg1MzE1MDU3NjY3QDE0MjIxMTY3MDI2MjQ%3D&el=1_x_2 (accessed on 2 January 2017).
- Batterham, R.W.; Hawkins, M.; Collins, P.A.; Buchbinder, R.; Osborne, R.H. Health literacy: Applying current concepts to improve health services and reduce health inequalities. Public Health 2016, 132, 3–12. [Google Scholar] [CrossRef] [PubMed]
- Alper, J. Integrating Health Literacy, Cultural Competence and Language Access Services: Workshop Summary; The National Academies Press: Washington, DC, USA, 2016. [Google Scholar]
- Lie, D.; Carter-Pokras, O.; Braun, B.; Coleman, C. What do health literacy and cultural competence have in common? Calling for a collaborative health professional pedagogy. J. Health Commun. 2012, 17 (Suppl. 3), 13–22. [Google Scholar] [CrossRef] [PubMed]
- Levesque, J.F.; Harris, M.F.; Russell, G. Patient-centred access to health care: Conceptualising access at the interface of health systems and populations. Int. J. Health Equity Health 2013, 12, 18. [Google Scholar] [CrossRef] [PubMed]
- Australian Commission on Safety and Quality in Health Care. Health Literacy: Taking Action to Improve Safety and Quality, Australian Commission on Safety and Quality in Health Care. 2014. Available online: http://www.safetyandquality.gov.au/wp-content/uploads/2014/08/Health-Literacy-Taking-action-to-improve-safety-and-quality.pdf (accessed on 1 February 2015).
|Site||Number of Service Locations||Number of Staff *||Number of Service Users||Types of Services Delivered|
Large community health service
|>40||>850||>110,000 service users||Medical services|
|Allied health services|
|Mental health services|
|Aged and disability services|
|Refugee health services|
|Specialist health services|
|Chronic disease programs|
Large public hospital
|2||>6500||>85,000 admissions per year||Acute medical services|
|Specialist medical services|
|Aged care services|
|Site #3 |
Medium community health service
|4||>240||>5900 service users||Allied health|
|Alcohol and drug services|
|Child and family services|
|Aged and disability services|
|Refugee health services|
|Chronic disease programs|
State wide Not for Profit (social service)
|1||110||Approximately 5000 client interactions per year.||Advisory Line|
|Respite Programs (Aged, Disability, Mental Health, Older Families and Young Carers)|
|Policy and Research|
|Workplace Training and Solutions|
|Site||Rationale for Participating||Approach to Assessment||Participants|
Large community health service
|To establish a baseline of current organisational practice and performance. A follow-up assessment to be completed in two years to determine progress.||Health literacy action plan already in place—opted not to undertake the reflection and priority setting activities.|
Two self-rating workshops (2 h each) were delivered.
The first covered assessment dimensions 4, 5 and 6. The second covered assessment dimensions 2, 3 and 7.
|A group of practitioners (N = 9) from various teams participated in the first workshop.|
A group of managers/senior managers (N = 3) participated in the second workshop.
Large public hospital
|To establish a baseline of current organisational practice and performance, as well as to identify and prioritise actions for implementation.||Due to time constraints, the organisation opted not to undertake the reflection and priority setting activities.|
Two self-rating workshops (1–2 h each) were delivered.
|Practitioners and managers (N = 11) from various disciplines from the medical department participated in the workshops.|
Medium community health service
|To identify gaps in health literacy work undertaken to date, identify and prioritise new actions for the future, and engage staff from across a wider range of teams in the planning and implementation of health literacy activities.||Implemented the Org-HLR process in full.|
A combined reflection and self-rating workshop (4 h) and a priority setting workshop (2 h) were delivered.
|Practitioners and managers (N = 13) from various teams across the organisation participated in the whole process.|
State wide not for profit (social service)
|To increase staff awareness of health literacy and to increase their engagement in health literacy activities.||Implemented the Org-HLR process in full. |
Two self-rating workshops (3 h each) and a priority-setting workshop (2 h) were delivered.
The first self-rating workshop incorporated reflection activity.
|Practitioners and staff (N = 7) from various teams across the organisation participated in the whole process.|
|Evidence-based||The Org-HLR tool was informed by empirical research and developed in the Australian context.||“Staff value and appreciate the evidence base that the work has come from.”|
|Appropriate scope and breadth||The scope and breadth of the Org-HLR tool is comprehensive and appropriate for assessing whole of organisation health literacy responsiveness.||“It needs to be broad… If you’re looking at a whole of organisation approach to something, you do have to have a broad assessment.”|
|Logically structured||The Org-HLR tool is logically and appropriately structured into relevant assessment dimensions and sub-dimensions.||“The way it has been broken down into the different domains of leadership and culture, and workforce… I found that really helpful… it is good to break it down into those subsections, otherwise it can be overwhelming.”|
|Facilitated workshop format (conversation-based).||The workshops format encourages participation from a broad range of people, which enables cross-team conversations, collaboration, team building and knowledge exchange.||“It was good to have people from different parts of the organisation… Having that diversity (of staff representation) is really useful…”|
“There was something different about this process. What I liked about this process that was different was the conversational component… There was that thing of really hearing (other) experiences.”
|Generates both quantitative and qualitative data||The use of a quantitative rating system supports the identification of strengths and limitations, as well as the benchmarking and monitoring of improvements over time. |
The qualitative component supports the documentation of examples that may inform planning.
|“I think the item level (rating) is important because it can drive some of that conversation around what our weaknesses and strengths are.”|
“The other thing I really like about the rating is that idea of being able to go back and do it again and see change.”
“Examples are good. Getting people to think about, reflect on examples and jot them down, and sharing that is useful.”
|Supports organisational planning processes||The process informed or will inform organisational planning processes, including strategic plans, operational plans, and specific health literacy action plans.||“(To support our) new strategic planning process, working out where the health literacy work and plan sits, who is responsible I think this process is making that clearer for us.”|
|Supports evaluation and monitoring||The process was useful for establishing a baseline of organisational performance in health literacy responsiveness, and this will be used to monitor and evaluate improvements over time.||“The primary purpose was to provide a kind of baseline assessment, and a method for ongoing assessment… and to understand whether we have achieved the objectives of our health literacy plan.”|
|Enables cross-team collaboration||The process provided an opportunity for and encouraged cross-team discussions and collaboration on health literacy responsiveness.||“Having some forums where there is cross-team discussions is the only way we break down silos, and I think that’s one of the great benefits of this exercise.”|
“Giving them the opportunity to be a part of (this process) is quite meaningful in itself. Hopefully it gives people a sense that this is something that they’re contributing to, that they are a part of.”
|Promotes knowledge exchange||The process enabled participants to share their perspectives on organisational performance, including examples of good practices and current challenges across their disciplines/work areas.||“It’s good to have other people’s perspectives because senior managers have a broader view of what’s happening, but they might not actually have the knowledge of what happens in practice.”|
|Promotes awareness and understanding of health literacy||The process increased staff awareness and understanding of health literacy and health literacy responsiveness, including strategies they could implement to address/improve them.||“People appreciated being able to come together and talk about health literacy and get a better understanding of what it means.”|
|Promotes reflection and learning opportunity||The process encouraged participants to reflect on their own practice and the practices of their organisation. They also reported that the process provided them with an opportunity to learn about health literacy responsiveness and to learn more about their organisation.||“I think that absolutely will make it easier for staff to realise it’s not just about words, it’s about how I behave, the spaces we have, the systems and processes (in place).”|
“It does raise your curiosity though, reading the different dimensions. For me I thought if I don’t know about it should I be finding out about it.”
|Terminology||The term health literacy is not used by some organisations, as it is not well understood by all staff and/or they perceive it to be jargon and abstract.||“One of the limitations of the words health literacy is that it very much points to literacy, to words and language, and I think that is its biggest handicap as a notion, as a concept.”|
“We don’t use the terminology health literacy, so everyone’s got a slightly different take on it or they take it very literally as literacy—reading and writing skills, rather than thinking broader than that.”
|Length of the tool||The Org-HLR tool was too long and repetitive in some assessment dimensions||“I did find it a bit drawn out.”|
“I’d like to see it simplified… from a usability point of view I tend to think shorter is better.”
|Global rating system and criteria of self-rating tool||The global rating system was perceived as confusing, complicated, and as not allowing for an accurate assessment of each performance indicator. |
The rating criteria were also perceived as overly complicated and as not applying well against some assessment areas.
|“I think it is easier for each statement to have a rating rather than just the overall (sub-dimension) rating.”|
“Make that a bit clearer around how to rate.”
|Criteria of priority setting tool||The rating criteria for the priority-setting tool were perceived as complicated.||“We did also talk about the priority setting tool rating system being two pronged—importance versus urgency.”|
|Duplication with other self-assessment tools and processes||Participants reported that this Org-HLR tool and process overlapped with other self-assessment tools and quality improvement processes (e.g., cultural competence and accreditation). This is may lead to the duplication of effort and action plans.||“Another issue is the overlap with existing accreditation (processes) and existing evaluation tools, and the fact that we’ve already been through this process and evaluated a whole stack of things.”|
“The risk there is, if we have action plans coming out of a number of different self-assessments that are looking at the same thing, we end up having different people approaching the same problem in different ways.”
|Time required||Some participants perceived the time required to complete the assessment to be prohibitive. As a result, some staff would not be able to participate (i.e., clinical staff) and it would be difficult to ensure consistent representation throughout the assessment process.||“At the beginning (of planning the assessment) I thought the time commitment was going to be a really hard ask.”|
|Staff roles and representation||Some participants perceived that parts of the tool were not relevant to their role or work area; therefore, they could not make an informed judgement about organisational performance in that area.||“It assumes, and this is why it’s important to have representation from across the organisation, that we know as individuals what’s going on (in other parts of the organisation) and we just don’t.”|
“I think for it to work here, chunking (breaking sections down) by who was responsible and their work group, rather than health literacy titles might make it easier to get it done.”
|Assessment Dimensions||Sub-Dimensions||Original Indicators||Revised Indicators|
|1. Supportive leadership and culture||1.1. Allocates financial resources||3||3|
|1.2. Demonstrates leadership and commitment||4||5|
|1.3. Makes health literacy an organisational priority||4||3|
|1.4. Promotes equity and diversity||4||4|
|1.5. Fosters a person-centred philosophy||3||3|
|2. Supportive systems, processes and policies||2.1. Undertakes data collection and community needs identification||9||7|
|2.2. Undertakes performance monitoring and evaluation||5||5|
|2.3. Undertakes service planning and quality improvement||7||5|
|2.4. Ensures effective communication systems and processes are in place||8||8|
|2.5. Ensures written internal policies and procedures are in place||6||6|
|3. Supporting access to services and programs||3.1. Provides and appropriate service environment||3||3|
|3.2. Supports initial entry and ongoing access to services and programs||8||8|
|3.3. Provides outreach services||3||3|
|4. Community engagement and partnerships||4.1. Undertakes community consultation and enables consumer participation||8||6|
|4.2. Works in partnership with other organisations||6||5|
|5. Communication practices and standards||5.1. Applies communication principles and standards||10||8|
|5.2. Provides health information effectively||6||5|
|5.3. Uses media and technology effectively||5||4|
|5.4. Provides health education programs||3||3|
|6. Recruiting, supporting and developing the workforce||6.1. Recruits an appropriate workforce||4||3|
|6.2. Provides supportive working environments, practice tools and resources||3 + 8 *||5|
|6.3. Provides ongoing professional development||11||8|
|Original Rating||Original Rating Description||Revised Rating||Revised Rating Description|
|1||There is no evidence that this occurs, and there is no support/commitment internally for undertaking work in this area.||0||Not at all|
|2||There is no evidence that this occurs, but the organisation has made a commitment to it and planning has commenced.||1||Minimally|
|3||There is evidence that this occurs sporadically across some parts of the organisation, but it is undertaken inconsistently and significant improvements are required.||2||Partially|
|4||There is evidence that this occurs consistently across most parts of the organisation, but improvements are required to embed it into organisational systems and processes.||3||Substantially|
|5||This is routine practice that is consistently undertaken across all areas of the organisation and has been embedded into organisational systems and processes.||4||Fully|
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).