Sustainability Assessment and Benchmarking in Higher Education Institutions—A Critical Reflection

Abstract

Higher Education Institutions (HEIs) play a crucial role in implementing practices for Education for Sustainable Development (ESD). This implementation should be done in different dimensions according to a holistic and whole-school approach. Different tools have been adapted and developed to assess this integrated approach. The aim of this research is to critically reflect the existing tools to assess and benchmark ESD implementation and to discuss their applicability in two case studies. Two public Universities in Southern Europe, with headquarters in the capitals of Portugal and Spain were selected to assess and compare the integration of ESD according to a whole-school approach—Universidade Aberta in Portugal and Universidad Autónoma de Madrid in Spain. After a critical analysis of the existing tools based on literature review and a list of criteria classified by experts, two tools were selected to be applied in the case studies. The online Sustainability Tracking, Assessment & Rating System Reporting Tool was used in Universidade Aberta and Green Metrics tool was used in Universidad Autónoma de Madrid. The tools were complemented with focus group with key-actors in both universities. The results obtained allowed to identify the need to define a common objective of the assessment tools and limitations they still have. The tools need improvements on their development namely to integrate the external impact of Higher Education Institutions on sustainability, to integrate participatory processes and to assess non-traditional aspects of sustainability. This research hopes to contribute to the continuous research about the usefulness of these assessment and benchmarking tools as drivers to HEIs improve their sustainability performance and their role as agents of changes.

1. Introduction

Higher Education Institutions (HEIs) have a critical responsibility in education for sustainable development (ESD) due to “the main component for raising awareness of SD among the population” [1]. In other words, ESD provides the knowledge and skills for students to start to create SD initiatives. Furthermore, HEIs should “lead by example” [2].

Sustainable development as a development model integrates environmental, social and economic considerations [3] and so HEIs must assume a holistic focus in all activities. Indeed, HEI with or without governmental policies and recommendations are moving towards holistic and systemic approaches when addressing Education for Sustainable Development. The Sustainable Development Goals (SDG), recently adopted by United Nations in 2015 [4] and in particular, SDG 4, are an additional driver for the implementation of sustainability at HEI in an integrative way.

According to the United Nations guidelines followed by several researchers the integrative approach to implement sustainability in HEIs includes six major ESD dimensions to allow a whole-school approach—(i) Facilities or Operations; (ii) Teaching and Curriculum; (iii) Organizational Management; (iv) External Community; (v) Research; (vi) Assessment and Communication (based on [5,6]).

Based on early work [7,8], some authors stress that the integration of sustainability in HEIs can be done at various levels, from national to the institution level [9]. The institutions that integrate sustainability according to the whole-school approach (in terms of dimensions and involvement of the whole HEIs community) can achieve the sustainability maturity curve, thus enabling them to be agents of change and transformation. In this context, Kapitulcinová [9] introduced the model from a “business-as-usual university” to a “sustainable university” where sustainability has been fully integrated under three degrees, initiation/awakening, implementation/pioneering and institutionalization/transformation, including in this last stage a consolidation of changes. Thus, the HEIs are under mounting pressure to partner with societal stakeholders and organizations to collaboratively create and implement sustainability-advancing knowledge, tools and societal transformations [10]. So, in recent years, an increasing number of institutions have begun the adjustment and restructuring of education, research, campus operations and community outreach towards sustainability [8,11,12]. However, deep and full integration of sustainable development at all of dimensions and community of HEIs is still lacking [13,14].

Different tools have been developed to assess and benchmark ESD implementation at HEIs but how well are being performed in case studies and how well they evaluate sustainability and its impact is still an open question. This study attempts to provide a holistic sustainability maturation path reflecting essential dimensions that HEIs need to approach and assess as whole-school integration towards a potential sustainability management change. The aim of this research is, in the first part, to critically analyze the existing tools to assess and benchmark ESD implementation in HEIs. In the second part of the article, the aim is to assess the integration of ESD according to a whole-school approach through the use of those tools in the case studies and discuss their applicability. Two of the analyzed tools were selected and applied in two HEIs in Portugal and Spain. The assessment was complemented with stakeholder’s engagement. European HEIs have been ahead in the implementation of sustainability at the different dimensions [13,15]. In particular, in southern Europe neighbor countries like Portugal and Spain, despite the lack of national policies, HEIs are working towards ESD implementation through the development of plans and actions in sectorial areas. The two case studies are public Universities (one smaller and in distance learning regime and others in traditional learning and bigger) engaged in the last years in integrating sustainability in different dimensions. They have their headquarters in the capitals of Portugal and Spain and are classified in levels of excellence in their teaching and learning methods: Universidade Aberta, Portugal and Universidad Autónoma de Madrid in Spain.

This article starts after this Introduction with a literature review about the tools to assess sustainability in HEI (Section 2). Section 3 presents the methods for the tool’s assessment and case studies evaluation. Section 4 describes the case studies, Universidade Aberta and Universidad Autónoma de Madrid. Section 5 discuss the results and the final section is dedicated to final conclusions and contribution to new knowledge and research implications.

2. Sustainability Assessment in Higher Education

Given the rapid growth of ESD initiatives at HEIs, the measures, assessment and reports of HEIs progress toward SDG has become increasingly important [16]. The assessment of sustainability in HEIs is one of the most important dimensions of EDS implementation in HEIs and could be conducted base on specific tools, that allows to assess whether all possible dimensions to the implementation of sustainability are being implemented and whether they are doing so holistically [9]. Also, when well-developed these tools can be used as benchmarking practices, comparing HEIs processes and performance metrics. Nevertheless, these tools must be able to uniformly evaluate the implementation of sustainability in HEIs without necessarily placing them in a ranking or competition. So, the tools must identify the important themes, be measurable and comparable, go beyond eco-efficiency, measure progress and motivations and be understandable to a broad set of key actors [17]. According to other authors [18], these tools can be based on indicators and conceptual models that support sustainability decisions, as well as facilitating communication efficiently and for a wide audience, knowing how to respond to complex processes capable of assessing the transformation for sustainability.

Much work has been done on the development of tools to assess sustainability specifically in HEIs, showing the importance of the theme. Several articles have reviewed these tools from different perspectives (e.g., [7,9,14,16,17,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33]). However, it is observed that in these various reviews, the systematization is not homogeneous, noting a lack of common designations and objectives and including tools that do not have as main objective the assessment of the implementation of ESD or are not per se an assessment tool. So, for example, the reviewed tools:

(i)

assess only one of the dimensions of implementing sustainability in HEIs such as Campus operations (e.g., Campus Sustainability Assessment Framework-CSAF, [20]; National Wildlife Federation’s State of the Campus, [34]) or Curricula (e.g., Sustainability Tool For Assessing Universities’ Curricula Holistically, STAUNCH, [11]);

(ii)

evaluate only one pillar of sustainability, namely Environmental (e.g., Campus Ecology, [35], Environmental Performance Survey, [34]);

(iii)

only serve as manuals or supportive conceptual models (e.g., Greening Campuses, [36]);

(iv)

assess only the level of literacy and knowledge of population sustainability (e.g., SULITEST, [37]);

(v)

adapt by an Institution but based on other existing metrics (e.g., UNI-Metrics—Value Metrics and Policies for a Sustainable University Campus, [31]);

(vi)

specific to a type of HEI (e.g., Business School Impact System BSIS, see Reference [14]);

(vii)

serve only as guidelines for supporting communication of performance for sustainability but are not themselves an assessment tool. This is the case of the guidelines developed by the International Campus Sustainability Network and Global University Leader Forum [38], two international networks that suggest the organization of sustainability reports for HEIs, based on GRI indicators and the Sustainability Tracking tool, Assessment & Rating System—STARS [38]. This is also the case of the work developed by Nixon [19] who developed other guidelines for the implementation of sustainability in HEIs (under the CSARP project “Campus Sustainability Assessment Review Project” developed at Western Michigan University), in its different dimensions and practices, based on a literature review of existing tools, proposing no new tool.

Thus, for this study a systematic review of the tool for sustainability implementation assessment in HEIs was conducted, based on the following conditions:

(a)

developed specifically for assessing the performance of sustainability implementation in HEIs;

(b)

covering at least two of the various dimensions of sustainability implementation in HEIs ((i) facilities or campus operations; (ii) teaching and curriculum; (iii) organizational management; (iv) external community; (v) research; (vi) assessment and communication;

(c)

covering at least two of the sustainability pillars (environmental, social. and economic), to guarantee that the tools in some way are based on a holistic and whole-school approach and since most are based on only two of the pillars.

Based on these criteria 27 tools were searched on google scholar using the key-words: “Sustainability assessment” and “Higher Education,” from October 2018 and March 2019. Each tool was then characterized (see

Table 1. Sustainability Assessment Tools in Higher Education (HE) and brief description.

Tool and Source	Brief Description and Region Where Applicable
AISHE 2.0 Assessment Instrument for Sustainability in Higher Education. (Latest version) AISHE 2.0 Assessment Instrument for Sustainability in Higher Education [39,40]	Based on narrative and indicators: 30 indicators, 5 dimensions (Operations, Education, Research, Society, Identity), less emphasis on the environmental component (just 1 indicator); incorporating the Deming cycle approach; the intended target is the university system; with a wide world application across the universities; the application domain adapts according to the university structure as an entire university, campus, buildings or research institute; AISHE first version was developed in 2000 and 2001 only focused in the educational role of universities, however, AISHE 2.0 has a wider scope in terms of the research, operations and relation with the society; Developed by a researcher in Europe; Tool not available online, only the manual. The Netherlands / international
AMAS—Adaptable Model for Assessing, Sustainability in Higher Education [29]	Based on 3 domains (Institutional commitment, Leadership, Advanced sustainability); with 4 levels of application hierarchy that lead to the use of standardized indicators (based on other existing tools), with different weights and key actors’ participation, allowing to be adapted by each institution but comparable in the same country; With an expert consultation system; Developed by a researcher in Chile; Tool not available online. Chile
ASSC—Assessment System for Sustainable Campus [41]	Based on a questionnaire and reported on graphical form; 26 indicators; 4 dimensions (Management, Education and Research, Environment, Local Community); based on other tools (STARS, Uni-metrics, GM, BIQ - AUA). Rating system with 4 levels, allowing to obtain a certification: platinum, gold, silver and bronze; It provides information on the strengths and weaknesses of implementing sustainability in HEIs and helps them decide future strategies; includes specificities of the country where it was developed (e.g., natural disasters); Developed by Hokkaido University in 2013, within CAS-NET JAPAN (Campus Sustainability Network in Japan) but used in other universities in Japan; Tool available online, https://www.osc.hokudai.ac.jp/en/action/assc (see Reference [41]). Japan
AUSP—Evaluación de las políticas universitarias de sostenibilidad como facilitadoras para el desarrollo de los campus de excelencia internacional [29,42,43]	Based on 4 areas (Organization, Teaching, Research, Environmental management); less emphasis on the social component; 176 indicators; data collection by questionnaire and interviews (self-assessment) and reviewed by an external organization; the purpose is to improve performance and policies in terms of social responsibility, environment (including public procurement), economy and implementation of the Sustainable Development Goals; graphical representation of indicators; with several updates (last in 2018); Developed by the De La Crue Sectoral Commission for Environmental Quality, Sustainable Development and Risk Prevention of CRUE - Conference of Rectors of Spanish Universities specifically for HEIs in Spain and tested in several Spanish Universities; Questionnaire available online, https://goo.gl/forms/Fol9qwVvYF2juTbC2 (see Reference [42]). Spain
BIQ-AUA—Alternative University Appraisal [29,44,45]	Based on self-assessment with questions to calculate indicators for benchmark (BIQ) and Dialogue. BIQ has a special focus on governance, education, research and communication. It is divided into 4 categories, 15 sub categories (with equal weight) and includes 30 indicators and 50 questions; does not include environmental management and social responsibility indicators; the highest rating is 100, thus allowing comparison; dialogue is the component that enables institutions to share their concerns, best practices and learning about ESD. Applied to 28 universities in Asia and the Pacific; Developed by ProSPER (Promotion of Sustainability in Postgraduate Education and Research Network), an academic alliance between Asia and the Pacific. It is composed of three components based on the evaluation according to the United Nations Decade for Sustainable Development; Tool not available online Ásian-Pacific
CITE AMB—Red de Ciencia, Tecnologia, Innovacion y Educación Ambienal em Iberoamerica [46]	Based on a questionnaire with 27 questions (Yes/No answers); 4 areas (Management, Research, Education, Community), without focusing on the environmental component of campus infrastructure and social component; Developed by the Network of Science, Technology, Innovation and Environmental Education in Iberoamerica; Not available online (existence of a link in the report for “google docs” but is no longer available), developed in 2014 but with no updates available. Colombia
DUK—German Commission for UNESCO AG HS (2011) fidé [14,28]	Based on indicators in 4 areas (Operations, Research, Education, Community); With a strong focus on the institutional part, the tool operates as moderator in the whole-school approach. It contains 10 action fields and each one offers 5 stages of implementation, allowing the HEIs option; Developed by the German Commission for UNESCO in 2011 for the German context; Tool not available online; only a report about the tool in German is available. Germany
ESDGC—Education for Sustainable Development and Global Citizenship [9,47]	Based on a ranking system with 5 areas (Commitment and leadership, Teaching and learning, Institutional Management, Partnerships, Research and Monitoring) and the categorization of 4 levels; adaptation of a maturity model and training usually applied to companies and the industrial sector; results with a semaphore system; Developed specifically for HEIs in Wales, UK and outlined by the Government of Wales to enable an assessment of implementation of ESD in Universities; Tool not available online. Wales / United Kingdom
GASU—Graphical Assessment of Sustainability in Universities tool [7]	Based on GRI report with adaptations to HEIs; applied in many universities, 8 dimensions (Direct economic impact, Environmental, Labor practices and decent work, Human rights, Society, Product liability, Curricula, Research), up to 126 indicators; graphical presentation of results; Developed by a researcher in Europe and marketed through a company; Tool not available online for free, only with a fee payment. United Kingdom / international
GC—Good Company’s Sustainable Pathways Toolkit [24,48]	Based on 20 key performance indicators plus an additional 10 indicators; more emphasis on campus operations; without focusing 2 categories of sustainability implementation in HEIs, namely, Research and Stakeholder involvement; the purpose is to aid decision support / management and benchmarking; Developed by a US company (Good company) and without support to key experts / actors; Tool not available online, neither report nor update. USA / international
GM—Green Metrics University Ranking [49,50]	Based on 6 domains (Scenario and infrastructure, Energy and climate change, Waste, Water, Transport, Education & Research); 33 indicators, two focus on the environment, no community involvement or other social components; ranking point system allowing benchmarking and comparison; with a wide world application across the universities; Tool available online, http://greenmetric.ui.ac.id (see Reference [50]. Indonesia/international
GMID—Graz Model for Integrative Development [51]	Based on narrative and domains: 5 domains (Leadership, Social Networks, Participation, Education and Learning, Research); applicable but not specific to HEIs; applied to the RCE-an international network of formal, non-formal and informal education organizations -, mobilized to provide ESD to the local and regional community at 3 levels; Developed by a researcher in Europe; Tool not available online. Austria / international
GP—Green Plan and the Label DD&RS ou Plan Vert [52]	Based on 5 domains (Strategy governance, Education and training research, Environmental management, Social policy, Regional presence); 44 indicators; can be audited and certified by internal and external stakeholders concerning the ISO 26000; purpose of assisting in the elaboration of sustainability plans/policies; Developed by Conférence des Grandes Ecoles, Conference of University Presidents, French Government and Non-Governmental Organizations within the Grenelle Environment Roundtable; Tool not available online at the present
HE 21—Higher Education 21’s sustainability Indicators or HEPS Higher Education Partnership for Sustainability [24,53]	Based on indicators (12 key indicators and 8 strategic management indicators); focusing mainly on parameters of organizational management change; less emphasis on social indicators and does not encompass in a balanced way all the dimensions of ESD in HEIs (more emphasis on governance); difficult to benchmarking; latest version and network activity in 2003; Developed for 18 universities in the UK that have partnered to support English universities and their monitoring in the implementation of sustainability-HEPS Higher Education Partnership for Sustainability; Tool not available online. United Kingdom
PSIR—Penn State Indicator Report [24,54]	Based on 33 indicators, covering the environmental dimensions of the campus, transport, decision support, research and community; results of each indicator reported in 4 levels of implementation and with proposals for improvement; less emphasis on social indicators and without teaching and curriculum components; last version available in 2000; Developed by the Penn State Green Destiny Council to be applied at US universities, in the State of Pennsylvania and to be communicated to the general public how sustainability is being implemented; Tool not available online, only on the report, http://www.willamette.edu/~nboyce/assessment/PennState.pdf (see Reference [54]). USA, Pennsylvania State
P&P—People & Planet University League [55]	Based on 13 indicators (not divided into dimensions), greater focus on environmental operations and less on community; graphical presentation of results; in operation for several years allowing the annual comparison and an annual ranking; data collection is carried out in the universities’ webpages and the UK Higher Education Statistics Agency; Developed by a network of UK students - People & Planet for universities in the UK and tested at various UK universities; Tool available online. https://peopleandplanet.org/university-league (see Reference [55]). United Kingdom
SAQ—Sustainability Assessment Questionnaire [56]	Based on narrative and indicators: 35 indicators, 8 dimensions (Curriculum, Research and scholarship, Operations, Faculty and staff, Extension and services, Student opportunities, Administration, Mission and planning); with greater emphasis on campus operations; presented through a questionnaire addressed to various internal stakeholders; Developed by the secretariat of the signatories of the Tailloires Declaration - Association of University Leaders for a Sustainable Future; Tool available online, http://ulsf.org/sustainability-assessment-questionnaire/ (see Reference [56]). International
SRC—Sustainability Report Card [57]	Based on narrative and indicators: 52 indicators, 5 dimensions (Campus operations, Meal service, Donation investment, Transportation, Involvement of key stakeholders); more focus on energy saving and less emphasis on education; presented through a questionnaire with a final grade from A to D; suspended in 2012; Developed by a North American Non-Governmental Institution - Sustainable Endowments Institute; Tool not available online. USA / Canada
STARS—Sustainability Tracking, Assessment & Rating System [58]	Based on narrative and indicators: 74 indicators, 5 dimensions (Academic, Involvement of key actors, Operations, Planning and Administration, Innovation and leadership); 5 levels of final classification, allowing the ranking (reporter, bronze, silver, gold, platinum); one of the most used tools internationally; updated every year; developed by a North American Non-Governmental Institution-Association for the Advancement of Sustainability in Higher Education and initially developed for HEIs in the US and Canada but applicable to any region; tool available online, https://reports.aashe.org/accounts/login/?next=/tool/ (see Reference [58]). USA / international
SUM—Sustainable University Model [8]	Based on narrative and indicators: 23 indicators, 4 dimensions (Education, Research, Dissemination and partnership, Campus sustainability); divided in 4 phases (Vision development, Mission, Sustainable committee, Audit of sustainability strategies) incorporating the Deming cycle approach; tested at various world universities; without updates; Developed by a researcher in Mexico; Tool not available online. Mexico / international
SLS—Sustainability Leadership Scorecard [59]	Based on performance indicators; 4 domains (leadership and governance, learning, teaching and research, operations); self-assessment developed specifically for colleges and universities to improve social responsibility and environmental performance through a whole-school approach; final scores with a range from 0–4 no weights in the indicators and final result in a dashboard index; adapted from the Green Scorecard and linked to Sustainable Development Objectives standards; Developed by a Non-Governmental Association of the United Kingdom and Ireland—The Alliance for Sustainability Leadership in Education; Tool available online for free to United Kingdom and Ireland, https://www.sustainabilityleadershipscorecard.org.uk/#!/login (see Reference [59]). United Kingdom / Ireland
SustainTool—Program Sustainable Assessment Tool [60]	Based on indicators, focused on areas / programs or at the institution level; 8 dimensions (Environmental support, Funding stability, Partnership, Organizational capacity, Program, Evaluation, Program adaptation, Communications, Strategic planning) with a low weight in the environmental component; presented in a 40 multiple-choice questions in self-assessment questionnaire, with answers being given individually or in a group; allows the communication, review and development of an action plan; available for several years with updates; Developed by a north American university—Washington University for any university, particularly in the North American context but especially directed to the health area; Tool available online, https://sustaintool.org/assess/ (see Reference [60]). USA / international
THE—Times Higher Education Impact University Ranking [61]	Based on the evaluation of the implementation of the Sustainable Development Objectives (ODS) in HEIs: 11 ODSs: 3,4,5,8,9,10,11,12,13,16, 17; each ODS has a small number of indicators associated with it; equal weight is given to each ODS; first version available for 7 ODS but still in development (1st version April 2019); Developed by the Times; Tool available online, requesting by email, https://www.timeshighereducation.com/how-participate-times-higher-education-rankings (see Reference [61]). International
TUR—Three Dimensional University Ranking [62]	Based on indicators: 15 indicators, weighted based on a participatory process and Analytical Hierarchical Process (AHP), 3 dimensions (Research, Education, Environment); less holistic approach; graphical presentation of results; allows ranking based on rankings of world universities, simplified sustainability only in 5 indicators; tested in the best universities but without updates; Developed by researchers in Europe; Tool not available online. International
UEMS—University Environmental Management System [21]	Based on EMAS / ISO14001 with a social responsibility component and indicators: 27 indicators, 3 dimensions (University EMS, Public participation and social responsibility, Teaching and research in sustainability); greater focus on the environment and campus areas; Developed by researchers in Saudi Arabia; Tool not available online. International
USAT—Unit-Based Sustainability Assessment tool [63]	Based on indicators: 75 indicators, 4 domains (Teaching, Research and community services, Operation and management, Student involvement, Written policy and statement); score of 1 to 4 indicators; adapted from SAQ, AISHE and GASU; can be used in the department, college or HE unit; without updates; Developed by the United Nations Environment Program (UNEP) for the African context; Tool not available online but questionnaire available online on report, https://www.ru.ac.za/elrc/publicationsandresources/unit-basedsustainabilityassessmenttoolusattool/ (see Reference [63]). Africa
uD-SiM model—Uncertainty-based quantitative assessment of sustainability for HEIs [9,64]	Based on indicators and the Models of Pressure, Exposure, Effects, Action (DPSEEA) and a multicriteria decision process (applying Fuzzy logic). Aggregate score in a final index that integrates the non-linear effects of the indicators, with different weights and normalized indicators. Indicators based on the GASU model; 4 areas (Environmental, Economic, Social and Education); applied to Canadian Universities but its implementation is international; calculation method is complex; Developed by researchers; Tool not available online. Canada/International

).

According to several authors, the overall implementation of these tools is still low and its development is still at an early stage [14,28,32].

The various tools for assessing the sustainability of HEIs are mostly based on indicators, using graphs or final rankings to communicate the results. Indicator-based tools have the advantage of being potentially more transparent, consistent and comparable, thus useful for monitoring and decision support [14,18], although support for decision making is not yet fully demonstrated [65].

Another common characteristic of these tools listed in Table 1 is the fact that they are filled out by self-assessment, requiring only a leader or researcher to complete them. In order to create a sustainable university, it is important not only to use assessment tools for a real application as well as integrate on the process different agents of Higher Education Institutions [32]. Stakeholder participatory approaches can be seen as a requirement, as well as a benefit towards the integration of SD into the university culture [66]. Furthermore, active stakeholder participation is essential to grow the model’s level of complexity, promote model ownership and use it in the organizational strategic planning process in a higher education organization [67]. Stakeholder engagement is also crucial to achieve the visions and goals for a Green university [68] and current SDG. The second draft of the People’s Sustainability Treaty on Higher Education [69] divided higher education stakeholders into three broad categories: (1) those engaged in the activities of higher education institutions: executive, academic managers, educators, researchers and students. (2) those engaged in the higher education system: administrative officers, ministries, assessment bodies, international organizations. (3) those forming part of the communities, which the HEI system serves: local communities, professional bodies, companies, among others. Hence, a socially responsible HEI considers stakeholder behavior and perception to better understand their expectations and priorities and use these to define the strategy and goals, to monitor the objectives in view of promoting activities and accountability and to enhance a community-university engagement. At the end it contributes to change management and to a more mature sustainable university [9]. Thus, sustainability reflects a condition based on the relationship between stakeholders and HEI [70].

Based on the characteristics of these tools (see Table 1), the 27 tools were critically analyzed to evaluate their real assessment of the sustainability implementation and integration in HEIs (see the sections of the methods and results).

3. Cases Studies

The Universidade Aberta (UAb) was founded in 1988 with the distinctive feature of being the only Portuguese public distance education university, a distinction that still remains today. Filling the Universidade Aberta vision and mission of being a global university, the Universidade Aberta offers undergraduate and graduate higher education courses and Lifelong Learning courses, which are especially dedicated to the whole Portuguese speaking country community. The Universidade Aberta campus corresponds to four facilities, namely, its headquarters in Lisbon and two other support buildings in Coimbra and Porto (regional offices). The Universidade Aberta also has other facilities in Lisbon and Local Learning Centers spread throughout the country but since they are not owned but rented, they were not considered in the sustainability evaluation. In 2018 Universidade Aberta community comprise the rounded numbers of 6000, which 5000 were full-time students and exclusively engaged in distance education and 340 employees, of whom 150 belongs to the academic and research staff and 190 to the administrative staff. It is structured in 5 academic departments: Science and Technology, Social and Management Sciences, Distance Learning and Humanities, Lifelong Learning, with an educational offer of 10 graduate Ph.D. programs; 22 graduated master programs, 11 under graduated programs and 9 post-graduated programs. UAb has 2 research institutes and 5 more research institutes with branches in this university.

The Universidade Aberta has focused on the quality of its service and has been distinguished by several national and international entities: (a) EFQUEL Award—European Foundation for Quality in E-learning in 2010; (b) the UNIQUe—The Quality Label for the use of ICT in Higher Education (Universities and Institutes) in 2010; (c) the 1st Level of Excellence Committed to Excellence (c2e2) of the European Foundation for Quality Management (EFQM) in 2011; (d) 2nd Level of Excellence Recognized for Excellence (R4E) of the European Foundation for Quality Management (EFQM) in 2016; (e) the certification of the International Standard Organization (ISO) 27001 by the Portuguese Association of Certification attesting the security of its platform of e-learning and ISO 9001 quality, in 2017. UAb has been applied sustainability in different ways, namely: (i) curricula (through a e-learning three cycle degrees system from undergraduate, to master and Ph.D. aiming to actively promote education for sustainable development, along with an increase in transdisciplinary across subjects and also through non formal courses about Climate Change, Education for Sustainability and Environment awareness), (ii) application of a quality management policy with a Recognition of Excellence, (iii) a specific inclusion program for students with disabilities. UAb formal compromise to Sustainability was achieved by being an institutional member of the Association for the Advancement of Sustainability in Higher Education (AASHE) and a signatory member of the Letter of Commitment for Sustainable Campus in Portugal, both in 2019. UAb has no formal office for Sustainability due to its small size but sustainability issues are informally addressed within the Quality Office.

The Universidad Autónoma de Madrid (UAM) is a Spanish public university established in 1968. In 2018, 26.733 students of all levels were enrolled at the UAM. This university has a teaching staff and/or researcher around 3.141 and an administrative and services staff around 1.053. The university is organized into eight schools: Sciences (biology, mathematics, physics and chemistry), Economics and Business, Law, Computer Science and Engineering, Arts (philosophy, history, philology, translation and interpretation), Education and Psychology, offering a wide range of programs in different scientific and technical fields and in the Humanities. UAM has 11 research institutes and these are located on campus, as well as the Madrid Science Park, with growing university-business collaboration (contracts, internships and sponsored chairs). In 2009, UAM was declared International Excellence Campus, at the same time with the Spanish Research Council (CSIC). Nowadays, UAM has been placed among the top universities for its levels of excellence in national and international rankings. In the QS World University Ranking 2019, the UAM has managed to locate itself in the first position in Spain and in the 159th place in the world.

Most of the faculties and specialized institutes are on the Cantoblanco campus, 15 km North of Madrid. The Faculty of Medicine is on another campus (near La Paz Hospital). These two campuses (Cantoblanco and La Paz) were considered in the sustainability evaluation. In the field of sustainability, this university, from the Rio de Janeiro Summit (1992) (where a global action plan for Sustainable Development was approved: Agenda 21), formalized its commitment to Agenda 21 through the ECOCAMPUS project. In 1997, the Ecocampus office was created, which has a special involvement in the maintenance of the campus as a sustainable territory and at the same time promotes activities related generally to sustainability. Social commitment and sustainability continue to be part of the frame of reference of the different lines of action for the entire institution in the 2025 strategy. UAM has also an SDG Lab that is a multi-stakeholder initiative that contributes to the implementation of the Sustainable Development Goals at the UAM.

4. Materials and Methods

4.1. Tools Assessment

The critical analysis of the tools listed in Table 1 was based on earlier reviews and according to the following criteria (based on the research of [9,18,24,28,65]: (i) comprehensibility; (ii) comparability; (iii) availability of baseline data; (iv) assessment of progress over time; (v) comprehensiveness and integration of sustainability dimensions in HEIs; (vi) usefulness for decision-making and communication; (vii) level of participation of the public or key actors and (viii) tool accessibility on the internet. The tools were then classified from 1 to 3 for each of the criteria (1. Low, 2. Medium and 3. High), based on documentary analysis of the tools and expert knowledge (according to the methodology referred in Reference [71]).

The classification of each tool was conducted independently by four judges (authors of this paper) using the defined criteria. The procedure proposed in Reference [72] has been followed in order to determine the level of agreement between the judges (values between 0 and 1 for each agreement between two judges):

Index of Agreement = (C1,2 + C1,3 + C1,4 + C2,3 + C2,4 + C3,4)/6

For data analysis, the mode and relative frequencies were calculated for each criterion. An average of the Index of the agreement was calculated for each criteria and tool. Two of the listed tools were not classified since their information was not available in English (DUK was only accessible in German and ASSC only accessible in Japanese).

Validity, reliability and generalizability are limitations associated with this type of qualitative approach [71] and were weighed up in the qualitative assessment and discussion of the results and when drawing the conclusions.

4.2. Sustainability Assessment in the Case Studies

The sustainability assessment process at the Universidade Aberta and the Universidad Autónoma de Madrid was carried out using different tools, STARS and GM, respectively, complemented with stakeholder’s engagement (focus groups/workshops) in each University. The focus groups and workshops were used for engagement and awareness of the assessment process and also to find paths of improvement (

Table 2. Operationalization design of the sustainability assessment in the case studies.

University	Data Collection	Method	Participants
Universidade Aberta	Raw data	Documents Analysis: Activity plans and reports, Strategic Plan and programs and courses study guides, databases.	Academics, Administrative services, Campus Operations and Rectorate
	Data standardization	Search in databases /websites	-
	Improvement Proposals	1st Focus group	Vice-rector for quality and 3 researchers (experts in sustainability assessment)
		2nd Focus group	16 (researchers, students, teachers, administrative staff)
Universidad Autónoma de Madrid	Raw data	Documents Analysis: Reports and University web page information	Administration and services staff, teacher
		1st Focus group	4 members of Eco-campus team (Eco-campus Manager, Infrastructure Manager, Environmental Participation Officer, Electric Cars on Campus Officer)
		2nd Focus group	3 Vice-Chancellors (Undergraduate Studies, Sustainability and Campus and Strategy and Planning)
	Improvement Proposals	3rd Focus group	6 students’ leaders from Faculty of Business and Economic Sciences
		UAM SDGs LAB Sustainable Development Goals Workshop	22 Professors and researchers 6 External experts

). The STARS and GM tools were selected since they were well classified according to the main criteria and also in terms of free access on the internet (see Section 5.1 heading the in results section).

For UAb, the implementation of the sustainability assessment was conducted across the year 2018, corresponding to a 3-year assessment analysis (2015 to 2017). STARS 2.1 version tool is based 5 dimensions with different weights each: Academic and Research (20%), Involvement of Key Actors (20%), Campus Operations (35%), Planning and Administration (15%), Innovation and Leadership (2%) and a total 74 indicators divided in each category. The indicators are quantified and filled in a web application with a written justification or document upload. The different indicators and methodological procedures are explained in AASHE [58]. The raw data collection was the first procedure, involving not only a web search as also the requirement of specific informants for the technical information, reach by a face to face interview or information request by email. Possessing the requested data, the STARS assessment was fulfilled, resulting in a diagnostic report on the implementation of sustainability at the Universidade Aberta, where the most and least punctuated indicators were identified, revealing the weaknesses and the strengths to the implementation of sustainability. Since UAb is an institutional member of AASHE, a final assessment of the STARS scores was conducted, through an internal (by the rectorate) and external validation (by the AASHE technical staff) to allow a final awarded rating between bronze, silver or gold label.

With the final STARS report information, two focus groups (one with university experts and others with different stakeholders from the all University—see Table 2) were held in June and July of 2018. The main objective of the participatory moments was to show and discuss the STARS results and development of proposals for improvement for the implementation of sustainability at the Universidade Aberta. In a first part of the focus groups the information provided in the STARS report was presented and improvements were then proposed according to STARS dimensions. The first focus group was conducted with university experts that have been working in UAb sustainability implementation, including the Vice-rector for quality (since the University does not have any green sustainability office and three researchers or teachers). The other focus group was organized with different stakeholders from the all University from the different departments (one professor per department chosen according to a convenience sampling), administrative staff (leader of each administrative service) and two students from environmental and sustainability graduation programs (chosen according to a convenience sampling). Detailed information about all the methods applied in UAb available in Reference [73]. These discussion groups were the first time the university stakeholders participated in discussions related to sustainability.

The process of evaluating the implementation of sustainability in the UAM has begun in 2013 and annually assessed (from 2014 to 2018), through collecting the data according to the six dimensions stipulated by GM with different weights each: Setting and Infrastructure (SI) (15%), Setting and Infrastructure (SI) (15%), Waste (WS) (18%), Water (WR) (10%), Transportation (TR) (18%), Education and Research (ED) (18%). All information about the indicators and methodologies are available at Green Metric [50]. The indicators are quantified and filled in a web application (survey type). Each year in a predefined calendar, Green Metrics validates the submitted universities and publishes the international rating scores.

Representatives of the various groups of the university have participated in the contribution of these data: administration and services staff, teacher and manager (see Table 2). They participated through face-to-face interviews and email, together with collecting the available information, for example in corporate reports and on the web page of the University. The GM assessment along the years allowed to observe the evolution of the university in each of the dimensions and to compare with the scores of participating universities worldwide in GM.

For a review of the sustainability implementation and improvement proposals of the university sustainability implementation, two participatory techniques were carried out, focus group and workshop) (see Table 2). The focus group and workshop were then conducted using semi-structured questions based on the four dimensions of STARS tools to allow better comparison with UAb case study (Academics; Engagement, Operations and Planning & Administration). The collection process was in May 2018. Participants were selected based on their crucial role in the university management system, specifically on the sustainability activities at the campus. In the 1st focus group, 4 members managers of the Eco-campus team participated and in the 2nd workshop the 3 Vice-Chancellors of UAM with responsibilities in the sustainability issues were called to participate. In the 3rd focus group, 6 students’ leaders from the Faculty of Business and Economic Sciences participated in the participative process (according to convenience sampling).

In a fourth participatory phase, SDGs LAB—Sustainable Development Goals Workshop was held. The UAM SDGs LAB was organized by the vice-chancellor of Sustainability and Campus and coordinated with professors and researchers in the field and 6 external experts from the town hall staff, already used to collaborate with the university, in a total number of 22. It took place in the UAM Campus for three days. It was the first participatory workshop to design a roadmap to improve the contribution of the university towards SDGs. This workshop was organized in three stages: (I) Inspiration: review of previous experiences and new inspiring ideas, (II) Observation: potential contribution to the campus in situ and proposals, (III) Implementation: discussion, analysis and outcome.

The results of the discussion groups in UAb and UAM were transcribed and then manually analyzed to identify recurring topics in the responses according to Reference [71]. The analyses were validated by the participants.

5. Results and Discussion

5.1. Tools Assessment

In an overall performance, Comprehensibility and Comparability were the criteria higher ranked by the judges (72% and 56% respectively classified as 3, see

Table 3. Mode and relative frequencies for each criterion within 25 HE sustainability assessment tools. Sum and Level of agreement for each tool and criteria.

Tool		Comprehensibility	Comparability	Data Access	Progress over Time	Sustainability Broadness	Support to Decision	Participation	Accessibility	Sum	Index of Agreement
1. AISHE	Mode	3	3	2	3	2	2	3	2	20	70.8%
	R.F. (1)
	R.F. (2)			1	0.5	0.75	0.75	0.5	1
	R.F. (3)	1	1		0.5	0.25	0.25	0.5
2. AMAS	Mode	2	2	2	2	2	2	2	1	15	52.1%
	R.F. (1)			0.25	0.25		0.25		1
	R.F. (2)	0.5	1	0.75	0.5	0.75	0.5	0.75
	R.F. (3)	0.5			0.25	0.25	0.25	0.25
3. AUSP	Mode	3	2	2	1	2	3	1	2	16	54.1%
	R.F. (1)				0.5			0.75	0.25
	R.F. (2)	0.5	1	1	0.25	0.75	0.5	0.25	0.75
	R.F. (3)	0.5			0.25	0.25	0.5
4. BIQ AUA	Mode	2	2	2	2	2	2	2	1	15	60.5%
	R.F. (1)				0.25		0.25		1
	R.F. (2)	0.75	0.75	1	0.5	1	0.5	0.75
	R.F. (3)	0.25	0.25		0.25		0.25	0.25
5. CITE AMB	Mode	3	2	3	1	2	2	2	1	16	48.0%
	R.F. (1)	0.25		0.25	0.75	0.25		0.25	1
	R.F. (2)	0.25	0.75	0.25	0.25	0.75	0.75	0.75
	R.F. (3)	0.5	0.25	0.5			0.25
6. ESDGC	Mode	3	2	2	3	3	3	1	1	18	43.8%
	R.F. (1)	0.25			0.25		0.25	0.5	1
	R.F. (2)	0.25	0.5	1	0.25	0.5	0.25	0.5
	R.F. (3)	0.5	0.5		0.5	0.5	0.5
7. GASU	Mode	3	3	2	3	3	3	2	1	20	56.1%
	R.F. (1)			0.5				0.25	0.75
	R.F. (2)	0.5	0.25	0.5	0.5			0.75	0.25
	R.F. (3)	0.5	0.75		0.5	1	1
8. GC	Mode	2	3	2	2	2	2	1	1	15	45.9%
	R.F. (1)	0.25			0.25	0.25		0.25	1
	R.F. (2)	0.5	0.5	0.75	0.5	0.75	0.75	0.75
	R.F. (3)	0.25	0.5	0.25	0.25		0.25
9. GM	Mode	3	3	3	1	2	3	1	3	19	56.1%
	R.F. (1)				0.5		0.5	0.5
	R.F. (2)		0.25	0.25	0.25	0.75		0.5
	R.F. (3)	1	0.75	0.75	0.25	0.25	0.5		1
10. GMID	Mode	2	1	1	2	2	2	2	1	13	35.4%
	R.F. (1)		0.75	0.5	0.25	0.25			0.75
	R.F. (2)	0.5	0.25	0.5	0.5	0.5	0.75	0.5
	R.F. (3)	0.5			0.25	0.25	0.25	0.5	0.25
11. GP	Mode	3	3	2	3	3	3	2	1	20	41.6%
	R.F. (1)		0.25					0.25	0.5
	R.F. (2)	0.25	0.25	1	0.5	0.5	0.5	0.75	0.25
	R.F. (3)	0.75	0.5		0.5	0.5	0.5		0.25
12. HE21	Mode	1	1	2	2	2	2	2	1	13	52%
	R.F. (1)	0.5	0.75		0.25	0.5	0.5	0.5	1
	R.F. (2)	0.25	0.25	1	0.75	0.5	0.5	0.5
	R.F. (3)	0.25
13. P&P People & Planet	Mode	3	3	3	1	2	2	1	3	18	50.1%
	R.F. (1)	0.25	0.25		0.5		0.25	1
	R.F. (2)	0.25	0.25	0.5	0.25	1	0.5
	R.F. (3)	0.5	0.5	0.5	0.25		0.25		1
14. PSIR	Mode	3	3	3	2	2	3	1	1	18	56.3%
	R.F. (1)			0.25	0.25			1	1
	R.F. (2)	0.25	0.5	0.25	0.5	1	0.5
	R.F. (3)	0.75	0.5	0.5	0.25		0.5
15. SAQ	Mode	3	1	3	2	2	1	2	3	17	54.1%
	R.F. (1)		0.5		0.5		0.75	0.25
	R.F. (2)		0.25		0.5	0.75	0.25	0.75	0.5
	R.F. (3)	1	0.25	1		0.25			0.5
16. SRC	Mode	3	2	2	1	3	3	2	1	17	39.5%
	R.F. (1)		0.25	0.5	0.5			0.25	1
	R.F. (2)	0.5	0.5	0.5	.5	0.25	0.5	0.75
	R.F. (3)	0.5	0.25			0.75	0.5
17. STARS	Mode	3	3	2	3	3	3	2	3	22	70.8%
	R.F. (1)			0.25
	R.F. (2)			0.75	0.25			0.5	0.5
	Freq. Relative (3)	1	1		0.75	1	1	0.5	0.5
18. SUM	Mode	3	3	2	3	3	3	2	1	20	68.8%
	R.F. (1)							0.25	1
	R.F. (2)	0.25	0.25	0.75			0.25	0.75
	R.F. (3)	0.75	0.75	0.25	1	1	0.75
19. SLS	Mode	3	3	2	3	3	3	1	2	20	54.1%
	R.F. (1)							0.5
	R.F. (2)		0.25	0.75	0.25		0.5	0.25	0.5
	R.F. (3)	1	0.75	0.25	0.75	1	0.5	0.25	0.5
20. SustainTool	Mode	3	3	3	1	2	2	2	3	19	64.5%
	R.F. (1)				0.75
	R.F. (2)	0.5	0.5		0.25	1	0.75	0.75
	R.F. (3)	0.5	0.5	1			0.25	0.25	1
21. THE	Mode	3	3	2	2	2	3	1	3	19	49.9%
	R.F. (1)		0.25					0.5
	R.F.(2)	0.25		0.5	0.5	0.75	0.25
	R.F. (3)	0.75	0.75	0.5	0.5	0.25	0.75	0.5	1
22. TUR	Mode	3	1	3	1	2	2	3	1	16	41.8%
	R.F. (1)		0.5		0.5		0.25	0.25	0.75
	R.F. (2)	0.25	0.25	0.25	0.25	1	0.5	0.25	0.25
	R.F. (3)	0.75	0.25	0.75	0.25		0.25	0.5
23. UEMS	Mode	2	2	2	2	2	2	3	1	16	54.1%
	R.F. (1)		0.25						1
	R.F. (2)	0.75	0.75	0.75	0.75	0.5	0.75	0.25
	R.F. (3)	0.25		0.25	0.25	0.5	0.25	0.75
24. USAT	Mode	3	3	3	3	3	3	2	2	22	54.1%
	R.F. (1)								0.25
	R.F. (2)		0.5	0.5	0.25	0.25	0.25	1	0.5
	R.F. (3)	1	0.5	0.5	0.75	0.75	0.75		0.25
25. uD-SiM model	Mode	2	3	2	3	3	2	2	1	18	58.3%
	R.F. (1)	0.25		0.25				0.25	1
	R.F. (2)	0.75	0.5	0.75	0.5		0.75	0.75
	R.F. (3)		0.5		0.5	1	0.25
Index of Agreement		51.3%	47.3%	59.3%	35.3%	70.0%	4.6%	47.3%	77.3%	-	-

Legend: R.F. (1)—Relative Frequency in (1) low; R.F. (2)—Relative Frequency in (2) medium; R.F. (3) —Relative Frequency in (3) high.

). Since the majority of the assessed tools are based on indicators (see Table 1) it is expected that they fulfil these criteria, as well as being a support tool for communication and decision-maker. Nevertheless, is still in doubt of real demonstration if indicators can indeed support decision-maker (only 48% were classified with the higher score in this criterion and also the index of agreement between judges was low—43%). This doubt is in accordance to Ramos [65] in his reflection article. Many of the assessed tools are adaptations to existing ones, often being used to contain geographic specificities (namely from North America, Latina America, Asia and Europe), what make the comparison at the national level easier but more difficult at the international level.

The more complex and complete the tools become, the more complex become to fill them and so the access to the basic data needed (only 32% of the analyzed tools had a maximum data availability rating, such as tools where is only need to answer “yes” or “no” or in the case of multiple-choice closed answers).

The heterogeneity of dimensions of sustainability implementation in HEIs covered by the tools is still remarkable, with the teaching and curriculum dimension and campus operations remaining the best addressed. Also, the environmental pillar is the more addressed, neglecting the social and economic pillars. These results are in accordance with [74], who also found in a study conducted in Spanish Universities that more attention is given to the environmental pillar and that is still necessary to achieve an integrated perspective of sustainability in universities. Those results justify the maximum ranking of only 36% of the tools in the criterion of coverage of sustainability implementation dimension and the same percentage in the criterion of measuring the progress of sustainability implementation in HEIs (see Table 3). Assessment of the progress over time was also the criterion with a higher level of disagreement between the judges (35%). According to Reference [75], variability between judges’ classification is usually expected. Nevertheless, this criterion can be subjective since the assessment along the years can also depend on the HEIs and not only the tool himself. These results are in line with recent studies (e.g., [14,16,32]). Also related to this criterion, the state of development of the tools is revealed by the lack of assessment of impact outside the institution, that means on society in general and long-term impact [76].

The Participation and Accessibility criteria were the ones that have the lowest classification (12% and 24% respectively classified as 3, see Table 3). Concerning the level of agreement between the judges, Accessibility was the criterion that reunited more consensus what is easily explained by the fact that the tool is or not available on the internet to be used (Index of agreement of 77%). Indeed, only a small number of tools have an easily accessible application on the internet to fill in the data and obtain the final result. Participatory approaches have gained increasing attention in the implementation of sustainability in higher education but often remain vague and less addressed in sustainability assessment procedures, as Disterhelft et al. [66] also defended. The policy agenda of ESD calls for innovative and more transformative approaches than reductionist practices, in order to respond better to the need for an institutional learning culture that envisions dialogue and change as stressed by the same authors.

The results of the classification of the tools by the judges (Table 3) highlighted that STARS and USAT have good performance in 6 criteria ranked with maximum classification (both had a sum of 22 points in a maximum of 24). More specifically, STARS ranked medium criteria only in Data access and Participation categories and USAT in Participation and Accessibility criteria. Additionally, from Berzosa et al. [32] experience of “USAT is simple to apply, however, complicates comparations and benchmarking, as there are not any mechanisms to standardize the interviewee and the evaluator.”

Following these tools were GASU, GM, SUM, SLS and GP (a sum of 20 points, see Table 3) with 5 criteria with the highest classification. For example, GASU has being pointed out by other authors (e.g., [7,16,29]) has having the advantages of giving institution visual illustration of sustainability, turning easy to compare and contracts the universities efforts towards sustainability within and among other universities. Also, AISHE was well classified with a sum of 20 points, with 4 criteria ranked with maximum classification (and a high index of agreement between the judges—71%). As discussed by other authors like Berzosa et al. [32] “AISHE score depends on stages of development of sustainability policies and actions within the organization and varies from an activity oriented (1) to a society oriented (5) approach.” In addition, according to Alghamdi et al. [16], AISHE was designed to incorporate only the most significant criteria and not necessarily the whole framework.

Regarding the tools that scored the lowest on the ranking, TUR, HE21 and GMID were the ones with 3 criteria with the lowest classification, where Comparability and Accessibility had poor scores in all three (see Table 3). HE21 [31] is difficult to measure and compare, also indicators may not cover most important issues. ESDGD, P&P and TUR are the tools that had a high divergence in the judgment agreement (44%, 50% and 42% respectively), evidencing a great variability in Understanding, Comparability, Progress Over Time and Support to Decision criteria. TUR and GMID are old tools only available in research papers. P&P, HE21 and ESDGD are tools adapted specifically to be applied in Universities in a geographical region (UK).

Considering the confounding results, THE and GM evidenced a disperse result for the Participation and the Support to Decision criteria, respectively, because the score was divided in the lowest and highest ranking (also with a low index of agreement between judges 50 and 56%, respectively). In GMID, there is a clear outlier in the Accessibility category (since the tool is not available online) and in THE, an outlier for the Comparability category (since it is an international ranking). These facts highlight the importance of the classification being made by several judges and reducing the bias of the results.

Overall, the results confirmed STARS as the tool that collected a high level of agreement among the judges (71%), with the four criteria expressing the same score, Understanding, Comparability, Sustainability Broadness, Support to Decision. Nonetheless, STARS major strength according to Sonetti et al. [31] is its preference for performance over strategy. Earlier studies that reviewed these kinds of tools highlighted that STARS, AISHE and SAQ have a higher incidence on the percentage of indicators for the Governance (in accordance with reference [25]) and Operations, as highlighted by Reference [28] dimensions. However, STARS has the widest coverage across all indicators, capturing a little of all areas compared, for example, with AISHE and SAQ [as also defended by other authors [24,28], SRC (in accordance with Shi and Lai [77]) or with GM [also as highlighted by Lauder et al. [49]. In addition, as authors like Berzosa et al. [32] claimed “the main weaknesses of SAQ are those related with open-ended questions, not establishing a final score so it is difficult to apply it as a tracking tool.”

In a more recent study, STARS appear to be comparable to AISHE and BIQ-AUA, considering the availability of academic and management staff as agents of change in the implementation of sustainability within universities [9]. Analyzing the filing process between STARS and AISHE, it can be seen that the former has a higher percentage of closed questions [24], so it is possible to deduce a greater ease of completeness and greater reliability when comparing the results, thus proving to be more efficient for a regular implementation [14,26,73]. This information is also underlined in a comparative study of STARS with other tools [78]. Authors such as Stough et al. [79] highlight various strengths in STARS namely innovation, understanding, popularity and be based on a holistic and integrative approach to sustainability, while also considering the United Nations Development Goals [79].

GM tool was also overall well classified, besides worst classification in terms of Participation and Progress Over Time criteria but has been also widely used since is free, easy to fill and good as a benchmark between universities worldwide, as stressed by many other authors (e.g., [16,49,74]). As Lauder et al. [49] stressed under scientific analysis, no ranking can be free from at least some limitations, resulting in unavoidable practical considerations, such as the need for the ranking to be at a level of complexity that can appeal to a wide audience. In addition, Sonetti et al. [31] mentioned one of the major weakness of GM is the use of generic quantitative indicators which does not underpin local dimensions as well as lack of the social dimension.

5.2. Application of the Tools in the Case Studies

Universidade Aberta scores for each STARS dimensions are shown in

Table 4. Summary of the results of Assessment and Improvements for Universidade Aberta regarding the STARS dimensions.

Academic
Assessment	Improvement
Comprehensive formal cycle: BSc Environmental science, MSc Environmental Citizenship and Participation, PhD Social Sustainability and Development Most of undergraduate programs with a least a module about Sustainability Non formal courses (Open classes, Massive Open Online Courses, MOOCS, related with Environment) Strategic research line about “Sustainability and Environment” (but with no financial support from University) There is no available place to register the SDG activities of the university.	Institution specific sustainability learning outcomes for all students All undergraduate students should enroll at least a sustainability course (3 ETCS) Enlarge research on sustainability linking students with the labor market, according to transdisciplinary research
Engagement, Planning and Administration
Assessment	Improvement
Staff training about better sustainable practices at work Open courses on Sustainability and Environment PhD thesis aiming at solving local problems related with SDG (action research, transdisciplinary) No formal Institutional sustainability policy/strategy Formal support to Students with disabilities Assessed employee satisfaction Participation of community members in the Institution governance (General Council) Engagement of UAb community in DREAMLAB for Sustainability (Dragon Dreaming technic)	Develop a University policy for sustainability Integrated in the Quality office the Sustainability Practices Inclusion within the program of Welcome of new employees (already existing) a sustainability performance kit Promote awareness within academic community for students to work with the local community problems in the realm of Local Learning Centers; Give more emphasis to the sustainability academic offer of the University
Operations and Innovation
Assessment	Improvement
No formal campus; e-learning regime (low ecological footprint) LED lighting has reduced consumption Use of local food/resources/services for events Videoconference as prime communication service to all events Decentralized Local Learning Centers in areas of low population density/close contact with society	Disseminate/Monitor the Ecological footprint – CO2 equivalent, water, waste (improvement of the GEE inventory) Sustainable procurement practices (also within national policies): e.g., recycle paper, hybrid cars, cleaning material) Separate bins in all facilities Engage all university community in an online collaborative platform for sustainable ideas

and Figure 1 for the period of 2015–2017. The overall scores allowed to awarded the University with a Bronze label on April 5th of 2019 after a three-time external review process by STARS technical staff where some scores rectifications were made after documentation and numbers checked. UAb STARS report is now listed on the website of AASHE. Academic and Research dimensions are the ones where UAb is better classified leaving much room for implementing sustainability in its Planning and Management dimension, unlike in other universities where the focus is more on the Campus Operations [80]. Since UAb is a distance learning institution there is no formal campus for students’ classes and the resources demanding are low (in particular in terms of energy) what justifies that difference. Nevertheless, Campus Operations improvements can be put in place as discussed in the focus groups (see Table 4). However, it is worth mentioning the importance of this approach for recognizing the advantages of the UAb teaching model for sustainability, from the perspective of its social dimension, namely that it facilitates education for all and at all stages. This is evidenced as an innovation according to the STARS criteria [73]. Another strong point that the assessment highlighted is the university bet in sustainability teaching not only at formal programs from 1st, 2nd and 3rd level (bachelor to Ph.D.) but also at non-formal programs like open and massive on-line courses (see Table 4).

Through the assessment of the sustainability in UAb using STARS the key-actors were able to discuss ways of improvement namely new paths to implement sustainability practices (see Table 4). The improvements found are feasible and inexpensive but UAb should integrate sustainability into organizational practices and allocate financial and human resources in the next strategic plan. In particular, the application of STARS can be an important basis for the definition of a currently non-existent University sustainability strategy/policy, one of the fundamental pillars for the whole-school approach, which agrees with other authors [13,32]. In addition, the participatory process that took place during the completion and evaluation of the tool alerted and sensitized the focus group participants on sustainability issues at UAb, contributing to the fact that some of the proposals are already being implemented at the moment. Examples are the registration of the SDG in the resources uploaded in the Open Repository of the University, the development of more moments of joy and sociability (e.g., Christmas lunch) and ongoing process of the dematerialization of the administrative process.

This tool was first used in Portugal and at a distance learning university. Its application in the UAb has also identified some points of better adaptation to European reality (in particular related to units of measurement and benchmarks) and distance learning universities (since these institutions do not have a formal campus with students). These adaptations were communicated to the tool implementation support services of STARS. Earlier studies indicate that e-learning has a lower impact on greenhouse gas emissions and climate change, as observed in the UAb assessment and according to other studies (e.g., [81]) but its direct and indirect impacts on sustainability need to be better studied, as also advocated by Findler et al. [14]. Indeed, the long-term impact of the practices being implemented and the impact of UAb on a more sustainable society are issues that the tool has not been able to assess on its own.

The scores result of the GM assessment from 2014 to 2018 at Universidad Autónoma de Madrid are available in Figure 2 and

Table 5. Summary of the results of Assessment and Improvements for Universidad Autónoma de Madrid regarding the STARS dimensions.

Academic
Assessment	Improvement
Sustainability plays a role in many subjects on the majority of the faculties, 632 courses are connected to environment and sustainability. The university is attempting to promote a participatory approach to sustainability activities together with the campus community, integrating an overall vision to a local impact There is no available tracking system to register and follow the SDG activities of the university. The SDG approach is implicit in the academic programs. Also, the research activities are scattered. When it comes to research and teaching, there are leading teams in different aspects of environmental, economic and social sustainability. UAM was one of the first universities in Spain to offer a degree in the environmental science and incorporate environment into other degrees. However, the learning method for these topics should consider a more practical method than only theory	Potential outreach in the society through scientific contribution including SDG areas in research and academic content Use the campus as an environmental classroom for practicum, thesis, etc. Collaborative online platform to show publications, projects, thesis, events connected with the SDGs Observatory for dissemination - Inventory research team SDG (dynamic, active) Involvement of teachers to motivate student participation
Engagement, Planning and Administration
Assessment	Improvement
There is a lack of awareness from the university community and society, as well as a low level of involvement and participation. UAM must manage energy and water consumption according to the restrictions defined by the City Council and National Government. UAM is working on a list of rigorous requirements for the process for suppliers in 2019. Authorities are aware of the university’s great expenses related to unsustainable actions on the campus. There is no evolution on the metrics or rankings due to a lack of financial support. Very high engagement of the UAM to sustainability: “SDG Lab Campus”; the presence of the UAM in Sustainability Conference of Rectors of Spanish Universities about Sustainability	Social center for sustainability reference: store, organic coffee shop, distribution point Association of Parents and Friends of the Disabled of the UAM (APADUAM) Social and community involvement Guide the UAM towards the circular economy Implementation of a SDGs Road Map 2018
Operations and Innovation
Assessment	Improvement
A good connection between Madrid and UAM campus by public transportation, a daily average of 8k12k cars on the campus Recycling initiatives by the implementation of technological resources. Green areas have a great impact on students’ attitudes toward the university. UAM nature protection areas are a great improvement in air quality. LED lighting has reduced consumption from 100% to 10%. There is not an evolution of the environmental image of the UAM. ECOCAMPUS is a leader of environmental programs at HEIs since 1997	Agro-ecological use: fruit trees, orchards, livestock, beekeeping. Analyze waste production and improve separation Pedestrianize the historic core area. Improve the treatment plant, analysis of water quality (contaminants); geofilter design Metro minute (Distance in meters and time on feet) located at access points (train station, bus stop) and central (Rectorate, Plaza Mayor)

. The results showed that UAM is in a good position compared with other worldwide universities, namely a 55th position in a total number of 719 in 2018. In 2018 UAM reached the highest scores due to considerable improvement in the Education and Infrastructures dimension but in 2014 its performance was better in terms of Water, Waste, Energy and Transportation. These results do not exactly mean the worst performance in the following years since the number of indicators related to these dimensions changed in the GM tool. An overall 34th position in the ranking of 2014 was obtained in a total of 361 universities ranked (corresponding to half of the universities compared to 2018).

According to the Vice-chancellors focus group (2nd focus group—Table 2), one critical area where the university should make an extra effort is the outreach, highlighting that university is a place to develop sciences, knowledge and new solutions able to contribute and affect to the society, local and global community. This factor is not considered in this tool, although considered in STARS. Participants from this 2nd focus group also highlighted the importance of undertaking the SDGs agenda in the university activities towards sustainability.

For the student focus group, a key factor to be included in the future is “the communication strategy” to foster the environmental message in the university community. This item is partly included in UI Green Metrics, in the Education and Research criteria, which requires the existence of a university-run sustainability website and the existence of a published sustainability report (Green Metrics [50]). This improvement was also highlighted in the UAb case study where it was proposed for better engagement of all university communities the development of an online collaborative platform for sustainable ideas (see Table 4).

Common barriers for sustainability implementation in both Southern European institutions are related to low levels of community participation and sustainability awareness, financial constraints and a lack of HEIs legislative framework. Those barriers are also found in earlier studies (e.g., [76]). Some weaknesses found in UAb are also found in UAM, namely in terms of SDG disclosure and low sustainability awareness from the university community and lack of financial support. Of course, in terms of campus operations, both universities’ reality is quite different due to their different sizes and regime of teaching. Nevertheless, they can learn from each other. For example, UAM, in terms of the Academic dimension can use teaching methods more based on new technologies like the ones used in distance learning at UAb, allowing also more transdisciplinary research and less theory (in accordance with authors like Lozano et al. [15]). UAM can also develop more lifelong learning programs or open courses to increase society sustainability awareness, like MOOCs, learning from the experience of UAb. In addition, UAb can learn from the experiences in terms of the SDG LAB and Eco campus initiatives that UAM uses and improve their worst performance in terms of Campus Operation. Even that UAb is a distance learning university without face to face students at the campus it has facilities where energy, water and waste measures can be implemented, learning from UAM experiences. More recently, at both Universities questionnaires are being developed to the students to understand their sustainability perceptions, engagement and motivations. With the questionnaires results new strategies can be put in place.

Looking into Figure 2 it can be seen that GM dimensions are mainly focused on environmental initiatives and actions on campus, hence, there are scarce indicators of policies, management, diversity, equity and community participation, what is in accordance with several authors (e.g., [17,29]). Also, this ranking’s main weakness could be considered that the information provided by the universities many times do not include evidence or in-situ verification. According to the experiences of the Eco campus focus group, the information sometimes can be considered subjective according to the understanding of the person in charge to provide the data. In the case of STARS, the data is validated by the rector team and also externally certified (like in UAb case study). However, the main strength is the accessibility and comparability of the outcome among the years (also in accordance with other authors like Lauder et al., [49]). The results from all university participants are available in the Green Metrics, easily accessible to compare by regions, countries and universities. From the benchmarking point of view, it could also facilitate the decision-making process for managers based on the potential analysis of the progress on these factors. The level of understanding is high and the authors of the GreenMetrics have provided the criteria, indicators and methodology used on their website (see for example the 2018 guide [50]).

5.3. Overall Discussion

The tools critical analysis on the tools and cases studies application, still raise the question of the effect of implementing these tools versus the actual integration of sustainability into an HEI. The integration of sustainability in HEIs should come along with the modification of existing structures and habits, which creates many challenges related to the involved actors, the available resources, values and strategic choices to be made [82,83]. Also, based on Kapitulcinova et al. [9], the “transformational change” should occur at the level of the entire HE. It is therefore essential that a critical mass of units comprising the institution adopt sustainable development principles in their respective tasks and duties. In addition, according to Alonso-Almeida et al. [84] and Beringer et al. [85], to achieve the sustainability maturation, the sustainability integration at HEIs should involve all dimensions into a whole-school approach.

For all those reasons it is considered that the implementation of integrative approaches and models still needs to be encouraged and further research is needed [9,14,31,76]. Regardless of this fact, the case studies demonstrated that the implementation of the tools make possible to assess the state of implementation of sustainability in HEIs, monitor it, communicate it, share it within and outside the organization and improve and stimulate change, often enabling low-cost measures to be implemented. These statements are also advocated by other authors (e.g., [16,24,28,32]).

A change process enforces an overall vision, an increasing need for change that is experienced by the stakeholders, resources to support the process and short-term gains that can be communicated [86]. In this research the application of the assessment tools was complemented with participatory processes. While participatory methods are not commonly considered in these tools, they are central support for more holistic implementation allowing for the best man-nature link and a reflection that can better respond to the transformation of the institution and individual towards sustainability [66]. These tools must have a stronger component of student community participation and involvement, as this community is a major agent of change. Given the characteristics of experimentation and research of HEIs, indicators that can be incorporated into these tools or that can be used independently, should be tested to allow a long-term evaluation of whether the transformation process has been successfully achieved. Disterheft et al. [66] suggested examples of these types of indicators, based on perspectives such as the whole-school approach, interconnection between man and nature, community cohesion, celebration and happiness and principles of democracy. As argued by Ramos [65] the challenges in the area of sustainability assessment indicators should be based on transdisciplinary, collaborative and innovative scientific development where communities and the individual play a central role. Greater emphasis should be given to the development of indicators to assess non-traditional aspects of sustainability, such as ethics, culture and art, aesthetics, governance efficiency, spirituality, solidarity, compassion and trust, which represent fewer tangible dimensions of society [65,87].

These tools are also too operational not evaluating the strategic processes, as also stressed by Arroyo [33]. being able to incorporate the unpredictable and not only knowing how to deal with linear problems but also being able to assess what external impact HEIs have in practice on sustainability and going beyond the limits of HEIs [14]. As an example, the impact assessment of the research that is developed in HEIs on ESD should be carried out. Since this cannot be done based on a simple citation counting and bibliometric analysis because they do not allow to accurately define the result of this same investigation in the SD, more qualitative and documentary analyses are necessary for a more robust evaluation [14].

Few studies have explored and evaluated the role of HEIs as agents of change [31]. Also, there is a weak link between HEIs and external networks and key actors and with local and regional policies, what does not contribute to change in organizational management [82], holistic integration of sustainability in HEIs and their impact abroad [14,88]. The external impact of implementing sustainable development in HEIs can be measured in the local economy and culture, in challenges in society, in the natural environment and in the policies. Impacts can be short-term and direct, such as by training skilled workers or reducing greenhouse gases in the facilities but also indirect and long-term, such as changing graduates’ sustainable lifestyles or in the implementation of Sustainable Development Goals (based on [14] suggestions).

6. Conclusions

A high increase has been seen in the implementation of sustainable development principles into Higher Education Institutions and in the research to assess its performance. In this article, a critical analysis of the existing tools to assess and benchmark ESD implementation was conducted by four judges according to a list of criteria. There are limitations associated with these kinds of critical review and qualitative evaluation but those limitations were weighed up in the mains findings. The tools applicability was then discussed in two case studies. The tool STARS—Sustainability Tracking, Assessment & Rating System was the one that was better classified in terms of understanding, comparability, sustainability broadness and support to decision, being one of the tools more worldwide used by different HEIs. The tool STARs was also used in one of the case studies. So, the use of STARS could be a good choice for HEIs that would like to assess and benchmark their sustainability performance according to a holistic and integrated approach.

The tools currently available for evaluating sustainability initiatives in HEIs do not all have the same objective or do not homogeneously evaluate the implementation of sustainability. As a consequence, some ambiguity is translated to its actual implementation and real contribution to the transformation for change. It is therefore recommended to set common general sustainability objectives in HEIs considering the integrative whole-school approach, regardless of some regions of institutions specificities (which may be indicated by specific objectives). Some improvements to the tools were discussed in this research and suggestions for future research. Sustainability assessment in HEIs should be viewed as a social construction, emerging from the different partners involved and according to mixed, bottom-up, top-down approaches, where the various actors, internal and external, contribute positively to the implementation of the sustainability assessment tool.

The implementation of Sustainability assessment tools in the HEIs case studies has shown that its use is an important driving force for the first diagnosis, a source for defining ways of improvement and also for future changes in organizational management. Also, the application of the tools in two HEIs of neighbor countries complemented with participation activities with key-actors, allowed to show the weakness of these tools namely the lack of the assessment of the impact of the sustainability performance of the university in the society and their real contribution for a sustainability transition. Bearing in mind that both countries have some similarities, namely cultural, this collaboration also brought up difficulties that need to be addressed in order to apply and implement EDS successfully. Barriers were identified and examples of improvement were given. Nevertheless, both universities learned from each other besides their differences and future developments for their sustainable performance improvement are already in place.

This article adds new insights about the main characteristic, common designations and objectives this type of tools should have and defined a list of criteria for their evaluation, trying to uniformize their meaning. Also, this research gives suggestions about improvements that are still needed on these tools so they can more fully answer their main purpose. Improvements can be in terms of incorporating in the tools an integrated process of stakeholder’s participation, adding of indicators to assess non-traditional aspects of sustainability and being able to assess what external impact HEIs have in practice on sustainability.