A Process for Monitoring the Impact of Architecture Principles on Sustainability: An Industrial Case Study

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

1. Paper summary

The authors propose a process pipeline to perform the sustainability analysis, i.e., to obtain PRSM models and their extension. The pipeline associates architecture principles with sustainability quality attributes, KPIs, and measurement tools to monitor them. The paper includes the visualization of the derived measurements in the form of two types of spider charts. The graphical representations can be used on the strategic, operational, and tactical levels to derive a principle’s performance on sustainability; Authors also add the application of the process pipeline and visualizations in a real-world context to conclude future studies.

2. Strengths

• The paper proposes a process pipeline to perform the sustainability analysis, i.e., to obtain PRSM models and their extension which can solve an important problem.

• The writing of the paper is generally good and easy to follow.

• The paper involves a human study.

3. Weaknesses

• There is no statistical significance test in the experiments.

4. Comments for authors

1. Significance

1. Authors may want to add the statistical significance test in the experiments.

2. Soundness

1. The paper includes the human study which is important for soundness.

2. Authors may want to discuss more about the recruitment process.

3. Novelty

1. The paper proposes a process pipeline to perform the sustainability analysis, i.e., to obtain PRSM models and their extension which enhances the novelty.

4. Presentation

1. The paper is easy to follow and understand. The presentation is good to me.

5. Verifiability

1. The paper includes a replication package.

6. Some minor comments

1. The first paragraph in the Introduction section is a little bit distracting.

Comments on the Quality of English Language

Line 9: utilised -> utilized

Line 226: documentation -> documentations

Line 390: categories -> category

authorisation -> authorization

Line 1133: an -> a

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The research is motivated by the increasing importance of sustainability in software engineering. It aims to extend the Principle, Rationale, Strategies, Measures (PRSM) model to a tool-dependent version (PRSM+T) that incorporates measurement tools for evaluating sustainability. This extension facilitates the operationalization of KPIs related to software architecture principles, enabling long-term monitoring of their sustainability impact.

The study's objectives are twofold: to identify accessible tools for measuring sustainability KPIs within an organization and to assess the extent to which these KPIs can be monitored automatically.

• The research introduces a process for evaluating and monitoring the impact of architecture principles on sustainability, demonstrated through a case study at a Dutch airport management company.
• Data collection included case analysis and expert interviews, utilizing Key Performance Indicators (KPIs), measurement tools, and spider charts for monitoring sustainability impacts.
• Findings suggest architecture principles can effectively address sustainability across all architecture layers within an enterprise, guiding the architecting process towards sustainability.
• The study aims to bridge the gap in current research by offering a comprehensive view on the influence of architecture principles on the four dimensions of sustainability: technical, economic, environmental, and social.
• A follow-up study extends the Principle, Rationale, Strategies, Measures (PRSM) model, evaluating it on a real-world software solution within the Schiphol Group, aiming for sustainable IT.
• The process pipeline developed associate’s architecture principles with sustainability quality attributes, KPIs, and measurement tools, visualized through spider charts for different organizational levels.
• The study's contributions include a method for sustainability analysis, the application of this method in a real-world context, and insights for future sustainability-oriented architecture practices.
• Key concepts such as SMART criteria for KPIs, the SQ model for sustainability quality, and tools like the SAF-Toolkit and D-matrix are detailed for understanding and applying sustainability measures in software architecture.
• The research underscores the importance of incorporating sustainability into software engineering to meet strategic goals and address global environmental challenges.

The authors can improve the paper by shortening it. The paper is too long (41 pages).

The authors should add a paragraph in the conclusion section discussing about the internal and external threats to validity and reliability of this study.

Comments on the Quality of English Language

NA

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

Concept of Architecture:

Authors consider the concept of Architecture from the viewpoint of “software architecture.” Please make it clear in the Abstract and in the Introduction.

Concept of Sustainability: 

Still trying to understand why we need to use the KPIs. The usual definition of sustainability is as follows:

“Sustainability means fulfillment of current needs without lowering Earth’s potential to fulfill future needs.”

N. N. Report of the World Commission on Environment and Development. Annex to General Assembly document A/42/427, Nairobi, Kenya. United Nations, New York, 1987.

This concept of sustainability leads to 17 SDGs.

N. N. The 17 Goals | Sustainable Development, URL:: https://sdgs.un.org/goals

Now, there are 232 SDG indicators, but only 93 are classified as Tier 1, and the rest are Tier 2 and Tier 3 indicators. Tier 1 indicators are clearly defined, and data are compiled by internationally accepted standards from at least 50 percent of the countries. Tier 2 indicators are clearly defined, but countries do not regularly produce the data. Tier 3 indicators are conceptually clear, but no internationally accepted standards are yet available.

S. MacFeely, “The Big (data) Bang: Opportunities and Challenges for Compiling SDG Indicators,” Global Policy, vol. 10, no. S1, pp. 121-133, 2019, doi: https://0-doi-org.brum.beds.ac.uk/10.1111/1758-5899.12595.

If the authors consider SMART, they only refer to Tier 1 indicators. In that case, how the architecture must be redesigned to cover other important indicators is not clear.

Based on the above contemplation, the reviewer asks the authors to highlight sustainability from the general perspective and then introduce their formulation of sustainability.

Typos:

Figure 7. there are some unnecessary full stops, i.e., “Fault tolerance.”> “Fault tolerance”

Line 753 … between [0-1] > “in the interval [0,1]”

Length of the manuscript:

The manuscript is too long. In fact, the architecture of the software is a concept map, i.e., semantically annotated metadata. Since the clime is the concept map takes care of the sustainability in the real-life practice, the current architecture and architecture driven by sustainability need to be compared. If the authors could have produced a method to come up with the metadata (concept map) representing the functional requirements of the software, it would have been much easier for the authors to put the argument concisely.

Comments on the Quality of English Language

please note typos

Author Response

Please see the attachment.

Author Response File: Author Response.pdf