Verification of Education Credentials on European Blockchain Services Infrastructure (EBSI): Action Research in a Cross-Border Use Case between Belgium and Italy
Round 1
Reviewer 1 Report
The article deals with the application of Blockchain technology for educational credential transfer and validation in a specific use case of cross-border exchange between a Belgian and Italian university. The article is interesting and based on an ongoing real-time project for a practical application that has been conceptually presented in detail. Still, some minor concerns have to be resolved that are:
1. Several technical concepts have been introduced, like self-sovereign identity, verifiable credentials, digital wallet etc., without introducing them in the article.
2. In section 2.1, blockchain use cases in the public sector have been discussed. Still, there has been no detailed discussion about blockchain applications in the education sector or in education credentials issues and verification.
3. Related work/ previous works in the domain have not been mentioned. There should be a detailed description and review of previous works. Cite the latest references.
4. In section 4.1.2, several goals have been identified for a digital credential framework, but how these have been provided in the blockchain-based framework is not clearly discussed.
5. In section 4.3. Implementation, the description is just a theoretical one without discussing the practical implementation aspects.
Author Response
Thank you very much for your encouraging comments. We have revised the manuscript based on your recommendations. The changes are highlighted in the revised article. We have addressed your comments as follows:
First, we have added definitions to the technical concepts mentioned in your 1st remark, in Section 2.3 on p.5.
Second, we have included a new section where we discuss the application of blockchain and DLT in the education sector. The section includes a broader discussion about the current practices of the verification of education credentials and their limitations.
Third, we added new content in the introduction that references previous works recently published about the use of blockchain technology in the education sector.
Fourth, we added new content in the discussion section to assess the compatibility of EBSI and ESSIF with the goals that are identified in section 4.1.2.
About your fifth remark, the section provides an overview of the implementation process. The actual implementation of the scenarios is provided as supplementary material due to the page limitations and difficulty in adjusting the content to the formatting requirements of the template. We attach this material to the response letter for your review.
Author Response File: 
Author Response.docx
Reviewer 2 Report
The article “Verification of education credentials on European Blockchain Services Infrastructure (EBSI): Action research in a cross-border 3 use case between Belgium and Italy” is very interesting as blockchain is an emerging technology; however, there are some concerns in the manuscript, which are mentioned in the following comments and must be addressed:
1. In the introduction section, please mention some work conducted in the recent past and have a strong link with your work.
2. It would be interesting to mention which countries have adopted the blockchain approach for the same sector and what type of challenges, advantages and issues they faced and how they covered those. A comprehensive comparative analysis in the form of a table, might be a good option.
3. Some research from relevant literature should be listed where action research is used as a research methodology to validate the results.
4. Problem statement and problem description should be stated in the introduction portion which justifies the need for current research. Explicitly define the research question and build the argument on that.
5. Please present a comparison of EBSI with already implemented policies for the verification of educational credentials and also mention the improvement in process of time through appropriate comparison charts.
6. Please mention the limitations of this research based on the pilot study. Can we generalize the findings of the research? If yes, why - If No, justify in detail.
7. What are the challenges and research gaps which can be explored by future researchers?
8. There are some studies related to the application of blockchain technologies against multiple applications, published in 2023, 2022 and 2021, for the purpose of defining the scope of blockchain technologies.
The discussion section should be improved by adding a comparison between the results of this study and other recent studies being conducted in a similar area.
Author Response
Thank you very much for your detailed reading of our manuscript, and your recommendations. We have adjusted our manuscript based on your suggestions. The changes are highlighted in the revised manuscript. Please find below our explanation of how we have addressed your comments:
First, we revised the introduction by including the recent work on the topic and highlighting the gap in the existing literature.
Second, we included examples from other country cases where blockchain technology is reported to be used for the verification of education credentials. However, we did not find any academic studies focusing on the actual implementation challenges of these identified cases. The existing literature is mostly stemmed from systematic literature reviews and exploratory cases. While this gap in the literature proves the added value of our research, it also makes it difficult to include a comparative analysis. That said, we have included findings from a systematic literature review by Mohammed and Vargas (2022) on the use of blockchain in the education sector. In the revised version, we used this study and similar studies to give an overview of the most frequently mentioned challenges in the literature for the adoption of blockchain technologies in the education sector.
Third, we included references where action research is used to validate the results of technology adoption.
Fourth, we included in the first paragraph of the Introduction section the problem statement, and based on that we built the argument on the rationale of our research.
For the fifth point, EBSI is moving to the production stage as of 2023, therefore, we do not have any data to provide a comparison chart. The pilot took place in a preproduction environment. However, we added some clarifications to estimate the efficiency gain in the existing verification processes. Especially, the scenarios have shown us the total verification process can be completed within minutes instead of days in the case of electronic verification, or weeks in the case of paper-based verification.
To address your sixth, seventh, and eighth points, we have revised the conclusion section and included new content about the limitations of the study, its generalization potential, the comparison of the findings with the recent literature, and the research gaps that can be explored in future research.
Author Response File: Author Response.docx
Reviewer 3 Report
The paper provides empirical findings from the cross-border pilot system of blockchain-based document verification. The solution employs European Blockchain Services Infrastructure (EBSI) -- a major policy initiative of the EU and European Blockchain Partnership. Despite the research being more empirical than practical, it clearly provides major challenges for the wider adoption of the EBSI and can be interesting for the BDCC audience. I can recommend the acceptance of this paper after reflecting on the following comments:
- Please provide more references to EBSI
- Please provide more details about the technical components of EBSI.
- Please clearly state the performance metrics, say the number of transactions per second
- Please provide a comparison with a similar solution, for example, Singaporean OpenCerts https://www.developer.tech.gov.sg/products/categories/blockchain/opencerts/overview.html
Author Response
Thank you very much for your suggestions. In the revised version, we have provided further information about the technical component of EBSI and added several references to open sources about EBSI. Since EBSI is currently moving to the production phase, we do not have performance metrics that can help us compare the network with other similar solutions. We have requested some performance measures from the EBSI testing team, and they have accepted to provide us the following stress tests about the performance:
- Request per second till the response time is below 50ms
- Request per second till the response time is below 100ms
- Against a node / against the network (using the load balancer) reading from an API.
Unfortunately, we did not receive the requested information by the deadline for the resubmission.
Thank you also for providing the Singaporean example. An analysis of the information in the link suggests that there are several similarities between OpenCerts and EBSI services. We have included this example as other country cases with similar solutions.
Author Response File: Author Response.docx
Reviewer 4 Report
The introduction part is difficult for the reader. The purpose of the study in the introduction part is not clear. The literature review part is weak. The central part of the discussion, conclusion and future scope of the study are weak. The author has not made reasonable efforts. Please address all the above issues.
Author Response
Thank you very much for your remarks. We have extensively reworked the Introduction, literature review and discussion and conclusion sections in line with the comments of other reviewers. The attached file gives an overview of all changes in the manuscript. We hope that the revised version fulfils your expectations.
Author Response File: Author Response.docx
Round 2
Reviewer 2 Report
A proper English grammar check is required.
Author Response
Thank you very much once again for your recommendations and review. We have extensively copy-edited the content and addressed the language issues.
Reviewer 3 Report
The authors did a great job and reflected most of my comments. The only non-reflected comment is performance results. Because the authors provided it in their response, and it is extremely important, I recommend accepting this paper only after they incorporated it into the text. Also, it would be interesting to provide a comparison with other similar blockchain-based solutions based on a permissioned blockchain framework, for example, based on Exonum or Hyperledger fabric. It is important to understand the practical soundness of the solution. A similar blockchain solution includes Covid19 vaccination certificate validation or any tradable assets validation (postage stamps, plastic pipes, and so on).
Author Response
Dear Reviewer,
Thank you very much for your supportive words. We have received the performance information from the EBSI technical team. Based on that, we have included the following paragraph on p.16 as part of the results:
"Finally, performance tests conducted by the EBP during the pilot phase suggested that the previous implementation of the software stack may require technical adjustments to support upscaling. When testing an EBSI node directly, the maximum rate was 120 requests per second before a light degradation in service was observable. The response time for EBSI nodes is partially affected by various security and networking measures, including an external firewall, web application firewall, and internal node firewall. Considering that the EBSI network is a full peer-to-peer mesh of nodes sharing the same ledger, its scalability is directly proportional to the number of nodes in the network. At present, 35 nodes are fully operational, decentralized, and distributed across all member states. Multiplying 35 nodes by 120 requests per second, the current EBSI Pilot Environment can handle 4,200 requests per second. Since many improvements were implemented at the API level, an increase in the previously measured rate is expected in the production phase.
In the design of the production environment, EBSI is considering a balance between throughput, scalability, and privacy, and avoiding single points of failure or centralization. EBSI governance reports that measures will be put in place to enforce anonymity of requests from relying parties to verify information using the trust model provided by the EBSI network. While a trade-off between performance and anonymity in education verification might be acceptable, different technical solutions can be sought after for upscaling in other use cases (e.g., social security) where anonymity of transactions is more crucial."
Unfortunately, we do not have verifiable performance information from other blockchain examples (e.g. Exonum, Hyberledger Fabric), but we assume that the performance information provided in the paragraph can be used for comparison with alternative solutions.
Reviewer 4 Report
The authors have made all the corrections as per the reviewer's comments.
Author Response
Thank you very much for your review. In the final version, we have copy-edited the text and addressed some language issues.
Round 3
Reviewer 3 Report
Authors reflected almost all of my comments about performance results. The only missing part is comparison with alternative solutions. Authors pointed out that they did not find proven performance results of alternative solutions. Below I pointed out some of them. Please consider possible inclusion of corresponding performance results into the manuscript.
https://0-dl-acm-org.brum.beds.ac.uk/doi/10.1002/nem.2099
https://0-ieeexplore-ieee-org.brum.beds.ac.uk/document/9003331
Author Response
Thank you very much for your suggestions. We have included a small section about the performance comparison of EBSI with other public-permissioned blockchains. Specifically, we did the following addition on p.16:
"This performance measurement is comparable to other public permissioned blockchains such as Hyberledger Fabric [26] and Exonum [27], which report 3000-5000 transactions per second. Although these numbers are comparatively lower than centralized storage systems that can handle up to 50.000 transactions per second [28], in the public sector domains, higher security and privacy in transactions might be preferable over high transaction speed.
In the design of the production environment, EBSI is considering a balance between throughput, scalability, and privacy, and avoiding single points of failure or centralization. Previous research suggests that permissioned blockchains can be re‐engineered to support 20 000 transactions per second through a series of optimizations focusing on input/output, caching, parallelism and efficient data access [29]. However, increased performance in blockchain often comes at the expense of security or transparency [30]."
Round 4
Reviewer 3 Report
Authors incorporated all comments and I can recommend paper acceptance
