Statistical Descriptive Analysis of Portuguese Public Procurement Data from 2015 to 2022

Round 1

Reviewer 1 Report

Statistical Descriptive Analysis of Portuguese Public Procurement Data from 2015 to 2022

This appears to be an interesting paper; however, the research approach is either not explained properly or it is faulty. Thus, the results are not reliable. I am therefore, providing the following comments that might be useful to revise and improve the paper.

1.       Whenever, we select the specific period for data collection, for instance, in this case, from 2015 to 2022, there has to be a convincing argument which should justify the choice/ section. I do not see such convincing arguments in this paper.

2.       The PPP Data should have been obtained from a government or independent database and thus should be properly cited. I do not understand the citing Jacques de Sousa et al. 2022; and Jacques de Sousa et al. 2023 for this purpose.

3.       The inclusion and exclusion criteria should be clearly established and stated. Since, you are using a qualitative / content analysis approach, I suggest to review this paper “Key factors influencing the implementation of three-dimensional printing in construction” and make necessary adjustment to your research approach. This will also help you to support your choice for research approach and how to justify it.

4.       There is a contraction between what is stated in the abstract and what is mentioned in the paper. this particularly important in the context of the study period.

5.       Name of the regions in figure 2 should be stated.

6.       Figure 3 (a) and figure 3 (b) are not clear. also provide a clear box plot with table 3. Do this with all figures/ table.

7.       Ideally you should quote all the money in US $ to engage better with your international readers. do not assume your readers will be only from Europe.

8.       The paper is unnecessary lengthy with a lot of figures and tables and many of them can be easily eliminated.

9.       The analysis should be more robust, just using percentage simply do not provide in depth analysis.

10.   After reading the whole paper, I am struggling to understand the contribution of this paper. when I said, the contribution, I am referring the contribution to the knowledge. Anyone else can simply get this data and analyse it in their own way and get some finding. This is not a research. that’s why I said before, your research approach should be robust with a rigid analysis technique. Avoid general discussion because it does not add value and make the paper unnecessary lengthy.

Comments for author File: Comments.pdf

Author Response

" Statistical Descriptive Analysis of Portuguese Public Procurement Data from 2015 to 2022"

We appreciate the time and effort of the reviewers. We send one marked copy of the manuscript in which it may be seen that all the suggestions of the Reviewers have been taken into account and the paper quality has improved from the additions to the text that were included.

A detailed analysis of the reviewer' comments is attached.

We consider that all the suggestions have been taken into account.

We trust that the manuscript will meet with your approval, but should any doubt remain, please let us know.

Thank you for your attention.

Kind regards.

Jacques de Sousa, Luís

Poças Martins, João

Sanhudo, Luís

Simões, Maria Lurdes

Moreira Da Costa, Jorge

Author Response File: Author Response.docx

Reviewer 2 Report

Some general comments to the article

 Abstract:

You are missing a purpose of study section. You are also missing the main findings. Finally, statistical analysis is mentioned but the research methodology is not evident from the abstract.

Abbreviation pppdata not explained.

Introduction:

Overall the number of references are very low, this is among others due to a very limited state of the art section. This section needs to be expanded to put the research into context and to present the research gap. 

It is mentioned that the study “seeks to identify trends and best practices in public pro-curement through a descriptive statistical analysis of 5172 contracts extracted from the websites” but the benefits of identifying trends and the actual gaps that it closes should be elaborated.

Also some information in the introduction belong more in a method section than in the introduction, like number of cases and when and how the data was captured.

The method section could be improved:

You mention a data refinement process, as I understand it is a outlier detection. How did you identify the outliers, why did you choose to include missing values?

How did you actually analyze the data? Are there enough data in each region to allow for region-vice analysis?

Findings:

Page 3: Publication year, it is unclear to me if published but not closed projects are included in the study. If only studies there is completed is concluded it will of course affect number of published works especially in the later years.

Page 5: “Concerning the mode, projects with 120 days of execution time are the most fre-quent.” Why is this of relevance? Moreover, if you was interesting in showing the actual variation you should consider to including histograms.

Page 6: You should consider the effect of the missing values. This is an issue throughout the article and it confuses your findings. ” Regarding the award criteria, 2321 projects used the price criterion, which amounts to 44.9 % of all contracts. The multifactor assessment criterion was used in 1281 contracts while 1570 contracts had a missing award criterion, respectively, 24.8 % and 30.3 %.” If the reward criteria is missing, it will be safe to assume it will be one of the two above. Thus, the percentages are two low.

Page 6: “The average weight given to the price factor in these contracts was approximately 85 %, while the mode and median were 100 %.”  Does this make sense - I am assuming that price is stated to 100 % in the only price contracts. Not sure if the data is normally distributed, but maybe a hypothesis test could identify the 95 % confidence interval of mean.

Page 6: “A t-test for paired samples was performed to assess whether the observed differences between the EP and IP would be statistically significant, which resulted in a mean difference equal to…” Do we know if the data is normally distributed? Later you use a non-parametric test. Why use a parametric test and then a non-parametric test? Also, what is your accept criteria.

Page 6: ”thus it can be considered that the EP and IP are in fact similar.” You have only show that you could not verify that they are dissimilar. How can you conclude that they are similar?

Tabel 3: what does C. variation stand for? Cost variation?

I will ask the authors to re-consider which of the findings is of general relevance and try to include only the most important findings. The article is very long and it will be beneficial to shorten it. It is unclear what gap in knowledge this research aims to close and thus it can for the reader be hard to see how this adds to our current body of knowledge. For instance why is publication and close year divided between district (in page 10 important) or why is the regional relationship between IP class, Award Criteria, and Performance on page 12 of important. Moreover you need to highlight your key findings.

Page 15: You mention that the Chi squared show a significant correlation, the test mainly show if the variables are associated; if there exists a relationship between them. Did you conduct a correlation test, for instance the Spearman or the Pearson test, depending on whether your data is normally distributed.

Research validity. With this high amount of missing data, how does the authors assure that the findings are valid? In general missing values are confusing and should be removed from the sample. Like in figure 5. It is hard to see if there is an actual increase in tenders using only price as criteria or it is just because there are fewer missing values.

Discussion merely reads like a summary of the findings in the result section. You need to start discussing the findings.

Conclusion: The conclusion states the findings of your study. You should not include references in the conclusion section. The conclusion contains some of the key findings, like that there does not exist a correlation between price and award criteria.

Author Response

" Statistical Descriptive Analysis of Portuguese Public Procurement Data from 2015 to 2022"

We appreciate the time and effort of the reviewers. We send one marked copy of the manuscript in which it may be seen that all the suggestions of the Reviewers have been taken into account and the paper quality has improved from the additions to the text that were included.

A detailed analysis of the reviewer' comments is attached.

We consider that all the suggestions have been taken into account.

We trust that the manuscript will meet with your approval, but should any doubt remain, please let us know.

Thank you for your attention.

Kind regards.

Jacques de Sousa, Luís

Poças Martins, João

Sanhudo, Luís

Simões, Maria Lurdes

Moreira Da Costa, Jorge

Author Response File: Author Response.docx

Reviewer 3 Report

First of all, I would like to thank the authors, I lived in Portugal (Aveiro) for many years and I know quite well the construction market in that Country.

This paper investigates the public construction projects tender phase in Portugal by a statistical analysis on PPPData from 2015 to 2022. 

The paper is well written and clear. Also, data a well presented and described as well as the analysis of the trends, factors and correlation. However, it needs some improvements before being considered for publication.

First of all, sections and subsection must be numbered.

Secondly, number of lines must be insert to make the review easier.

Abstract: the last section should report some more findings to make the reader decide if the paper is of interest.

Introduction: second section (The awarding criteria….different factors, mainly cost, time and quality, each with different weights: the authors should provide some example and expand the literary review. Also “to identify best practices”: examples should be given. 

Table 1.: number the thematic groups. The table is completely unreadable.

Section “Publication and close year”. “published contract has increased through the years”: is it linked to National plans? Or to public procurement? Please, explain the trend. The same observation with the subsequent sentence about the exceptions (2015 and 2020).

“Submission and execution deadlines”: is the submission time in line with National regulation? What affect the trend?

Table 3.:format in a clearer way the table (i.e. aligning data, inserting raws, etc..)

Lastly, in the data analysis the authors should try to justify the observed trends, relating the survey to the National and European situation (politics and construction market) to make it clearer the possible dynamics occurred in Portugal in contrast to the international ones.

Author Response

" Statistical Descriptive Analysis of Portuguese Public Procurement Data from 2015 to 2022"

We appreciate the time and effort of the reviewers. We send one marked copy of the manuscript in which it may be seen that all the suggestions of the Reviewers have been taken into account and the paper quality has improved from the additions to the text that were included.

A detailed analysis of the reviewer' comments is attached.

We consider that all the suggestions have been taken into account.

We trust that the manuscript will meet with your approval, but should any doubt remain, please let us know.

Thank you for your attention.

Kind regards.

Jacques de Sousa, Luís

Poças Martins, João

Sanhudo, Luís

Simões, Maria Lurdes

Moreira Da Costa, Jorge

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

Thank you for addressing my previous comments. The paper is ready to be accepted. 

The english is fine, but a proofread can improve the quality 

Author Response

We appreciate the time and effort of the reviewer. We send one marked copy of the manuscript in which it may be seen that all the reviewers' suggestions have been taken into account and the paper quality has improved from the additions to the included text. 

A detailed analysis of the reviewer' comments is listed below. 

Reviewer #1:

The english is fine, but a proofread can improve the quality.

We thank the reviewer for the positive comments and the time taken to analyse the article. We would also like to confirm that the paper has been proofread to improve the quality of the text.

We consider that all the suggestions have been considered. 

We trust that the manuscript will meet with your approval, but should any doubt remain, please let us know. 

Thank you for your attention. 

Kind regards. 

Jacques de Sousa, Luís 

Poças Martins, João 

Sanhudo, Luís 

Simões, Maria Lurdes 

Moreira Da Costa, Jorge 

Reviewer 2 Report

Introduction:

More effort I still needed regarding showing the current state of the art and the research gap. What previous research has been conducted within the area of tendering and award criteria. What is still to be investigated and how does this particular research try to fill up this gap. Currently it is just explained what is analysed (page 2 line 75-85). It is not explained why this should be of interest and how this adds to the current body of knowledge. You could also consider the implications of your research.

Research Methodology

Regarding the outliers. Thank you for the elaboration. How did you identify the significant variation? A classical approach is to base it on the number of standard deviations from mean (typically if more than 3), but other approaches exist.  

Regarding the content of the method section, you could consider shortening it. It is very long, and it does not focus on the most important role, to elaborate on the data and on the actual methods applied to allow for the reader to in theory replicate the research. Moreover, it should also consider the validity and trustworthiness of the findings. Please explain the methods you applied and your considerations in relation to ensure the validity, for instance how you ensured that missing data did not affect the findings or how you ensured that enough data was present. You could also explain why you apply a t-test for paired samples.

In relation to that t-test, I agree that in case the central limit theorem is applicable then the data will tend to behave as normally distributed. But if the central limit theorem is applicable, then due to the size of the dataset you will be able conduct a normality test. If it fails, you should not assume the data to be normally distributed.

Agreed regarding the Chi-squared test, it will be applicable with categorical variables.

Regarding the missing values, I think they affect all your findings. It is not enough just to add a paragraph telling of this issue. I think some of your findings are misleading due to the missing values. I think you should consider adjusting your approach on how to treat missing values in your analysis. I am afraid that your key finding that there is no relationship between award criteria and performance might be affected by the high number of missing values.

The text is still very long, actually it is two pages longer than the previous version. It seems like if you want to include all your findings, you will need to go through the text to identify the most important findings in relation to the gap you want to close, which is still unclear. As a reader you end up being overwhelmed by all these different findings.

Defining the gap and the purpose of the research also makes the appropriate content of the discussion and conclusion section more obvious. As by now the conclusion focuses on your findings, which is fine, but it should also close the loop to your gap and answer your “research question”. I know you don't have an explicit question but you should try to state how the research fulfilled its “purpose”. 

Author Response

We appreciate the time and effort of the reviewer. We send one marked copy of the manuscript in which it may be seen that all the reviewers' suggestions have been taken into account and the paper quality has improved from the additions to the included text. 

A detailed analysis of the reviewer' comments is listed below. 

Reviewer #2: 

More effort I still needed regarding showing the current state of the art and the research gap. What previous research has been conducted within the area of tendering and award criteria. What is still to be investigated and how does this particular research try to fill up this gap. Currently it is just explained what is analysed (page 2 line 75-85). It is not explained why this should be of interest and how this adds to the current body of knowledge. You could also consider the implications of your research.

Thank you for your feedback and suggestions regarding our study's current state of the art and research gap. We acknowledge that we need to provide a more comprehensive overview of previous research conducted in the tendering and award criteria area. In this sense, in lines 50-53, we emphasise the research gap that exists in understanding the impact of these tender criteria options on the performance of public projects and in recognising the correlations between the different tender variables. Furthermore, no previous research has comprehensively investigated the correlations between the tender variables analysed in this study over an extensive time period.

Regarding the outliers. Thank you for the elaboration. How did you identify the significant variation? A classical approach is to base it on the number of standard deviations from mean (typically if more than 3), but other approaches exist. 

We would like to thank the reviewer for their positive comment. Regarding identifying significant variations, we used another classical approach that relies on extreme percentiles. In this sense, the seven contracts with significant deviations between initial and effective prices were eliminated because they were outside the ]10;90[ percentiles of the price proportions. As mentioned in the manuscript, the other 32 contracts were legally classified as "Not compliant", so these were also eliminated. In line with the reviewer's suggestion, a clarification on the elimination criteria was added to the manuscript in lines 114 to 117.

Regarding the content of the method section, you could consider shortening it. It is very long, and it does not focus on the most important role, to elaborate on the data and on the actual methods applied to allow for the reader to in theory replicate the research. Moreover, it should also consider the validity and trustworthiness of the findings. Please explain the methods you applied and your considerations in relation to ensure the validity, for instance how you ensured that missing data did not affect the findings or how you ensured that enough data was present. You could also explain why you apply a t-test for paired samples.

We have extensively revised the methods section with the aim of both shortening it and enhancing its clarity. The revised section now includes a detailed explanation of how we accessed the main trends in the data and utilised statistical tests, such as the t-test for paired samples. Additionally, we acknowledge that missing values in the reported data may have influenced the research findings. However, we argue that these missing values provide valuable insights into the current status of construction reporting and that the dataset studied represents the whole Portuguese public construction contracts in the analysed timeframe.

In relation to that t-test, I agree that in case the central limit theorem is applicable then the data will tend to behave as normally distributed. But if the central limit theorem is applicable, then due to the size of the dataset you will be able conduct a normality test. If it fails, you should not assume the data to be normally distributed.

According to the reviewer's comments, we performed a normality test on the IP and EP variables which failed. Although the data may not follow a normal distribution, we applied the t-test. We also conducted the Wilcoxon non-parametric test to exclude any remaining doubts, which yielded consistent results (p-value = 0.394). We added this explanation to the manuscript in lines 262 to 271.

Agreed regarding the Chi-squared test, it will be applicable with categorical variables.

We want to thank the reviewer for the positive comment.

Regarding the missing values, I think they affect all your findings. It is not enough just to add a paragraph telling of this issue. I think some of your findings are misleading due to the missing values. I think you should consider adjusting your approach on how to treat missing values in your analysis. I am afraid that your key finding that there is no relationship between award criteria and performance might be affected by the high number of missing values.

While the number of missing fields in the remaining variables is minimal, we recognise the significance of addressing the missing award criteria in our analysis. As previously stated, we aimed to shed light on the data sourcing issue prevalent in public procurement repositories in Portugal to examine the reduction of missing information over time. Still, we agree with the reviewer's comment that the high number of missing values might affect the relationship between award criteria and performance. In this sense, we have stated this limitation in the manuscript in lines 497 to 499.

The text is still very long, actually it is two pages longer than the previous version. It seems like if you want to include all your findings, you will need to go through the text to identify the most important findings in relation to the gap you want to close, which is still unclear. As a reader you end up being overwhelmed by all these different findings.

To improve readability, the manuscript increased in size from the first to the second version due to an increase in figure size. Following the reviewer's comments, an effort has been made to shorten the manuscript. Chapter 4 has been fully revamped to respond to the extension problem; figures 11, 12 and 13 have been compiled in a table (Table 6). The manuscript now has 20 pages of text, including references. Additionally, we believe that the changes made to the manuscript have improved the text clarity, including the research gap it tries to close.

Defining the gap and the purpose of the research also makes the appropriate content of the discussion and conclusion section more obvious. As by now the conclusion focuses on your findings, which is fine, but it should also close the loop to your gap and answer your "research question". I know you don't have an explicit question but you should try to state how the research fulfilled its "purpose". 

The conclusions have been enhanced based on the reviewer's suggestion to improve their quality. They now explicitly address the paper's main findings and reiterate the research gap identified in the introduction section. By closing this logical loop and highlighting how this paper fills the identified research gap, we aim to enhance the readability and coherence of the article.

We consider that all the suggestions have been considered. 

We trust that the manuscript will meet with your approval, but should any doubt remain, please let us know. 

Thank you for your attention. 

Kind regards. 

Jacques de Sousa, Luís 

Poças Martins, João 

Sanhudo, Luís 

Simões, Maria Lurdes 

Moreira Da Costa, Jorge 

Round 3

Reviewer 2 Report

State of the art:

Ok, even though very limited research has been added to the state of the art regarding what previous studies have conducted and as a reader it is not very convincing that this covers the state of the art.

You state: “Therefore, studying the effects of the tender's variables on the project's performance may help construction professionals to identify best practices when selecting the best bidder 47 and proposal structure, and improve the overall efficiency of the procurement process 48 (Carbonara and Pellegrino 2020). 49 Previous studies have analysed the use of tender criteria in Portuguese public ven- 50 tures but have not investigated the impact of these options on performance or the correla- 51 tions between the other tender variables analysed in this study over such a broad time 52 period (IMPIC 2019).

Which is fine, even though you have still not added what knowledge or value the study creates. I am only looking for a sentence stating the gap it aims to close and what value it adds and to whom.

You have mentioned this elsewhere, for instance in the abstract where you mention something regarding procurement benchmarking, and you mention in the conclusion something regarding influence of award criteria (even though your findings c.f. your bullet points are broader).

T-test vs. Wilcoxon

The T-test can be applied to compare mean values and differences in mean while Wilcoxon only compares rank. The Wilcoxon test is, therefore, only able to conclude if there are differences in rank (median) and not the size of the difference. You cannot just assume that the central limit theorem is applicable, this will not always be the case. Maybe the data tends to follow a different distribution with higher skewness or a logarithmic distribution or something else.

Missing data:

I cannot see the argument of why to include missing data in an analysis between award criteria and performance.

Author Response

We appreciate the time and effort of the reviewer. We send one marked copy of the manuscript in which it may be seen that all the suggestions of the Reviewer have been taken into account and the paper quality has improved from the additions to the text that were included. 

A detailed analysis of the reviewer’s comments is listed below. 

State of the art:

Thank you for your comment. We appreciate your feedback and have taken it into consideration. In response to your suggestion, we added a new sentence from lines 50 to 55 that explicitly highlights the research gap this study aims to close. This statement aligns with the abstract and the paper's findings, emphasising the study's value and contribution.

T-test vs. Wilcoxon:

Thank you for your comment regarding the comparison between the t-test and the Wilcoxon test. It is true that the t-test is used to compare means and differences, while the Wilcoxon test compares only the order or rank of values. Therefore, we agree with the reviewer’s point of view, that it is important to consider the data distribution and the appropriate assumptions when choosing between the t-test and the Wilcoxon test. From the exploratory data analysis, the data exhibits high skewness and both variables follow the exponential distribution. So, the Wilcoxon test is the suitable option. The text has been modified to consider only the comparison of paired samples using the Wilcoxon test, testing the hypothesis of a median difference. We hope that we’ve addressed your concerns.

Missing data:

Thank you for your comment. We appreciate the reviewer's suggestion regarding the missing data in the analysis between award criteria and performance. In response to this suggestion, we have re-evaluated the analysis and disregarded the missing values. Consequently, a new chi-squared test was conducted, which resulted in an asymptotic significance of 0.304. This updated analysis demonstrates that there is no correlation between the two variables. The changes made in lines 491 to 495 and Figure 8 reflect these updated findings.

We consider that all the suggestions have been considered. 

We trust that the manuscript will meet with your approval, but should any doubt remain, please let us know. 

Thank you for your attention. 

Kind regards. 

Jacques de Sousa, Luís 

Poças Martins, João 

Sanhudo, Luís 

Simões, Maria Lurdes 

Moreira Da Costa, Jorge