Squill: Testing DBMS with Correctness Feedback and Accurate Instantiation

Round 1

Reviewer 1 Report

Greetings,

I did not find any useful information in this paper with respect to the earlier MS Session 3E: Fuzzing/Trusted Execution Environments, by some other authors. Why authors have released thier own squill  at https://github.com/ 124 imbawenzi/Squill.

Following are the contributions

We investigate the challenges of current grey-box DBMS fuzzing and offer our views on how to address them. • We propose correctness-guided mutation, which utilizes the correctness of seed exe- cution as feedback to guide fuzzing and improve efficiency. Moreover, we propose semantics-aware instantiation to address the challenge of correct semantics generation for SQL statements with nested structures. We implement Squill, a coverage-guided DBMS fuzzer that applies the two solutions above. • We evaluated Squill on several real-world DBMSs and found 63 bugs, proving its 123 effectiveness. We have released the source code of Squill at https://github.com/ imbawenzi/Squill.

The other paper also has

Wemakethe following contributions in this paper: • Wepropose syntax-preserving mutation and semantics-guided instantiation to address the challenges of fuzzing DBMSs. • Wei mplement Sqirrel, an end-to-end system that combines mutation and generation to detect DBMS bugs. • Weevaluated Sqirrel on real-world DBMSs and identified 63 memory error issues. The result shows that Sqirrel outperforms existing tools on finding bugs from DBMSs. Wehave released the code of Sqirrel online (https://github.com/ s3team/Squirrel) to help DBMS developers test their products and to boost future research in securing DBMSs. 

 

Author Response

Response to Reviewer 1 Comments

Point 1: I did not find any useful information in this paper with respect to the earlier MS Session 3E: Fuzzing/Trusted Execution Environments, by some other authors. Why authors have released thier own squill at https://github.com/ 124 imbawenzi/Squill.

Response 1: Thank you for your rigorous comment. Our work is implemented based on Squirrel, which is the state-of-the-art DBMS fuzzing tool, and we made it clear in the manuscript, such as in the background section. We cited Squirrel in the 11th of the reference. In the experimental section, we also did comparative experiments with Squirrel. The main contributions of our paper are:

1. We propose a new seed scheduling mechanism based on the correctness of the seed, which Squirrel doesn’t have.

2. We propose a new instantiation method that can correctly fill semantics to SQL statements with nested structures, which Squirrel can’t do.

We have implemented all our methods in the code we released on GitHub. For example, you can see the code of Semantics-Aware Instantiation at https://github.com/imbawenzi/Squill/blob/main/src/validate.cpp. Thank you again for your valuable comments.

Reviewer 2 Report

Overall, I find this paper well-structured and interesting. The authors try to address the issues of DMBS grey and black-box testing techniques, such as the lack of efficiency due to the excess of randomly generated operations with little probability of triggering bugs or lack of semantic or syntactic compliance of generated operations. Particularly regarding squirrel, which is mentioned as a state-of-the-art grey fuzzer, authors refer its shortcomings regarding nested structures, due to lack of perception regarding context and depth.

The authors' proposal, named squill, tries to address the aforementioned shortcomings by introducing correctness-guided mutation to steer the seed selection and mutation process, thus addressing the correctness feedback and instantiation issues found in squirrel.

The squirrel analisys that is presented as part as the background work is pertinent and useful, as well as the related work section, which is adequate and comprehensive for this paper type. The presentation of the proposed method is easy to follow, well-structured and detailed. The evaluation strategy looks sound, with a proper result analysis. 

I appreciate the introduction of section discussing the limitations of the present implementation, which is commendable.

A small remark (related to the abstract and introduction): I wouldn't consider a DBMS to be a critical infrastructure. Also, some figures (such as the graphs in figure  7 and 8) are quite small and difficult to read.

Author Response

Response to Reviewer 2 Comments

Point 1: A small remark (related to the abstract and introduction): I wouldn't consider a DBMS to be a critical infrastructure.

Response 1: We have revised the representation. Thank you for your nice advice.

Point 2: Some figures (such as the graphs in figure 7 and 8) are quite small and difficult to read.

Response 2: We have modified the size of the graph. We gratefully appreciate your valuable suggestion.

Reviewer 3 Report

• Quality of Figures: Figures are legible but as it is important part of study so need to be more clear. They support the information provided in the paper but add more information and provide proper referenced within the text. Study shows computer-generated figures which is encouraging. 

• Quality of English language: The spelling, Sentences, grammar, word usage, punctuation, etc. of the text need to be scrutinized. Check it again, there are mistakes. 

• Organization: The paper content and style used is presented in a manner that is easy for the reader to follow.

 • Completeness: Appropriate citations in the reference list, descriptive captions on all figures and tables, and conclusions substantiated by statements within the previous text is presented clearly.  

 

1.      Structure your abstract as follows- 1) Background 2) Aim/Objective 3) Methodology 4) Results 5) Conclusion. Write 2-4 lines for each  and  merge everything in one paragraph without any subheading

2.      Abstract must contain the motivation and objective of the article. The Abstract must be very clear and the motive of the paper should be represented in a nutshell.

3.      Introduction should be of 5-7 solid paragraphs and provide structure of work at the end of the Introduction section.

4.      Add more contribution to your study field.

5.      The purpose of study not clear.

6.      Make highlight for objectives

7.      Remove any table or figure which is taken from web. Otherwise you have to get approval from publisher and author in a provided form by springer.

8.      Please avoid to write definitions of terms like coverage-based greybox fuzzing; database testing, etc., which are already available over web, try to cite work for such information.

9.      In summary, only provide useful content in your work.

10.   These are Title related to your area, you may use.

 

 

Yan, L., Ahmad, M.W., Jawarneh, M., Shabaz, M., Raffik, R. and Kishore, K.H., 2022. Single-Input Single-Output System with Multiple Time Delay PID Control Methods for UAV Cluster Multiagent Systems. Security and Communication Networks, 2022.

 

Gao, H., Kareem, A., Jawarneh, M., Ofori, I., Raffik, R. and Kishore, K.H., 2022. Metaheuristics Based Modeling and Simulation Analysis of New Integrated Mechanized Operation Solution and Position Servo System. Mathematical Problems in Engineering, 2022.

Author Response

Response to Reviewer 3 Comments

Point 1: Structure your abstract as follows- 1) Background 2) Aim/Objective 3) Methodology 4) Results 5) Conclusion. Write 2-4 lines for each and merge everything in one paragraph without any subheading

Response 1: We have restructured the abstract. Thank you for your nice advice.

Point 2: Abstract must contain the motivation and objective of the article. The Abstract must be very clear and the motive of the paper should be represented in a nutshell.

Response 2: We have condensed the abstract and highlighted the motivation and objective of the article.

Point 3: Introduction should be of 5-7 solid paragraphs and provide structure of work at the end of the Introduction section.

Response 3: We have restructured the introduction and added a paragraph at the end of the introduction to introduce the structure of our work.

Point 4: Add more contribution to your study field.

Response 4: We have enriched the content of the contribution section.

Point 5: The purpose of study not clear.

Response 5: We have changed the paragraph in the introduction to clarify the motivation of the study.

Point 6: Make highlight for objectives.

Response 6: We have revised the motivation section and highlighted the objective of the study at the end of each subsection.

Point 7: Remove any table or figure which is taken from web. Otherwise you have to get approval from publisher and author in a provided form by springer.

Response 7: Thanks for your kind reminder. There is no table or figure taken from the web in our article.  

Point 8: Please avoid to write definitions of terms like coverage-based greybox fuzzing; database testing, etc., which are already available over web, try to cite work for such information.

Response 8: We have changed these definitions and cited them from the web.

Point 9: In summary, only provide useful content in your work.

Response 9: We have removed redundant content in the conclusion section.

Point 10: These are Title related to your area, you may use.

Response 10: These articles give us many references, and we have cited them in our article. Thank you again for your valuable comments.

Reviewer 4 Report

This paper investigate e the challenges of current grey-box DBMS fuzzing and offer point of  views on how to address them. And then propose correctness-guided mutation, which utilizes the correctness of seed execution as feedback to guide fuzzing and improve efficiency

The paper deal with  an important matter that falls in the series of the journal, but The article  has many weaknesses, in my opinion (Minor revision ) - in the following - I clarify the highest shortcomings in terms of structure, objective, declaration of the problem.

The logic of the abstract and introduction is not clear enough, the author should explain it more clearly. The motivation and novelty are a bit unclear. Further elaboration could help improving the clarity of this paper.

 

More Information about the results depicted in  " Figure 2. Contributions of seeds with different correctness in a DBMS fuzzing process" ((Is required))

In " Figure 3. Squirrel’s instantiation of SQL statements with a subquery" ((not clear more detail and explication is required))

" Algorithm 1 Correctness-Focused Seed Selection"((must rewritten in slandered format of Algorithm))

Change the capture of " Figure 4. Overview of Squill" to be more meaningful

State the performance metrics for evaluating the proposal

What is the value of section " 7. Related Work" at the end of the paper

Author Response

Response to Reviewer 4 Comments

Point 1: The logic of the abstract and introduction is not clear enough, the author should explain it more clearly. The motivation and novelty are a bit unclear. Further elaboration could help improving the clarity of this paper.

Response 1: We have restructured the abstract and the introduction to highlight the motivation and objective of the article. We gratefully appreciate your valuable advice.

Point 2: More Information about the results depicted in  " Figure 2. Contributions of seeds with different correctness in a DBMS fuzzing process" ((Is required)).

Response 2: We have supplemented the meaning of the horizontal and vertical coordinates in Figure 2 to make the results better understandable.

Point 3: In " Figure 3. Squirrel’s instantiation of SQL statements with a subquery" ((not clear more detail and explication is required))

Response 3: We have added more details to the description of Figure 3 for better understanding.

Point 4: " Algorithm 1 Correctness-Focused Seed Selection"((must rewritten in slandered format of Algorithm))

Response 4: We have rewritten Algorithm 1 to make the process more intuitive.

Point 5: Change the capture of " Figure 4. Overview of Squill" to be more meaningful

Response 5: We have redrawn Figure 4 and added more information to the figure.

Point 6: State the performance metrics for evaluating the proposal

Response 6: We have stated the performance metrics and illustrated the relationship between results and performance before presenting the results.

Point 7: What is the value of section " 7. Related Work" at the end of the paper

Response 7: The related work section mainly presents some recent academic works on DBMS fuzzing and explains the difference between this paper and those works. It can introduce the research background to readers and highlight the contribution of this paper. Thank you again for your valuable comments.

Reviewer 5 Report

-a table with the differences of Squill and Squirrel according to the paper experiments would be useful

-the reason(s) that the Squill outperform over Squirrel should be stated

-The two instantiation examples with Squirrel (Figure 1, Figure 3) with their relations using the data dependency graphs examples (DDG) should be executed using Squill as well. Any similarity /difference should be explained

Author Response

Response to Reviewer 5 Comments

Point 1: a table with the differences of Squill and Squirrel according to the paper experiments would be useful

Response 1: We have added a table to show the difference in experimental results between Squill and Squirrel. Thank you for your kind advice.

Point 2: the reason(s) that the Squill outperform over Squirrel should be stated

Response 2: We have discussed the reason why Squill outperforms Squirrel in the summary part of the experiment (section 5.2).

Point 3: The two instantiation examples with Squirrel (Figure 1, Figure 3) with their relations using the data dependency graphs examples (DDG) should be executed using Squill as well. Any similarity /difference should be explained

Response 3: We have redrawn Figure 6 to illustrate the process of using data dependency graphs in Squill. And in the following description, we have added more details about the instantiation process in Figure 6 and discussed the differences with Squirrel. Here we only select a part of the instantiation process in the example to introduce the method of Squill because this part can go through the complete process of Squill's instantiation. Using Squill to fully instantiate the examples in Figures 1 and 3 would make the description too cumbersome. We gratefully appreciate your valuable advice.

Round 2

Reviewer 1 Report

Greetings,

In my opinion, the authors have done a good job and answered all my questions