Reliability and Agreement of Three Devices for Measuring Implant Stability Quotient in the Animal Ex Vivo Model

Round 1

Reviewer 1 Report

The paper matches the Journal's items focusing on methods for measuring implant stability, with a particular interest in determining inter- and intra- rater reliability and agreement of particular devices in comparative terms. The study is an investigation of the stability of 30 implants placed in three different mandibles evaluated using three devices (Osstell® Beacon, Penguin® and MegaISQ®).

The paper presents the statistical analysis of the relative measurements coded by two operators and a sufficient discussion of the results of the tests performed on the implants.

In the study, enough details concerning the methodology used and the conditions for the analysis are provided, calculating the specific coefficients with respect to the three selected devices and the two orientations indicated (parallel and perpendicular). The obtained results are illustrated comparing the ISQ and correlation values. Some limitations in the research process are also discussed for a purpose of future developments.

The authors have highlighted enough clearly the aim of the study and their contribution with respect to previous studies on the item. However, in order to increase the quality of the paper, the work needs to be revised. Some suggestions are recommended to the authors as follows:

- In the title, to be more precise, it is suggested to replace the word "instruments" with "devices" as this is constantly used in the paper.

- In the Abstract, it is not necessary to insert for the reader the 4 subtitles (background, methods, results and conclusion). Please remove them from the text and just merge the content of the section.

- In the Introduction section, after stating the goal of the research study, the authors should clarify the main procedural aspects of the set-up of the work that will then be explained in the following sections.

- Another important aspect inherent in the measuring of Implant Stability that could be treated by the author concerns the Modal analysis. The subject is well covered in the following works [*]that could be included in the references and discussed in the Introduction.

[*] Zanetti, E. M., Ciaramella, S., Calì, M., Pascoletti, G., Martorelli, M., Asero, R., & Watts, D. C. (2018). Modal analysis for implant stability assessment: Sensitivity of this methodology for different implant designs. Dental Materials, 34(8), 1235-1245.

Atsumi, M., Park, S. H., & Wang, H. L. (2007). Methods used to assess implant stability: current status. International Journal of Oral & Maxillofacial Implants, 22(5).

- In section 2, the subsection 2.2 Statistical analysis should be changed into 2.1. Please correct it.

- In the results section, the tables inserted could be better introduced in the text with respect to the self-explanatory title of each table (especially the first two tables). Please also check that they are as small as possible.

- The conclusions only report the obtained results while the experimental findings could be better described strictly in relation to the issue of interest, the implant micromotion, and with respect to a successful osseointegration.

- As far as the English language is concerned, it can be improved. The paper is written sometimes too simply with repetitive terms which could be replaced by appropriate synonyms. A wider and richer use of vocabulary is thus recommended. Please also correct the following sentences:

Abstract

- however, inter- and intra- rater reliability and agreement of some devices remains unknown.

- by 2 different operators

Introduction

- and ITV only assess conditions at the time of implant installation

- Moreover, ITV and ISQ values shows an inverse correlation with implant micromotion,

- The industry has provided different devices for measuring ISQ.

- To the best of our knowledge, inter- and intra- rater reliability and agreement of Osstell® Beacon, Penguin® and MegaISQ® to the others remains unknown.

Section 2

- ...., a minimum of 20 samples were necessary on each group to identify a statistically significant difference greater than or equal than to 2 units.

- Based on a recent study [14], standard deviation was assumed to be 5  and the correlation coefficient measurements was 0.9.

- All procedures were repeated by 2 different experienced and calibrated operators

- Test–retest was used to calculate the intraclass correlation coefficient (ICC) using a mixed model with a random effect for the individual for assessing the intra- and inter-rater reliability.

Section 4

- The reliability of Penguin® was good when measuring in parallel, and moderate to good reliability when measuring perpendicular.

- Other study with high ICC scores for Osstell®, reported a good reliability for both Penguin® and Osstell®

- This observation suggest that the values obtained from these devices can be operator dependent.

- Osstell® Beacon showed lower values that previous studies using different types of Osstell®

- In our study, the highest ISQ values were obtained using Penguin® and the lowest values were using MegaISQ®.

- However, the difference with the mean was 2 times lower with MegaISQ® than with Penguin, then the underestimation of MegaISQ® has not the magnitude of Penguin® overestimation.

- Other limitation was the manual tightening of the transducers, but this technique was reported to be objective and reliable previously

Author Response

Reply to reviewer 1

Reviewer 1. Thank you for your comments in order to improve the manuscript.

The paper matches the Journal's items focusing on methods for measuring implant stability, with a particular interest in determining inter- and intra- rater reliability and agreement of particular devices in comparative terms. The study is an investigation of the stability of 30 implants placed in three different mandibles evaluated using three devices (Osstell® Beacon, Penguin® and MegaISQ®).

The paper presents the statistical analysis of the relative measurements coded by two operators and a sufficient discussion of the results of the tests performed on the implants.

In the study, enough details concerning the methodology used and the conditions for the analysis are provided, calculating the specific coefficients with respect to the three selected devices and the two orientations indicated (parallel and perpendicular). The obtained results are illustrated comparing the ISQ and correlation values. Some limitations in the research process are also discussed for a purpose of future developments.

The authors have highlighted enough clearly the aim of the study and their contribution with respect to previous studies on the item. However, in order to increase the quality of the paper, the work needs to be revised. Some suggestions are recommended to the authors as follows:

- In the title, to be more precise, it is suggested to replace the word "instruments" with "devices" as this is constantly used in the paper.

Response: Thank you for pointing this out. We have replaced "instruments" by "devices" as suggested by the reviewer.

Text change: We have replaced “instruments" for "devices" in line 2

- In the Abstract, it is not necessary to insert for the reader the 4 subtitles (background, methods, results and conclusion). Please remove them from the text and just merge the content of the section.

Response: Thank you for your suggestion. The subtitles of the Abstract section have been removed.

- In the Introduction section, after stating the goal of the research study, the authors should clarify the main procedural aspects of the set-up of the work that will then be explained in the following sections.

Response: Thank you for your comment. We have added a sentence at the final part of the Introduction section.

Text change: This study was conducted in vitro by two investigators to obtain the inter- and intra- rater reliability and the agreement level among these three devices.

- Another important aspect inherent in the measuring of Implant Stability that could be treated by the author concerns the Modal analysis. The subject is well covered in the following works [*] that could be included in the references and discussed in the Introduction.

[*] Zanetti, E. M., Ciaramella, S., Calì, M., Pascoletti, G., Martorelli, M., Asero, R., & Watts, D. C. (2018). Modal analysis for implant stability assessment: Sensitivity of this methodology for different implant designs. Dental Materials, 34(8), 1235-1245.

Atsumi, M., Park, S. H., & Wang, H. L. (2007). Methods used to assess implant stability: current status. International Journal of Oral & Maxillofacial Implants, 22(5).

Response: We agree with the reviewer, this topic and the two references have been added to the Introduction section.

Text change: The sentence “Several existing methods have addressed measuring implant stability, including theoretical [6] and experimental modal analysis [7].” has been modified.

The sentence “Periotest and ISQ are considered modal analysis methods based on the displacement signal secondary to an external impulse force [7].“ has been added.

- In section 2, the subsection 2.2 Statistical analysis should be changed into 2.1. Please correct it.

Response: Thank you for pointing it out to us. It has been corrected.

- In the results section, the tables inserted could be better introduced in the text with respect to the self-explanatory title of each table (especially the first two tables). Please also check that they are as small as possible.

Response: Authors agree with the reviewer. A new sentence has been added to the first paragraph of the Results section and Table 2 has been deleted and will be sent as supplementary information.

Text change: The paragraph: “Table 1 shows the implant stability measurements obtained by each device. As shown, using Penguin in par-allel measurement yields the highest ISQ values; in contrast, using the MegaISQ® in a perpendicular measurement results in the lowest values.” has been added.

- The conclusions only report the obtained results while the experimental findings could be better described strictly in relation to the issue of interest, the implant micromotion, and with respect to a successful osseointegration.

Response: This description has been added in the Discussion and the Conclusions sections.

Text change: In the first paragraph of the Discussion Section the sentence “From these results, Penguin® should be used to monitor the implant micromotion and the evolution of osseointegra-tion.” has been added.

In the third paragraph of the Discussion Section, the term “stability” has been replaced by the term “micromotion”.

In the Conclusions Section, “the implant micromotion” has been added.

- As far as the English language is concerned, it can be improved. The paper is written sometimes too simply with repetitive terms which could be replaced by appropriate synonyms. A wider and richer use of vocabulary is thus recommended. Please also correct the following sentences:

Abstract

- however, inter- and intra- rater reliability and agreement of some devices remains unknown.

- by 2 different operators

Introduction

- and ITV only assess conditions at the time of implant installation

- Moreover, ITV and ISQ values shows an inverse correlation with implant micromotion,

- The industry has provided different devices for measuring ISQ.

- To the best of our knowledge, inter- and intra- rater reliability and agreement of Osstell® Beacon, Penguin® and MegaISQ® to the others remains unknown.

Section 2

- ...., a minimum of 20 samples were necessary on each group to identify a statistically significant difference greater than or equal than to 2 units.

- Based on a recent study [14], standard deviation was assumed to be 5 and the correlation coefficient measurements was 0.9.

- All procedures were repeated by 2 different experienced and calibrated operators

- Test–retest was used to calculate the intraclass correlation coefficient (ICC) using a mixed model with a random effect for the individual for assessing the intra- and inter-rater reliability.

Section 4

- The reliability of Penguin® was good when measuring in parallel, and moderate to good reliability when measuring perpendicular.

- Another study with high ICC scores for Osstell®, reported a good reliability for both Penguin® and Osstell®

- This observation suggest that the values obtained from these devices can be operator dependent.

- Osstell® Beacon showed lower values that previous studies using different types of Osstell®

- In our study, the highest ISQ values were obtained using Penguin® and the lowest values were using MegaISQ®.

- However, the difference with the mean was 2 times lower with MegaISQ® than with Penguin, then the underestimation of MegaISQ® has not the magnitude of Penguin® overestimation.

- Other limitation was the manual tightening of the transducers, but this technique was reported to be objective and reliable previously

Response: Thank you for your suggestions. The text has been revised with a native English editor, the sentences suggested by the reviewer were corrected and several changes have been performed in the paper.

Author Response File: Author Response.docx

Reviewer 2 Report

The article is an interesting comparison of different ISQ measurement methods. Such a publication seems necessary.

However there are some major issues:

1. Introduction

- it is too short and does not introduce the reader to the subject of methods of measuring the stabilization of implants

-  how the ITV and ISQ values are measured should be discussed in more detail

2. Materials and Methods

• the choice of the animal model is the choice of the authors. Although a model would be more appropriate in a clinical setting, the use of a cadaver animal model seems sufficient in this situation, but it should be discussed as a study limitation.

3. Results / Discussion

What is the definition of reliability according to the authors?

Based on the tests performed, it cannot be determined that Osstell® Beacon and MegaISQ® exhibited lower reliability than Penguin®. The devices were tested against each other rather than against a validated standard setting control device. At most it can be concluded that Osstell® Beacon and MegaISQ® in the measurements showed a larger deviation in the measurements.

I encourage the authors to continue their research in this area, however, on a larger group and under different conditions.

Author Response

Reviewer 2

Thank you for your comments in order to improve our manuscript.

Please see the response to your comments below.

The article is an interesting comparison of different ISQ measurement methods. Such a publication seems necessary.

However, there are some major issues: 

1. Introduction

 - it is too short and does not introduce the reader to the subject of methods of measuring the stabilization of implants

Response: We agree with the reviewer. Some information regarding the methods of measuring implant stability and two new references have been added.

Text Change: In the Introduction section, the sentences “Several existing methods have addressed measuring implant stability, including theoretical [6] and experimental modal analysis [7]. Among these, Periotest, insertion torque value (ITV), and implant stability quotient (ISQ) using resonance frequency analysis [8] are the ones widely used clinically.” and “Periotest is a damping method that requires to strike the implant abutment [9], ITV measures the newton centimeters used to screw the implant into the bone [5], and ISQ sensors register the response of the electromagnetic stimu-lation of an abutment fixed to the implant called transducer peg, measuring the implant stability [10].” have been modified.

The sentence “Periotest and ISQ are considered modal analysis methods based on the displacement signal secondary to an external impulse force.” has been added.

The paragraph “Successfully integrated implants involve low implant micromotion levels, which usually correspond to low Periotest values and high ITV and ISQ values. These ITV and ISQ values are inversely correlated with low implant micromotion. However, the relationship between ITV and implant micromotion becomes exponential for higher ITV values [5]. Besides, ITV only measures implant stability at the moment of the insertion [8]. For these reasons, ISQ is usually the preferred method to measure implant stability.” has been rewritten.

-  how the ITV and ISQ values are measured should be discussed in more detail

 Response: The authors have expanded the Introduction section and this information has been included.

Text Change: The sentence “Periotest is a damping method that requires to strike the implant abutment [9], ITV measures the newton centime-ters used to screw the implant into the bone [5], and ISQ sensors register the response of the electromagnetic stimulation of an abutment fixed to the implant called transducer peg, measuring the implant stability [10]” has been modified.

1. Materials and Methods

• the choice of the animal model is the choice of the authors. Although a model would be more appropriate in a clinical setting, the use of a cadaver animal model seems sufficient in this situation, but it should be discussed as a study limitation.

Response:  Thank you for your suggestion. This limitation has been added to the Discussion Section.

Text Change: In the Discussion section, the sentence “Finally, it was not possible to blind the investigators within the instrument used, and the study was performed in an animal model. Therefore, further research is needed to clinically assess in vivo the behaviour of these devices.” has been modified in order to include this limitation.

1. Results / Discussion

What is the definition of reliability according to the authors?

Based on the tests performed, it cannot be determined that Osstell® Beacon and MegaISQ® exhibited lower reliability than Penguin®. The devices were tested against each other rather than against a validated standard setting control device. At most it can be concluded that Osstell® Beacon and MegaISQ® in the measurements showed a larger deviation in the measurements.

 Response:  Thank you for your comment. Our definition of instrument reliability is the degree to which an instrument yields consistent results. Inter- and Intra- rater reliability are common measures used to test internal consistency of an instrument.

We agree with the reviewer that the devices were not tested against a validated gold standard. In this study, the Penguin device was used as control due to the reliability provided by previous studies, as stated in the Materials and methods section.

According to the reviewer comment, the first sentence of the Conclusions section has been modified.

Text change: Within the limitations of this study, Osstell® Beacon and MegaISQ® showed a larger deviation in the measurements than Penguin®

I encourage the authors to continue their research in this area, however, on a larger group and under different conditions.

Author Response File: Author Response.docx

Reviewer 3 Report

Introduction and discussion should be improved in terms of language editing; 

Please perform Bland Altman plotting for each outcome and test and present the results, then improve discussion accordingly 

Table 2 should be a supplemental table 

Author Response

Reviewer 3

Introduction and discussion should be improved in terms of language editing; 

Response: Thank you for your suggestion. Several changes have been performed in the Introduction and Discussion sections in order to improve language errors.

Please perform Bland Altman plotting for each outcome and test and present the results, then improve discussion accordingly 

Response: The Bland-Altman plot has been performed for each outcome and included in the results section as a Figure. Moreover, a new paragraph has been written in the Results Section and the Discussion section has been modified including details of the Bland-Altman plot.

Text Changes: The paragraph “The difference between the values obtained with each device and technique and the mean ISQ values is shown in Figure 2. The mean of these differences was 0.13 (95% CI: -5.79 to 6.05) for the Osstell® Beacon in a parallel measurement; -1.45 (95% CI: -7.78 to 4.88) for the Osstell® Beacon in a perpendicular measurement; 4.2 (95% CI: -6.20 to 14.60) for the Penguin® in a parallel measurement; 1.03 (95% CI: -7.40 to 9.47) for the Penguin® in a perpendicular measurement; -2.02 (95% CI: -7.30 to 3.27) for the MegaISQ® in a parallel measurement and -1.09 (95% CI: -13.03 to 9.23) for MegaISQ® in a perpendicular measurement.” has been added.

The sentence “However, the same instrument obtained the lowest correlation when measuring perpendicular (lowest Pearson correlation coefficient and widest difference of agreement with the mean ISQ values in the Bland-Altman plot).” has been modified including information obtained from the Bland-Altman plot.

The sentence “On the other hand, Osstell® Beacon obtained good correlation for measuring both parallel and perpendicular (and the narrowest difference of agreement with the mean ISQ values), and Penguin® had similar correlation values with the other methods measuring parallel and perpendicular.” has been modified including this information.

Table 2 should be a supplemental table 

Response: Authors agree with the reviewer. This Table has been deleted from the main manuscript and will be sent as a supplementary material.

Author Response File: Author Response.docx

Round 2

Reviewer 2 Report

The manuscript has been improved, however I would suggest to change the title by addding:

"ex vivo animal study"

so every reader would know that this was not clinical.

Or

Reliability and validity of three instruments for measuring Implant Stability Quotient in the animal ex vivo model.    

Author Response

Reviewer 2

The manuscript has been improved, however I would suggest to change the title by addding:

"ex vivo animal study"

so every reader would know that this was not clinical.

Or

Reliability and validity of three instruments for measuring Implant Stability Quotient in the animal ex vivo model.    

Response: Thank you for your suggestion. It has been added.

Text Change: The tittle has been changed. “Reliability and agreement of three devices for measuring Implant Stability Quotient in the animal ex vivo model.”