Effect of Patient Clinical Variables in Osteoporosis Classification Using Hip X-rays in Deep Learning Analysis

Round 1

Reviewer 1 Report

The authors present a well written and conducted study on osteoporosis diagnosis using an ML approach. The manuscript is well written and the methods are well reported. However, I think several points need further improvement:

1. Please be clearer with your aim statement. Also, aiming for a `statistical significant difference` is usually to be avoided. You maybe expect your model to perform better in diagnosing etc... And the tool you use for that is statistics (sign difference or not..)
2. Methods reporting: check Smets et al. JBMR 2021 - in ML it is always helpful following a standard way in reporting methods.
3. Data preprocessing: please comment on the quality issues that might raise from using photoshop in the process. 
4. Data preprocessing: It is unclear whether all 6 ortho surgeons went through each image, or they were divided in groups of two or... Please be give more details about that.
5. Figure 1: Was the exact same ROI used in each image, so the ROI was standard and positioned adequately in each image, or a different ROI was selected in each image? To be reported as an information.
6. Discussion: please report the sample size as a limitation. Please comment on the conceptual difference between using a complicated model instead of the clinical intuition to take into consideration the age, sex and BMI of the patient, for a clinician in his routine. This investigates of how such approach might improve the osteoporosis diagnosis by using very basic patient`s information - and its value lies in the fact that such basic model might be used to elaborate on more complex ones by future studies (to be pointed out in the discussion).

Author Response

Responses to Reviewers’ Comments

Thank you very much for your invaluable comments and kind acceptance. We have incorporated all the reviewers’ comments and suggestions into our manuscript; the corresponding changes are highlighted in red font in the revised manuscript.

We would like to say thank you once again for the  suggestions, which were very helpful in further improving our manuscript.

Comments from Reviewers and Responses

Reviewer 1

The authors present a well written and conducted study on osteoporosis diagnosis using an ML approach. The manuscript is well written and the methods are well reported. However, I think several points need further improvement:

Comment 1) Please be clearer with your aim statement. Also, aiming for a `statistical significant difference` is usually to be avoided. You maybe expect your model to perform better in diagnosing etc... And the tool you use for that is statistics (sign difference or not..)

Response:

We thank you for this helpful comment. The aim of our research is to evaluate the usefulness of the ensemble model by statistical evaluation methods. As you pointed out, the manuscript has been revised as follows.

Line 65: We aimed to statistically assess the diagnostic ability of osteoporosis using DL with hip radiographs alone and in combination with patient variables.

→We aimed to compare the diagnostic ability of osteoporosis using DL with hip radiographs alone and in combination with patient variables.

Line 68: Such statistically significant difference would clarify the importance of adding patient variables and contribute to the future development of AI diagnostic research in osteoporosis.

→Such significant difference would clarify the importance of adding patient variables and contribute to the future development of AI diagnostic research in osteoporosis.

Comment 2) Reviewer1: Methods reporting: check Smets et al. JBMR 2021 - in ML it is always helpful following a standard way in reporting methods.

Response:

We thank you for this helpful comment. According to Smets et al. JBMR 2021, when evaluated objectively, our previous paper evaluated that the following two points were deficient.

1) comparison with appropriate statistics; The performance comparison between baseline (if used) and proposed models is clearly presented with confidence intervals or appropriate statistical significance

→In our research, we strengthened the statistical points and repeated the examination.

2) Reproducibility and transparency; A link to the model development code, the final developed model or the data have been reported allowing replication.

→We will refrain from publishing the source code for this study for now because it is necessary to discuss with the Intellectual Property Department.

Comment 3) Reviewer1: Data preprocessing: please comment on the quality issues that might raise from using photoshop in the process.

Response:

Thank you for this helpful comment. In this study, we used Photoshop to manually crop, but there are slight differences between workers within the range of the crop. In order to develop a better osteoporosis detection model, it is necessary to further study the range of crop, the difference in resizing, and the processing of padding. As a final goal, it will be necessary to develop and study a method for automatically cutting out from the hip XP. We added this point to the restrictions of the manuscript.

Comment 4) Reviewer1: Data preprocessing: It is unclear whether all 6 ortho surgeons went through each image, or they were divided in groups of two or... Please be give more details about that.

Response:

Thank you for this helpful comment. In this study, the six orthopedic surgeons have cropped one hip digital image each under the director of the orthopedic experts. We fixed the manuscript.

Comment 5) Reviewer1: Figure 1: Was the exact same ROI used in each image, so the ROI was standard and positioned adequately in each image, or a different ROI was selected in each image? To be reported as an information.

Response:

Thank you for this helpful suggestion. In this study, different ROIs have been selected for each image. This description has been added.

Comment 6) Reviewer1:  Discussion: please report the sample size as a limitation. Please comment on the conceptual difference between using a complicated model instead of the clinical intuition to take into consideration the age, sex and BMI of the patient, for a clinician in his routine. This investigates of how such approach might improve the osteoporosis diagnosis by using very basic patient`s information - and its value lies in the fact that such basic model might be used to elaborate on more complex ones by future studies (to be pointed out in the discussion).

Response:

Thank you for this helpful comment. The following sentences have been added as a limit to sample size. Please check.

“Fourth, we could not consider sample size in the method because previous studies did not report effect size or clinical importance difference. In this study, we reported each effect size on each performance metric. Therefore, researchers in the further research can conduct sample size calculation.”

Reviewer 2 Report

Dear authors

Congratulation to accomplish this great work. It's impressive and showed great clinical impact on this field. Hip plain films ensemble clinical factors to predict osteoporosis with deep leaning. 

I have some comments about this article.

1. Lack of external validation might limit the impact of this manuscript because high possibility of overfitting of authors' institutional data which might not be applied to other places, please comment this

2. You should present the hyperparameters used for training the neural network to make the follow researchers to repeat this study.

3. No regularization techniques described which is used to prevent overfitting of training data. Please comment it.

4. One paradox in deep learnings for analyzing medical images is the black box mechanism. The model may use another part of the image rather than the true lesion site to produce the answer. Therefore, the authors should offer more evidence to indicate the finding of the algorithm is really related to hip osteoporosis. Visualization is a choice but something else is welcome.

Thank you again for the opportunity to review this manuscript.

Best,

Author Response

Responses to Reviewers’ Comments

Thank you very much for your invaluable comments and kind acceptance. We have incorporated all the reviewers’ comments and suggestions into our manuscript; the corresponding changes are highlighted in red font in the revised manuscript.

We would like to say thank you once again for the suggestions, which were very helpful in further improving our manuscript.

Comments from Reviewers and Responses

Reviewer 2

Congratulation to accomplish this great work. It's impressive and showed great clinical impact on this field. Hip plain films ensemble clinical factors to predict osteoporosis with deep leaning.

I have some comments about this article.

Comment 1) Lack of external validation might limit the impact of this manuscript because high possibility of overfitting of authors' institutional data which might not be applied to other places, please comment this

Response:

We thank you for this helpful comment. As you pointed out, you need to increase the external validity to create a more robust prediction model. In terms of that point, we added to the part of discussion limitation.

Comment 2) : You should present the hyperparameters used for training the neural network to make the follow researchers to repeat this study.

Response:

We thank you for this helpful comment. Hyper parameters are described in 177 lines to 181 lines. We fixed it.

Comment 3): No regularization techniques described which is used to prevent overfitting of training data. Please comment it.

Response:

Thank you for this helpful comment.

The following method was adopted for measures to prevent overfitting of training data in our study. We added it to the manuscript. Please check them.

• The training algorithm used k = 4 for k-fold cross-validation to avoid overfitting and bias and to minimize the generalization error.
• Early stopping methods were adopted to prevent overfitting.

Comment 4): One paradox in deep learnings for analyzing medical images is the black box mechanism. The model may use another part of the image rather than the true lesion site to produce the answer. Therefore, the authors should offer more evidence to indicate the finding of the algorithm is really related to hip osteoporosis. Visualization is a choice but something else is welcome.

Response:

Thank you for this helpful comment. The purpose of our study was to statistically evaluate the effectiveness of the ensemble model. Ensemble algorithm construction and validity and visualization of feature areas by deep learning are conducted in our previous paper, so please check.

Yamamoto, N.; Sukegawa, S.; Kitamura, A.; Goto, R.; Noda, T.; Nakano, K.; Takabatake, K.; Kawai, H.; Nagatsuka, H.; Kawasaki, K.; Furuki, Y.; Ozaki, T. Deep Learning for Osteoporosis Classification Using Hip Radiographs and Patient Clinical Covariates. Biomolecules 2020, 10 (11), 1–13.  doi: 10.3390/biom10111534