Performance of Qualitative and Quantitative Antigen Tests for SARS-CoV-2 Using Saliva

Round 1

Reviewer 1 Report

The authors described and utilized a previously developed rapid detection method of SARS-CoV-2, CLEIA, in comparison to ICA and PCR detection, and suggested use of this detection method in certain situations where quick results are required.

1) In materials and methods, the authors did not described how they screened samples. Are these previously not tested in any of authors' publications?

2) Fig. 1A, 1B, 1C do not show 34 cases. Are these all 34 cases included?

3) Figure S2 does not match with Fig.2 at all. In Fig.2, there are 8 cases in between 0.5-1 pg/ml, but only 2 cases in Fig. S2. Between 0.1-0.5, there are 68 cases. The authors need to check all numbers and cases again.

4) The authors discussed two cases with high antigen concentration retested by CLEIA. Authors should rule out if the PCR results are false negative or not in this study. 

5) How was the cut off determined?

6) "ICA" and "CLEIA" need to be included in Keywords.

7) In Fig. 2, Y-axis would be better with “the number of cases” than “frequency”.

8) Delete lines 118-121. These sentences are from the template.

9) Line 101, “in larger cohort” to “in a larger cohort” or “in larger cohort studies”.

Author Response

Reviewer 1: The authors described and utilized a previously developed rapid detection method of SARS-CoV-2, CLEIA, in comparison to ICA and PCR detection, and suggested use of this detection method in certain situations where quick results are required.

1) In materials and methods, the authors did not described how they screened samples. Are these previously not tested in any of authors' publications?

- We used 343 samples that were the remainder of each sample after used for PCR testing at our hospital and frozen at -80 degree. Among these, 78 samples were used in our previous study (Ref 8) and the remaining 265 samples were collected after that study. This information has been added on lines 35-38.

2) Fig. 1A, 1B, 1C do not show 34 cases. Are these all 34 cases included?

- We checked again and confirmed all 34 samples were included. We have replaced the original figure with a new figure with higher resolution.

3) Figure S2 does not match with Fig.2 at all. In Fig.2, there are 8 cases in between 0.5-1 pg/ml, but only 2 cases in Fig. S2. Between 0.1-0.5, there are 68 cases. The authors need to check all numbers and cases again.

- Thank you for your careful pickup. Log-transformed axis was made midpoints to be 3, that is, 0.01-0.03, 0.04-0.10, 0.11-0.30, 0.31-1.00, and so on. We have redrawn Figure 2 by removing 0.03 and 0.30 midpoints.

4) The authors discussed two cases with high antigen concentration retested by CLEIA. Authors should rule out if the PCR results are false negative or not in this study. 

- There are two possibilities: false-negative PCR or true-negative PCR with false-positive CLEIA. Unfortunately, there were no residual samples and we were not able to determine which was correct.

5) How was the cut off determined?

- The cut-off value for CLEIA was according to the manufacturer’s instruction. This information has been added on line 49.

6) "ICA" and "CLEIA" need to be included in Keywords.

- We have added these in Keywords.

7) In Fig. 2, Y-axis would be better with “the number of cases” than “frequency”.

- We have revised accordingly.

8) Delete lines 118-121. These sentences are from the template.

- We have deleted.

9) Line 101, “in larger cohort” to “in a larger cohort” or “in larger cohort studies”.

-We have revised accordingly on line 110.

Reviewer 2 Report

Summary:

The manuscript by Yokota et al entitled, “Performance of qualitative and quantitative antigen tests for SARS-CoV-2 using saliva” describes a study that aims to determine the usability of two antigen-based tests for SARS-CoV-2 from saliva samples and nasopharyngeal swabs. The investigators compared the test results of the antigen-based test against a quantitative RT-PCR assay on the same samples that were tested. The results show that the chromogenic antigen-based test (ICA) was less sensitive than the chemiluminescence antigen-based assay (CLEIA). Specifically, ICA poorly detected PCR-positive samples that were collected beyond 10 days-post symptoms onset. These results suggest that ICA is best used only a few days after symptom onset. On the other hand, CLEIA appears to have utility at a wider range of time points after symptoms onset. Additionally, good correlation between nasopharyngeal swabs and saliva samples was also observed in CLEIA assay, thereby suggesting that saliva can be used in lieu of nasopharyngeal swabs.

Major comments:

• It seems that there is some background from PCR-negative samples for CLEIA. Indeed, in Figure 2, at antigen concentrations between 0.1-1, most of the samples are PCR negative. Is there a cutoff a value to determine if a sample is positive or negative by CLEIA that is recommended by the manufacturer or otherwise?
• Please consider calculating sensitivity and specificity for the ICA and CLEIA test using the PCR results as a benchmark. I think this is especially useful for the CLEIA results which appears to have false positives.
• Was the ICA test also performed on the negative samples? Were there false positives with the ICA assay as well?
• Can you also comment on the effect of one freeze-thaw cycle of the samples? Are the original PCR results or Ct values available? I can imagine that the antigen test can be more sensitive if the sample is fresh and not frozen. Moreover, as I understand it, most antigen tests are performed on fresh samples without a freeze-thaw cycle.

Minor comments:

Line 30: Please make sure to define acronyms for each section

Line 28: “instructive” to “inclusive” (if indeed this is the intended definition)

Lines 106-107: Please consider using a more speculative statement and use “unlikely infectious” rather than “not to be contagious”. Moreover, the studies that were cited in this statement performed infectivity in vitro assays on cells and thus, do not strictly test contagiousness or transmission between individuals. Indeed, infectivity and contagiousness are related but distinct properties of a virus.

Discussion: Can you also discuss costs and time it takes for each antigen test? I think this information would be useful for someone is who deciding which test would be the best to use in their situation.

Figure 2. Please specify what types of samples were included in the histogram (saliva vs NPS)

Supplementary: Figure S1 and S2 are tables rather than figures.

Author Response

Reviewer 2: The manuscript by Yokota et al entitled, “Performance of qualitative and quantitative antigen tests for SARS-CoV-2 using saliva” describes a study that aims to determine the usability of two antigen-based tests for SARS-CoV-2 from saliva samples and nasopharyngeal swabs. The investigators compared the test results of the antigen-based test against a quantitative RT-PCR assay on the same samples that were tested. The results show that the chromogenic antigen-based test (ICA) was less sensitive than the chemiluminescence antigen-based assay (CLEIA). Specifically, ICA poorly detected PCR-positive samples that were collected beyond 10 days-post symptoms onset. These results suggest that ICA is best used only a few days after symptom onset. On the other hand, CLEIA appears to have utility at a wider range of time points after symptoms onset. Additionally, good correlation between nasopharyngeal swabs and saliva samples was also observed in CLEIA assay, thereby suggesting that saliva can be used in lieu of nasopharyngeal swabs.

Major comments:

It seems that there is some background from PCR-negative samples for CLEIA. Indeed, in Figure 2, at antigen concentrations between 0.1-1, most of the samples are PCR negative. Is there a cutoff a value to determine if a sample is positive or negative by CLEIA that is recommended by the manufacturer or otherwise?

• The cut-off value for CLEIA was determined according to the manufacturer’s instruction. This information has been added on line 49.
•  

Please consider calculating sensitivity and specificity for the ICA and CLEIA test using the PCR results as a benchmark. I think this is especially useful for the CLEIA results which appears to have false positives.

• We have revised the sentence as “On the other hand, in PCR-positive samples CLEIA yielded 91% (95%CI: 76–98%) positivity, with 82% (95%CI: 57–96%) positivity in saliva and 100% (95%CI: 80–100%) positivity in NPS. CLEIA yielded 99.4% (95%CI: 97.7–99.9%) negativity in PCR-negative samples” on lines 65-68. Here, if the term “specificity” was used with the PCR results as a benchmark, it could regard two PCR-negative cases with high antigen concentrations in CLEIA as CLEIA false-positive. However, we were not able to determine PCR or CLEIA which was correct in these two cases. Therefore, to avoid misleading we instead used the term “positivity” and “negativity”.

Was the ICA test also performed on the negative samples? Were there false positives with the ICA assay as well?

• No, ICA was tested only in PCR-positive samples and we were not able to assess false-positivity in PCR-negative samples. We have clarified this on line 45.

Can you also comment on the effect of one freeze-thaw cycle of the samples? Are the original PCR results or Ct values available? I can imagine that the antigen test can be more sensitive if the sample is fresh and not frozen. Moreover, as I understand it, most antigen tests are performed on fresh samples without a freeze-thaw cycle.

• In our previous study, we evaluated effects of freezing and thawing of the samples on PCR testing. Ct values did not change significantly after the freeze–thaw steps (Fukumoto, Int J Inf Dis 98, 16-17, 2020). We also evaluated effects of freeze-thaw on antigen and culture tests (Oguri, J Infection 83, 119-, 2021). Freeze-thaw reduced titers by approximately 25%. This has been discussed on page 110-114.

Minor comments:

Line 30: Please make sure to define acronyms for each section

• We did it accordingly. Thank you.

Line 28: “instructive” to “inclusive” (if indeed this is the intended definition)

• We did it accordingly.

Lines 106-107: Please consider using a more speculative statement and use “unlikely infectious” rather than “not to be contagious”. Moreover, the studies that were cited in this statement performed infectivity in vitro assays on cells and thus, do not strictly test contagiousness or transmission between individuals. Indeed, infectivity and contagiousness are related but distinct properties of a virus.

• We have revised accordingly.

Discussion: Can you also discuss costs and time it takes for each antigen test? I think this information would be useful for someone is who deciding which test would be the best to use in their situation.

• We agree, however, unfortunately due to word limit, we were not able to address this issue here.

Figure 2. Please specify what types of samples were included in the histogram (saliva vs NPS).

• We have revised Figure 2 to separate saliva and NPS. As stated in the method, all PCR negative samples were saliva.

Supplementary: Figure S1 and S2 are tables rather than figures.

• We have revised accordingly.

Reviewer 3 Report

Please note the attached pdf.

Comments for author File: Comments.pdf

Author Response

Reviewer 3: In their manuscript „Performance of qualitative and quantitative antigen tests for SARS-CoV-2 using saliva“ Yokota et al. report on varying sensitivity in ICA and CLEIA based approaches on detecting SARS-CoV-2 antigen from PCR confirmed nasopharyngeal swab and saliva samples. Depending on time of sampling aftersymptom onset the authors describe different rates of positivity. The authors present an argument for using ICA early after symptom onset and conclude that CLEIA yields higher overall sensitivity. The manuscript outlines and discusses on favorable scenarios using ICA and CLEIA for detecting SARS-CoV-2 though being limited by the small sample size. Some revisions are required before accepting it for publication.

Major − Paragraph 1 (Introduction) Lines 28–29: Describing all PCR testing as “time-consuming” is not addressing the existence of rapid point-of-care real-time PCR with broadly comparable sensitivity and specificity to more conventional PCR [1]. I suggest rephrasing the sentence with more emphasis put on ease of use of antigen-based test like ICA and CLEIA.

• The reviewer was correct. We have rephrased the sentence not to focus on PCR as suggested on lines 30-31.

− Paragraph 2 (Materials and Methods) line 33 and the following: Given the limited sample size more information on tested patients (inpatients vs. outpatients) and location of testing (on one or more sites) should be provided. This can help to interpret heterogeneity and help to classify the results in a broader clinical context. If necessary, resulting limitations should be discussed.

• PCR-positive samples were taken inpatients, while PCR-negative samples were taken outpatients. Testing was performed at one site in our hospital. This information has been added on lines 38-40.

− Lines 34–36: No information is given on the type of PCR-device used to confirm positivity for SARS-CoV-2-RNA. It is especially of interest if multiple devices were used and therefore potentially introduce heterogeneity. Notable differences in specificity and sensitivity rates could pose limitations, while also add merit to the discussion of false negative PCR results discussed in the latter (Paragraph 4 “Discussion” lines 110–113).

• All PCR analysis was conducted according to the manual for the Detection of Pathogen 2019-nCoV Ver.2.9.1 and using the same PCR machine (line 42).

− Paragraph 4 (Discussion) Lines 110–113: A citation outlining the likelihood of false negative PCR is missing here. For example pooled analysis of literature by Kucirka et al. [2] shows correlation of false negative rates depending on the time PCR-testing was done after symptom onset.

• We have referenced this paper (ref 14) on line 126.

Minor

− Line 16 (abstract): “positivity” should be linked to another noun, for example “positivity rates”

• We have revised accordingly.

− Paragraph 1 (Introduction) Line 30: Please use the comparative form “quicklier” rather than “quickly.”

• We did it.

− Paragraph 2 (Material and Methods) Line 34: The use of “remaining” in this sentence is implying that not all PCR tested samples where available for ICA and CLEIA in the latter. Using something like: “the remainder of each sample” would be less misleading.

• We have rephrased accordingly on lines 35-.

− Paragraph 3 (Results) Lines 50–51: The phrase: “the virus was positive … in saliva …” does not make sense in its current form. Something like “positive results for the virus…” is more consistent.

• We have rephrased accordingly on line 57.

− Line 73: Please use ‘are’ instead of ‘were’ here.

• We changed it.

− Paragraph 4 (Discussion) Lines 106–107: The use of semicolon in this sentence is deterring from readability. I suggest formulating two separate sentences.

• We changed it.

− Figure 1: Is a higher resolution of this figure available? Though not hindering readability it comes off as blurred.

• Figure 1 was replaced by a higher resolution one.

Round 2

Reviewer 1 Report

The manuscript is much improved and most of the comments have been addressed. There are some minor comments.

1. Figure 1 has the samples overlapped due to log scale of the Y-axis. It is confusing and it would be better if the authors can mark the overlapped samples (For example, In Fig.1C, there are 4 samples in day 7, but it looks like there are only 3 samples).
2. Lines 79 and 122:  307 PCR-negative samples to 309.
3. How was it standardized to determine the cutoff score? It is very important when a diagnostic test is developed. It should not be just following the manual.

Author Response

Please see the attachment

Author Response File: Author Response.pdf