Next Article in Journal
Host Response of Syrian Hamster to SARS-CoV-2 Infection including Differences with Humans and between Sexes
Previous Article in Journal
Development and Evaluation of Bacteriophage Cocktail to Eradicate Biofilms Formed by an Extensively Drug-Resistant (XDR) Pseudomonas aeruginosa
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Validation of a SARS-CoV-2 Surrogate Virus Neutralization Test in Recovered and Vaccinated Healthcare Workers

by
Lina Mouna
1,*,†,
Mehdi Razazian
1,2,†,
Sandra Duquesne
1,
Anne-Marie Roque-Afonso
1 and
Christelle Vauloup-Fellous
1
1
Virology Department, Hôpital Paul Brousse, INSERM U1193, AP-HP, Université Paris Saclay, 94804 Villejuif, France
2
Institute for Physiology and Pathophysiology, Johannes Kepler University Linz, 4040 Linz, Austria
*
Author to whom correspondence should be addressed.
These authors contributed equally to this work.
Submission received: 13 October 2022 / Revised: 3 January 2023 / Accepted: 28 January 2023 / Published: 2 February 2023
(This article belongs to the Section SARS-CoV-2 and COVID-19)

Abstract

:
Vaccination against COVID-19 is the main public health approach to fight against the pandemic. The Spike (S) glycoprotein of SARS-CoV-2 is the principal target of the neutralizing humoral response. We evaluated the analytical and clinical performances of a surrogate virus neutralization test (sVNT) compared to conventional neutralization tests (cVNTs) and anti-S eCLIA assays in recovered and/or vaccinated healthcare workers. Our results indicate that sVNTs displayed high specificity and no cross-reactivity. Both eCLIA and sVNT immunoassays were good at identifying cVNT serum dilutions ≥1:16. The optimal thresholds when identifying cVNT titers ≥1:16, were 74.5 U/mL and 49.4 IU/mL for anti-S eCLIA and sVNT, respectively. Our data show that neutralizing antibody titers (Nab) differ from one individual to another and may diminish over time. Specific assays such as sVNTs could offer a reliable complementary tool to routine anti-S serological assays.

1. Introduction

The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) epidemic, which was first reported in Wuhan, China, spread rapidly worldwide and caused the COVID-19 pandemic [1]. As different vaccines become available, a clearer understanding of their efficacy in inducing sufficient public immunity against COVID-19 is of major importance [2]. The spike (S) glycoprotein of SARS-CoV-2 is the principal target of the neutralizing humoral response and is therefore used for the development of vaccines and monoclonal antibody (Ab) treatments [3,4,5,6,7]. Virus-specific neutralizing Abs (NAbs) play a key role in reducing viral replication and accelerating the viral clearance of SARS-CoV-2 [8,9]. NAbs mainly act against the receptor-binding domain (RBD) of the SARS-CoV-2 S glycoprotein, effectively blocking viral entry by preventing its binding to the angiotensin-converting enzyme 2 (ACE-2) receptor [9,10,11,12].
There is some evidence that immune protection against other human coronaviruses (HCoV) is not long-lasting, but it is also suggested that Abs against SARS-CoV-1 persist for years and display potent neutralizing activity, even in the absence of detectable IgG directed against the SARS-CoV-1 nucleocapsid protein (N) [13,14]. However, recent descriptions of SARS-CoV-2 reinfections have supported recommendations to offer vaccination to highly exposed individuals such as healthcare workers (HCW) who have recovered from a mild form of COVID-19 [15,16]. These authors also suggested that vaccination against SARS-CoV-2 will probably have a short-lived protective effect, meaning that most people should be vaccinated periodically. Until now, most serological studies have focused on hospitalized patients, but specific information on serologic responses in individuals with mild infection remains scarce and is mainly focused on seroconversion rates [17,18,19,20,21,22,23,24]. Marot et al. recently showed that the neutralizing activity of Ab appears to be transient with a decline, or even loss of NAb titers from two months after disease onset, thus supporting the recommendation to vaccinate infected HCW [25]. However, in the context of vaccine shortages and/or a mistrust of vaccines, it might be highly beneficial to determine who really needs a boost and how many doses might be necessary depending on their clinical history together with accurate serological information. Serological testing, especially to detect NAbs, is therefore essential to identify individuals who are potentially at a lower risk of hospitalization or serious COVID-19 infection.
In COVID-19 patients, NAbs can be detected within two weeks of symptom onset, but in some cases this is substantially longer [26,27,28,29]. However, the dynamics of NAbs and their correlation with anti-S Ab have not been explored in COVID-19 patients more than six months after the onset of symptoms. Moreover, it is still unclear whether current serological assays that detect anti-S Ab will predict neutralizing activities or protection against re-infection with the virus [30].
NAbs can be detected using conventional virus neutralization tests (cVNTs), and their presence is often correlated with protective immunity [31]. cVNTs are considered to be the gold standard [21] but they must be performed in a biosafety level-3 laboratory (BSL-3); they are time-consuming, labor-intensive, require several days of work by highly skilled operators and are hardly standardized when compared to other serologic assays. Very few laboratories can therefore run such tests, which is not appropriate for mass use. On the other hand, enzyme-linked immunosorbent assays (ELISA), and ELISA variants such as the lateral flow assay (LFA) and chemiluminescence immunoassay (CLIA) can detect anti-S or anti-RBD Ab with high sensitivity but vary in their ability to predict NAb activity [32,33,34,35,36]; thus, the results may not directly correlate with protection [37,38].
During the present study we evaluated the analytical and clinical performance of a surrogate virus neutralization test (sVNT) (iFlash-2019-nCoV NAb assay, YHLO, China) that had been designed to detect total NAbs in an isotype and species-independent manner and can be completed in 1–2 h in a BSL-2 laboratory [39]. The anti-S eCLIA assay (Elecsys anti-SARS-CoV-2 S, Roche Diagnostics, Switzerland) can detect total binding antibodies but not specifically NAb. Indeed, although RBD might be the target for NAb, some RBD Ab cannot fully block the interaction between RBD and ACE2, which might interfere with re-infection prevention. We compared cVNT, sVNT and anti-S eCLIA assays in immunocompetent subjects (mainly healthcare workers) at 9–11 months after a COVID-19 infection, after the first dose of vaccine, after the second dose of vaccine and in patients with a history of COVID-19 after one dose of vaccine.

2. Materials and Methods

This study was conducted in the Virology Laboratory at Paul Brousse Hospital (Paris, France) in accordance with the requirements of the Declaration of Helsinki as a retrospective non-interventional study with no addition to standard-of-care procedures. The reclassification of biological remnants into research material after completion of the virological tests that had been ordered was registered under number DC 2009-965 and received ethical approval from CPP Ile de France 7 (N°CO-15-000) in accordance with French law.

2.1. Clinical Samples

First of all, 54 sera were selected to determine three thresholds for anti-S eCLIA and sVNT values predictive of neutralizing activity, using a cVNT as the gold standard (Figure 1):
  • 11 serum samples from vaccinated individuals, collected at least 14 days after the first dose (n = 7) or at least 7 days after the second dose (n = 4); BNT162b2 mRNA COVID-19 vaccine;
  • 41 serum samples from recovered COVID-19 patients collected at least 3 months after the onset of infection (n = 36) or at least 6 months after the onset of infection and 7 days after the first vaccine dose (n = 5); BNT162b2 mRNA COVID-19 vaccine;
  • As a control, two samples were collected from unaffected and non-vaccinated individuals.
In a second step, 394 samples were collected to compare the results of the anti-S eCLIA and sVNT assays (Figure 2):
  • 103 serum samples collected before the COVID-19 pandemic (2019) including three serum samples from patients with other HCoV infections (2 HcoV-OC43, 1 HcoV-NL63);
  • 27 serum samples collected at least three weeks after infection from patients with a confirmed SARS-CoV-2 variant (alpha, beta, or delta);
  • 168 serum samples collected from infection-naïve HCW before vaccination (n = 73), at least 14 days after the first dose of vaccine (n = 36) and at least seven days after the second dose of vaccine (n = 59);
  • 96 serum samples from COVID-19 recovered HCW collected 9–11 months after the onset of infection (n = 73) and at least seven days after one dose of vaccine (n = 23).
The median age of the HCW was 53 years (range: 22 to 86 years) and 64% were women. None were immunocompromised. All were vaccinated with the BNT162b2 mRNA COVID-19 vaccine.

2.2. Conventional Virus Neutralization Test (cVNT)

All procedures related to cVNTs were performed in a BSL-3 laboratory in accordance with WHO guidelines. Vero E6 cells (African green monkeys, kidney) were grown at 37 °C under 5% CO2 in Dulbecco’s modified eagle medium (DMEM, 319966), supplemented with L-Glutamine, 10% fetal bovine serum (FBS, FCS, Sigma-Aldrich, #F9665), and 1% penicillin/streptomycin/neomycin (PSN, GIBCO 15640055). The SARS-CoV-2 virus was isolated from a RT-PCR-confirmed COVID-19 patient hospitalized in Paul Brousse Hospital (Villejuif, France) in December 2020, and amplified in Vero cells. We confirmed the virus titer using TCID50 (10 7.5 PFU/mL) and RT PCR (CT: 9.5). Serum samples were decomplemented by heat inactivation (56 °C for 30 min) and subjected to serial two-fold dilution in PBS 1X (1:2 to 1:2048, each dilution 250 µL). Different serum dilutions were incubated with 250 µL diluted virus (103 TCID50/mL) in DMEM with 2% FBS at 37 °C, under 5% CO2 for 1 h, and then 100 µL of each dilution was added to Vero cells seeded (3 × 104/well) in 96-well plates (four technical replicates for each sera dilution; n:4). The plates were incubated at 37 °C under 5% CO2 for 96 h until microscopic examination on day 4 to assess the cytopathic effect (CPE). The NAbs titer was confirmed with the highest dilution of sera that inhibited 100% of the CPE (absence of cytopathic effect). cVNT titers ≥1:16 were considered to be positive for SARS-CoV-2 NAbs.

2.3. Surrogate Virus Neutralization Test (sVNT)

The iFlash-2019-nCoV NAb assay is a paramagnetic particle chemiluminescent one-step immunoassay for the quantitative determination of NAb in serum using an automated analyzer. This assay is based on the Ab-mediated blockage of virus–host interactions between the ACE2 receptor protein and the RBD of the viral S protein. It is designed to mimic the virus–host interaction by direct protein–protein interactions in a test tube. In brief, NAb in the sample reacts with RBD antigen-coated microparticles to form a complex. In a second step, an ACE2 conjugate is added to competitively bind to the RBD-coated particles which have not been neutralized by NAb from the sample. Bound particles are then detected through a chemiluminescent reaction, and a calibration curve enables quantification of the amount of NAb. Results are expressed in IU/mL with the manufacturer providing a cut-off point of 24 IU/mL.

2.4. Elecsys Anti-S eCLIA Assay (Anti-S eCLIA)

The anti-S eCLIA assay was used for the quantitative determination of antibodies to the SARS-CoV-2 S protein receptor-binding domain (RBD). The quantification range of anti-S eCLIA is 0.4 to 250 U/mL (results < 0.4 U/mL are considered to be non-reactive, and anti-S titers outside the quantification range are expressed as >250 U/mL). The manufacturer’s positive cut-off point is 0.4 U/mL.

2.5. Statistical Analysis

Statistical analysis was performed with GraphPad Prism Software 9.1.2 (GraphPad, La Jolla, CA, USA). The Wilson-Brown statistical method with a 95% confidence interval (p value < 0.0001) was performed. A correlation test (Spearman) and receiver operating characteristic curve (ROC) were conducted using Analyse-it software v5.65. Results with p values < 0.05 were considered to be significant.

3. Results

3.1. Correlation of Anti-S eCLIA, sVNT and cVNT Results

Fifty-four samples from naïve, infected, and/or vaccinated patients were used to assess the correlation between anti-S eCLIA, sVNT and cVNT (Figure 1). Overall, 3/54 had no detectable anti-S eCLIA (titer <0.4 U); the remaining samples had titers ranging from 1.21 to > 250 U/mL. sVNTs were performed on 45/54 samples with titers ranging from 0 to 12,377.784 IU/mL. The control samples displayed no neutralizing activity and no reactivity in any immunoassay. cVNTs were performed on these 54 samples with neutralization results that ranged from 0 to 1:1024, and 20/54 (37%) achieved significant neutralization in a dilution ≥1:16.
The anti-S eCLIA and sVNT titers were correlated (Spearman’s rs = 0.761, p < 0.0001). Both the Roche and iFlash immunoassays were able to identify cVNT serum dilutions >1:16, with AUC (area under the curve) values of 0.950 (95% CI 0.889–1.011), and 0.870 (95% CI 0.752–0.988) (p < 0.0001) for anti-S eCLIA and sVNTs, respectively. The immunoassays were equally discriminant. The optimal thresholds to identify cVNT titers ≥ 1:16 were 74.5 U/mL and 49.4 IU/mL for anti-S eCLIA and sVNTs, respectively (so-called Nabs cut-offs). Applying these thresholds, sensitivity and specificity reached 95% and 93.4% for anti-S eCLIA, and 82.4 and 85.7% for sVNTs (Figure 3).

3.2. Performance of sVNT vs. cVNT

The specificity and sensitivity of sVNTs were assessed on sera collected from patients before the COVID-19 pandemic (expected to be negative) and from patients who had recovered following a confirmed COVID-19 infection (expected to be positive). All those expected to be negative were indeed negative (103/103; 100% specificity), including the samples collected from patients with other HCoV infections. Among the 100 sera taken from recovered patients, 91% had detectable titers, including 64/73 (87.7%) samples collected ≥ 9 months after infection, and 27/27 (100%) samples from patients with recent confirmed COVID-19 infection by a variant.

3.3. Anti-S eCLIA vs. sVNT

Seventy-three serum samples were collected before vaccination. All (73/73) were negative in the Roche and sVNT assays with mean titers <0.4 U/mL and 10.82 ± 5 IU/mL, respectively (Table 1).
At least 14 days after the first vaccine dose and before the second dose, 28/36 (78%) of the serum samples displayed positive anti-S eCLIA titers (ranging from 2.88 to >250 U/mL; mean titer: 64.78 ± 90), and 21/36 (58%) were positive with sVNTs (titers ranging from 24.792 to 1188.864 IU/mL; mean titer: 86.26 ± 198). Based on established NAb cut-offs, the number of individuals with potential protective antibody levels fell to 12/36 (33%) with sVNTs and to 10/36 (28%) with anti-S eCLIA. At least seven days after the second dose, 55/59 (93%) displayed positive anti-S eCLIA titers (ranging from 1 to >250 U/Ml; mean titer: 201.58 ± 98)), and 51/59 (86%) were positive with sVNTs (titers ranging from 26.88 to 240,000,000 IU/mL; mean titer: 36,611,419 ± 82,643,261)). Based on established NAb cut-offs, 50/51 (85%) individuals had sVNT titers >50 IU/mL and 47/59 (80%) had an anti-S eCLIA >75 U/mL.
Between 9 and 11 months after recovering from COVID-19 infection, 73 samples were collected from COVID-19 healthcare workers. All were symptomatic with mild symptoms (mainly cough, fever, breathlessness, aching body, headache, diarrhea or anosmia). Overall, 72/73 (99%) displayed the presence of anti-S eCLIA (titers ranging from 10.39 to > 250 U/mL; mean titer: 169 ± 92), and 64/73 (88%) were positive with sVNTs (titers ranging from 25 to 2697 IU/mL; mean titer: 394 ± 755). Based on established NAb cut-offs, the number of individuals with protective antibody levels fell to 42/73 (58%) with sVNTs and to 56/42 (77%) with anti-S eCLIA. Among these recovered COVID-19 HCW, 23 had received one dose of vaccine at least 6 months after the onset of infection. All (23/23; 100%) displayed the presence of anti-S eCLIA (titers ranging from 60.89 to > 250 IU/mL; mean titers: 250 ± 41) and positive sVNTs (titers ranging from 35.712 to 240,000,002.4 IU/mL; mean titer: 31,306,349 ± 85,874,735). Based on established NAb cut-offs, 22/23 (96%) displayed sVNT titers >50 IU/mL and anti-S eCLIA titers >75 U/mL.
In conclusion, anti-S eCLIA Abs levels from hybrid immunized individuals (infected–vaccinated) and two-dose vaccine immunized individuals were higher than those found in other situations (p < 0.03). NAb levels (sVNT) in sera from hybrid immunized individuals (infected–vaccinated), two-dose vaccine immunized individuals and patients who had recovered from variants of SARS-CoV-2 infection were also higher than in the other situations. (p < 0.0003) (Figure 4; Table 1).

4. Discussion

We evaluated analytical and clinical performance of a surrogate virus neutralization test (sVNT) by comparison with a conventional virus neutralization test (cVNT), before studying the neutralizing activity of the serum samples, a biological marker that is often correlated to protective immunity [31]. Both assays can determine the functional ability of Ab to prevent viral binding to the ACE2 receptor. These assays also have the advantage that they can study overall serum neutralizing activity (both isotypes and epitope-specific Ab). However, cVNTs cannot be performed routinely on a large number of samples, unlike the automated sVNT assay. Our results indicated that sVNTs displayed high specificity and no cross-reactivity. We also established that a threshold of 50 IU/mL with sVNTs was predictive of the neutralizing activity of anti-SARS-CoV-2 Ab in cell culture (vs. the 24 IU/mL cutoff point recommended by the manufacturer) with sensitivity and specificity reaching 82.4% and 85.7%, respectively.
It is well known that the positive results of current large-scale serological ELISA or CLIA assays, even those which specifically target the S protein of SARS-CoV-2, do not necessarily indicate neutralizing immunity because they detect some but not all NAb and may also detect non-NAb. Indeed, although RBD might be the target for NAb, some RBD Ab cannot fully block the interaction between RBD and ACE2, which might fail in preventing re-infection [40]. With the anti-S eCLIA assay, we were able to establish a threshold of 75 IU/mL anti-S, which was related to the neutralizing activity of anti-SARS-CoV-2 antibodies in cVNTs with sensitivity and specificity of 95% and 93.4%, respectively. This threshold differs quite markedly from the 0.8 IU/mL cut-off point recommended by the manufacturer and will certainly have an impact on the clinical interpretation of results.
Our understanding of the duration and nature of protective immunity against SARS-CoV-2 is currently very limited. The kinetics of antibody-mediated immunity to SARS-CoV-2 infection, and how long this may last, remain unknown. Working in ferrets, some authors recently showed that re-infected SARS-CoV-2 animals with high Nabs displayed attenuated viral replication and rapid viral clearance. Direct-contact transmission was only observed from re-infected ferrets with low NAb titers, demonstrating a close correlation between NAb titers and protection in an animal model [40]. Although a recent study in a rhesus macaque model did not find any evidence of reinfection and therefore suggested a robust protective immune response when the animals were re-exposed to SARS-CoV-2 one month after the initial viral infection [41], only monkeys with high NAb titers were used for the reinfection studies and a few of them displayed low levels of replication following reinfection. Overall, these studies indicate that the NAb titer is a critical determinant in providing protection against reinfection; initial exposure to other human coronaviruses are known to fail in eliciting a sufficient protective immune response, in some cases with a higher viral load [42]. This result, consistent with our findings and those of other groups, indicates that SARS-CoV-2 infection elicits a neutralizing activity of sera, at least temporarily, which may be correlated with a protective humoral response.
Our data showed that NAb titers in patients may vary and diminish over time depending on different individuals, which is in accordance with findings in patients infected with other human HCoV [43,44]. The short-term humoral immune response in COVID-19 patients is also highly consistent with that observed in patients infected with SARS-CoV-1 and MERS-CoV [45,46] who showed a rapid decrease in virus-specific antibody titers within 3–4 months. Among the 73 recovered patients in our study, 64 (88%) displayed low NAb titers (mean titer: 394 ± 755) 9 to 11 months after infection, suggesting that the Ab titers may diminish over time, or that some recovered patients may not produce a high-titer neutralizing response during SARS-CoV-2 infection. Although a positive correlation between NAb titers and IgG has been shown in patients within three months of infection, our results suggest that more than nine months after infection, anti-S serological tests may fail to find NAb and that specific assays such as a sVNT may offer a valuable complementary tool [47]. However, their response after one booster vaccine (mean titer: 31,306,349.36 ± 85,874,735) revealed equivalent NAb titers compared to naïve patients who received two doses of vaccine (mean titer: 36,611,419.09 ± 82,643,261). This finding encourages the performance of serological tests in the general population in order to identify individuals who might have experienced asymptomatic infection in order to optimize vaccination schedules, as is now recommended in most countries.
As the COVID-19 pandemic continues to spread worldwide, most people infected with SARS-CoV-2 will recover from primary infection. To hamper the continuous spread of this virus, recovered individuals may have sufficient NAb to protect themselves against reinfection. Recent studies have reported that asymptomatic COVID-19 patients exhibit a weaker Ab response than patients with severe disease. Moreover, a rapid decline of Ab in asymptomatic patients has been seen when compared to severe COVID-19 patients [48,49,50]. sVNTs may therefore be more reliable for the assessment of herd immunity and humoral protective immunity in recovered/vaccinated patients. As sVNTs will not be available in all laboratories, its use could be considered for the routine revaluation of serological assays in order to facilitate screening in the general population and target individuals who require one or two doses of vaccine. Further, in patients who are known to have a weak immune response to vaccination (e.g., those undergoing dialysis or receiving rituximab), it would be helpful to identify individuals in whom a third dose of vaccine is warranted.
Although cases of suspected SARS-CoV-2 reinfection have continued to rise among recovered COVID-19 patients, their immune response to the virus, and particularly the role of Nab, has not been fully characterized [51]. sVNTs have been reported recently and they are already commercially available, but its value needs to be analyzed and assessed for each specific context [52]. Our understanding of the duration and nature of protective immunity to SARS is currently very limited. Our data suggest that NAb may be absent in some patients who have recovered from COVID-19 and in these specific individuals, a single booster vaccination will enhance their pre-existing immunity and reach a protective level. Given the current situation where a significant proportion of the population continues to display a degree of mistrust in vaccines, or in the event of vaccine shortages, sVNT assays or routine serological assays with newly established NAb cut-offs would avoid the vaccination of individuals whose titers are indicative of protective immunity. Studies are currently ongoing to evaluate the usefulness of sVNTs in immune patients at risk of severe COVID-19.
The present study had several limitations, such as a lack of systematic follow-up and limited sequential sampling. More studies are necessary to assess the protection correlates after SARS-CoV-2 infection, such as the minimal NAb titer required for protection. Further, it will be pertinent to study the protective role of NAb during reinfection in recovered COVID-19 patients and the role of the cellular immune response. It will also be important to assess changes in the levels of antibodies directed against the S protein and RBD over time, in order to improve our understanding of the potential protective immune response to SARS-CoV-2 infection.

Author Contributions

Conceptualization: C.V.-F., L.M. and A.-M.R.-A.; Methodology: L.M., M.R., C.V.-F. and A.-M.R.-A.; Software: L.M.; Validation: C.V.-F.; Formal analysis: L.M.; Data curation: L.M.; Investigation: L.M. and S.D.; Writing—original draft preparation: C.V.-F. and L.M. Writing—review and editing; Supervision: C.V.-F., L.M. and C.V.-F. Funding acquisition: C.V.-F. and A.-M.R.-A. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding. Thanks to Robert-Debré association for supporting MR’s salary during 6 months.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki. Data collection and processing have received ethical approval from CPP Ile de France 7 (N°CO-15-000) in accordance with French law.

Informed Consent Statement

The reclassification of biological remnants into research material after completion of the virological tests that had been ordered was registered under number DC 2009-965.

Data Availability Statement

Data can be provided by the authors on demand.

Acknowledgments

We are grateful to Rémi THOME and Kevin RICQUIER for helpful technical assistance. YHLO Shenzhen Biotech kindly provided part of reagents.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Huang, C.; Wang, Y.; Li, X.; Ren, L.; Zhao, J.; Hu, Y.; Zhang, L.; Fan, G.; Xu, J.; Gu, X.; et al. Clinical Features of Patients Infected with 2019 Novel Coronavirus in Wuhan, China. Lancet 2020, 395, 497–506. [Google Scholar] [CrossRef] [PubMed]
  2. Piccoli, L.; Park, Y.-J.; Tortorici, M.A.; Czudnochowski, N.; Walls, A.C.; Beltramello, M.; Silacci-Fregni, C.; Pinto, D.; Rosen, L.E.; Bowen, J.E.; et al. Mapping Neutralizing and Immunodominant Sites on the SARS-CoV-2 Spike Receptor-Binding Domain by Structure-Guided High-Resolution Serology. Cell 2020, 183, 1024–1042. [Google Scholar] [CrossRef]
  3. Zhang, H.; Penninger, J.M.; Li, Y.; Zhong, N.; Slutsky, A.S. Angiotensin-Converting Enzyme 2 (ACE2) as a SARS-CoV-2 Receptor: Molecular Mechanisms and Potential Therapeutic Target. Intensive Care Med. 2020, 46, 586–590. [Google Scholar] [CrossRef]
  4. Chen, Y.; Liu, Q.; Guo, D. Emerging Coronaviruses: Genome Structure, Replication, and Pathogenesis. J. Med. Virol. 2020, 92, 418–423. [Google Scholar] [CrossRef]
  5. Wu, A.; Peng, Y.; Huang, B.; Ding, X.; Wang, X.; Niu, P.; Meng, J.; Zhu, Z.; Zhang, Z.; Wang, J.; et al. Genome Composition and Divergence of the Novel Coronavirus (2019-NCoV) Originating in China. Cell Host Microbe 2020, 27, 325–328. [Google Scholar] [CrossRef] [PubMed]
  6. Enjuanes, L.; Zuñiga, S.; Castaño-Rodriguez, C.; Gutierrez-Alvarez, J.; Canton, J.; Sola, I. Molecular Basis of Coronavirus Virulence and Vaccine Development. In Advances in Virus Research; Elsevier: Amsterdam, The Netherlands, 2016; Volume 96, pp. 245–286. ISBN 978-0-12-804736-1. [Google Scholar]
  7. Du, L.; He, Y.; Zhou, Y.; Liu, S.; Zheng, B.-J.; Jiang, S. The Spike Protein of SARS-CoV—A Target for Vaccine and Therapeutic Development. Nat. Rev. Microbiol. 2009, 7, 226–236. [Google Scholar] [CrossRef]
  8. Duan, K.; Liu, B.; Li, C.; Zhang, H.; Yu, T.; Qu, J.; Zhou, M.; Chen, L.; Meng, S.; Hu, Y.; et al. Effectiveness of Convalescent Plasma Therapy in Severe COVID-19 Patients. Proc. Natl. Acad. Sci. USA 2020, 117, 9490–9496. [Google Scholar] [CrossRef] [PubMed]
  9. Shen, C.; Wang, Z.; Zhao, F.; Yang, Y.; Li, J.; Yuan, J.; Wang, F.; Li, D.; Yang, M.; Xing, L.; et al. Treatment of 5 Critically Ill Patients With COVID-19 With Convalescent Plasma. JAMA 2020, 323, 1582–1589. [Google Scholar] [CrossRef]
  10. Iwasaki, A.; Yang, Y. The Potential Danger of Suboptimal Antibody Responses in COVID-19. Nat. Rev. Immunol. 2020, 20, 339–341. [Google Scholar] [CrossRef] [PubMed]
  11. Cao, Y.; Su, B.; Guo, X.; Sun, W.; Deng, Y.; Bao, L.; Zhu, Q.; Zhang, X.; Zheng, Y.; Geng, C.; et al. Potent Neutralizing Antibodies against SARS-CoV-2 Identified by High-Throughput Single-Cell Sequencing of Convalescent Patients’ B Cells. Cell 2020, 182, 73–84. [Google Scholar] [CrossRef]
  12. Wu, Y.; Wang, F.; Shen, C.; Peng, W.; Li, D.; Zhao, C.; Li, Z.; Li, S.; Bi, Y.; Yang, Y.; et al. A Noncompeting Pair of Human Neutralizing Antibodies Block COVID-19 Virus Binding to Its Receptor ACE2. Science 2020, 368, 1274–1278. [Google Scholar] [CrossRef] [PubMed]
  13. Shi, R.; Shan, C.; Duan, X.; Chen, Z.; Liu, P.; Song, J.; Song, T.; Bi, X.; Han, C.; Wu, L.; et al. A Human Neutralizing Antibody Targets the Receptor-Binding Site of SARS-CoV-2. Nature 2020, 584, 120–124. [Google Scholar] [CrossRef]
  14. Lin, Q.; Zhu, L.; Ni, Z.; Meng, H.; You, L. Duration of Serum Neutralizing Antibodies for SARS-CoV-2: Lessons from SARS-CoV Infection. J. Microbiol. Immunol. Infect. 2020, 53, 821–822. [Google Scholar] [CrossRef]
  15. Anderson, D.E.; Tan, C.W.; Chia, W.N.; Young, B.E.; Linster, M.; Low, J.H.; Tan, Y.-J.; Chen, M.I.-C.; Smith, G.J.D.; Leo, Y.S.; et al. Lack of Cross-Neutralization by SARS Patient Sera towards SARS-CoV-2. Emerg. Microbes Infect. 2020, 9, 900–902. [Google Scholar] [CrossRef] [PubMed]
  16. To, K.K.-W.; Hung, I.F.-N.; Ip, J.D.; Chu, A.W.-H.; Chan, W.-M.; Tam, A.R.; Fong, C.H.-Y.; Yuan, S.; Tsoi, H.-W.; Ng, A.C.-K.; et al. Coronavirus Disease 2019 (COVID-19) Re-Infection by a Phylogenetically Distinct Severe Acute Respiratory Syndrome Coronavirus 2 Strain Confirmed by Whole Genome Sequencing. Clin. Infect. Dis. 2020, 73, ciaa1275. [Google Scholar] [CrossRef] [PubMed]
  17. Tillett, R.L.; Sevinsky, J.R.; Hartley, P.D.; Kerwin, H.; Crawford, N.; Gorzalski, A.; Laverdure, C.; Verma, S.C.; Rossetto, C.C.; Jackson, D.; et al. Genomic Evidence for Reinfection with SARS-CoV-2: A Case Study. Lancet Infect. Dis. 2021, 21, 52–58. [Google Scholar] [CrossRef] [PubMed]
  18. Houlihan, C.F.; Vora, N.; Byrne, T.; Lewer, D.; Kelly, G.; Heaney, J.; Gandhi, S.; Spyer, M.J.; Beale, R.; Cherepanov, P.; et al. Pandemic Peak SARS-CoV-2 Infection and Seroconversion Rates in London Frontline Health-Care Workers. Lancet 2020, 396, e6–e7. [Google Scholar] [CrossRef]
  19. Chen, Y.; Tong, X.; Wang, J.; Huang, W.; Yin, S.; Huang, R.; Yang, H.; Chen, Y.; Huang, A.; Liu, Y.; et al. High SARS-CoV-2 Antibody Prevalence among Healthcare Workers Exposed to COVID-19 Patients. J. Infect. 2020, 81, 420–426. [Google Scholar] [CrossRef]
  20. Rudberg, A.-S.; Havervall, S.; Månberg, A.; Falk, A.J.; Aguilera, K.; Ng, H.; Gabrielsson, L.; Salomonsson, A.-C.; Hanke, L.; Murrell, B.; et al. SARS-CoV-2 Exposure, Symptoms and Seroprevalence in Healthcare Workers in Sweden. Nat. Commun. 2020, 11, 5064. [Google Scholar] [CrossRef]
  21. Ni, L.; Ye, F.; Cheng, M.-L.; Feng, Y.; Deng, Y.-Q.; Zhao, H.; Wei, P.; Ge, J.; Gou, M.; Li, X.; et al. Detection of SARS-CoV-2-Specific Humoral and Cellular Immunity in COVID-19 Convalescent Individuals. Immunity 2020, 52, 971–977. [Google Scholar] [CrossRef]
  22. Guo, L.; Ren, L.; Yang, S.; Xiao, M.; Chang, D.; Yang, F.; Dela Cruz, C.S.; Wang, Y.; Wu, C.; Xiao, Y.; et al. Profiling Early Humoral Response to Diagnose Novel Coronavirus Disease (COVID-19). Clin. Infect. Dis. 2020, 71, 778–785. [Google Scholar] [CrossRef] [PubMed]
  23. Yu, H.; Sun, B.; Fang, Z.; Zhao, J.; Liu, X.; Li, Y.; Sun, X.; Liang, H.; Zhong, B.; Huang, Z.; et al. Distinct Features of SARS-CoV-2-Specific IgA Response in COVID-19 Patients. Eur. Respir. J. 2020, 56, 2001526. [Google Scholar] [CrossRef]
  24. Suthar, M.S.; Zimmerman, M.G.; Kauffman, R.C.; Mantus, G.; Linderman, S.L.; Hudson, W.H.; Vanderheiden, A.; Nyhoff, L.; Davis, C.W.; Adekunle, O.; et al. Rapid Generation of Neutralizing Antibody Responses in COVID-19 Patients. Cell Rep. Med. 2020, 1, 100040. [Google Scholar] [CrossRef] [PubMed]
  25. Robbiani, D.F.; Gaebler, C.; Muecksch, F.; Lorenzi, J.C.C.; Wang, Z.; Cho, A.; Agudelo, M.; Barnes, C.O.; Gazumyan, A.; Finkin, S.; et al. Convergent Antibody Responses to SARS-CoV-2 in Convalescent Individuals. Nature 2020, 584, 437–442. [Google Scholar] [CrossRef]
  26. Marot, S.; Malet, I.; Leducq, V.; Zafilaza, K.; Sterlin, D.; Planas, D.; Gothland, A.; Jary, A.; Dorgham, K.; Bruel, T.; et al. Rapid Decline of Neutralizing Antibodies against SARS-CoV-2 among Infected Healthcare Workers. Nat. Commun. 2021, 12, 844. [Google Scholar] [CrossRef] [PubMed]
  27. Long, Q.-X.; Liu, B.-Z.; Deng, H.-J.; Wu, G.-C.; Deng, K.; Chen, Y.-K.; Liao, P.; Qiu, J.-F.; Lin, Y.; Cai, X.-F.; et al. Antibody Responses to SARS-CoV-2 in Patients with COVID-19. Nat. Med. 2020, 26, 845–848. [Google Scholar] [CrossRef]
  28. Van der Heide, V. Neutralizing Antibody Response in Mild COVID-19. Nat. Rev. Immunol. 2020, 20, 352. [Google Scholar] [CrossRef]
  29. Harvala, H.; Robb, M.L.; Watkins, N.; Ijaz, S.; Dicks, S.; Patel, M.; Supasa, P.; Wanwisa, D.; Liu, C.; Mongkolsapaya, J.; et al. Convalescent Plasma Therapy for the Treatment of Patients with COVID-19: Assessment of Methods Available for Antibody Detection and Their Correlation with Neutralising Antibody Levels. Transfus. Med. 2021, 31, 167–175. [Google Scholar] [CrossRef]
  30. Ng, D.L.; Goldgof, G.M.; Shy, B.R.; Levine, A.G.; Balcerek, J.; Bapat, S.P.; Prostko, J.; Rodgers, M.; Coller, K.; Pearce, S.; et al. SARS-CoV-2 Seroprevalence and Neutralizing Activity in Donor and Patient Blood. Nat. Commun. 2020, 11, 4698. [Google Scholar] [CrossRef]
  31. Luchsinger, L.L.; Ransegnola, B.P.; Jin, D.K.; Muecksch, F.; Weisblum, Y.; Bao, W.; George, P.J.; Rodriguez, M.; Tricoche, N.; Schmidt, F.; et al. Serological Assays Estimate Highly Variable SARS-CoV-2 Neutralizing Antibody Activity in Recovered COVID-19 Patients. J. Clin. Microbiol. 2020, 58, e02005-20. [Google Scholar] [CrossRef]
  32. Plotkin, S.A. Updates on Immunologic Correlates of Vaccine-Induced Protection. Vaccine 2020, 38, 2250–2257. [Google Scholar] [CrossRef] [PubMed]
  33. Therrien, C.; Serhir, B.; Bélanger-Collard, M.; Skrzypczak, J.; Shank, D.K.; Renaud, C.; Girouard, J.; Loungnarath, V.; Carrier, M.; Brochu, G.; et al. Multicenter Evaluation of the Clinical Performance and the Neutralizing Antibody Activity Prediction Properties of 10 High-Throughput Serological Assays Used in Clinical Laboratories. J. Clin. Microbiol. 2021, 59, e02511-20. [Google Scholar] [CrossRef] [PubMed]
  34. Patel, E.U.; Bloch, E.M.; Clarke, W.; Hsieh, Y.-H.; Boon, D.; Eby, Y.; Fernandez, R.E.; Baker, O.R.; Keruly, M.; Kirby, C.S.; et al. Comparative Performance of Five Commercially Available Serologic Assays to Detect Antibodies to SARS-CoV-2 and Identify Individuals with High Neutralizing Titers. J. Clin. Microbiol. 2021, 59, e02257-20. [Google Scholar] [CrossRef]
  35. Amanat, F.; Stadlbauer, D.; Strohmeier, S.; Nguyen, T.H.O.; Chromikova, V.; McMahon, M.; Jiang, K.; Arunkumar, G.A.; Jurczyszak, D.; Polanco, J.; et al. A Serological Assay to Detect SARS-CoV-2 Seroconversion in Humans. Nat. Med. 2020, 26, 1033–1036. [Google Scholar] [CrossRef]
  36. Caruana, G.; Croxatto, A.; Coste, A.T.; Opota, O.; Lamoth, F.; Jaton, K.; Greub, G. Diagnostic Strategies for SARS-CoV-2 Infection and Interpretation of Microbiological Results. Clin. Microbiol. Infect. 2020, 26, 1178–1182. [Google Scholar] [CrossRef] [PubMed]
  37. Liu, W.; Liu, L.; Kou, G.; Zheng, Y.; Ding, Y.; Ni, W.; Wang, Q.; Tan, L.; Wu, W.; Tang, S.; et al. Evaluation of Nucleocapsid and Spike Protein-Based Enzyme-Linked Immunosorbent Assays for Detecting Antibodies against SARS-CoV-2. J. Clin. Microbiol. 2020, 58, e00461-20. [Google Scholar] [CrossRef]
  38. Manenti, A.; Maggetti, M.; Casa, E.; Martinuzzi, D.; Torelli, A.; Trombetta, C.M.; Marchi, S.; Montomoli, E. Evaluation of SARS-CoV-2 Neutralizing Antibodies Using a CPE-based Colorimetric Live Virus Micro-neutralization Assay in Human Serum Samples. J. Med. Virol. 2020, 92, 2096–2104. [Google Scholar] [CrossRef]
  39. Bohn, M.K.; Lippi, G.; Horvath, A.; Sethi, S.; Koch, D.; Ferrari, M.; Wang, C.-B.; Mancini, N.; Steele, S.; Adeli, K. Molecular, Serological, and Biochemical Diagnosis and Monitoring of COVID-19: IFCC Taskforce Evaluation of the Latest Evidence. Clin. Chem. Lab. Med. CCLM 2020, 58, 1037–1052. [Google Scholar] [CrossRef] [PubMed]
  40. Tan, C.W.; Chia, W.N.; Qin, X.; Liu, P.; Chen, M.I.-C.; Tiu, C.; Hu, Z.; Chen, V.C.-W.; Young, B.E.; Sia, W.R.; et al. A SARS-CoV-2 Surrogate Virus Neutralization Test Based on Antibody-Mediated Blockage of ACE2–Spike Protein–Protein Interaction. Nat. Biotechnol. 2020, 38, 1073–1078. [Google Scholar] [CrossRef]
  41. Kim, Y.-I.; Kim, S.-M.; Park, S.-J.; Kim, E.-H.; Yu, K.-M.; Chang, J.-H.; Kim, E.J.; Casel, M.A.B.; Rollon, R.; Jang, S.-G.; et al. Critical Role of Neutralizing Antibody for SARS-CoV-2 Reinfection and Transmission. Emerg. Microbes Infect. 2021, 10, 152–160. [Google Scholar] [CrossRef]
  42. Chandrashekar, A.; Liu, J.; Martinot, A.J.; McMahan, K.; Mercado, N.B.; Peter, L.; Tostanoski, L.H.; Yu, J.; Maliga, Z.; Nekorchuk, M.; et al. SARS-CoV-2 Infection Protects against Rechallenge in Rhesus Macaques. Science 2020, 369, 812–817. [Google Scholar] [CrossRef]
  43. Kiyuka, P.K.; Agoti, C.N.; Munywoki, P.K.; Njeru, R.; Bett, A.; Otieno, J.R.; Otieno, G.P.; Kamau, E.; Clark, T.G.; van der Hoek, L.; et al. Human Coronavirus NL63 Molecular Epidemiology and Evolutionary Patterns in Rural Coastal Kenya. J. Infect. Dis. 2018, 217, 1728–1739. [Google Scholar] [CrossRef]
  44. Callow, K.A.; Parry, H.F.; Sergeant, M.; Tyrrell, D.A.J. The Time Course of the Immune Response to Experimental Coronavirus Infection of Man. Epidemiol. Infect. 1990, 105, 435–446. [Google Scholar] [CrossRef] [PubMed]
  45. Cao, W.-C.; Liu, W.; Zhang, P.-H.; Zhang, F.; Richardus, J.H. Disappearance of Antibodies to SARS-Associated Coronavirus after Recovery. N. Engl. J. Med. 2007, 357, 1162–1163. [Google Scholar] [CrossRef]
  46. Zhang, J.-S.; Chen, J.-T.; Liu, Y.-X.; Zhang, Z.-S.; Gao, H.; Liu, Y.; Wang, X.; Ning, Y.; Liu, Y.-F.; Gao, Q.; et al. A Serological Survey on Neutralizing Antibody Titer of SARS Convalescent Sera. J. Med. Virol. 2005, 77, 147–150. [Google Scholar] [CrossRef]
  47. Ko, J.-H.; Seok, H.; Cho, S.Y.; Eun Ha, Y.; Baek, J.Y.; Kim, S.H.; Kim, Y.J.; Park, J.K.; Chung, C.R.; Kang, E.-S.; et al. Challenges of Convalescent Plasma Infusion Therapy in Middle East Respiratory Coronavirus Infection: A Single Centre Experience. Antivir. Ther. 2018, 23, 617–622. [Google Scholar] [CrossRef] [PubMed]
  48. Wang, K.; Long, Q.-X.; Deng, H.-J.; Hu, J.; Gao, Q.-Z.; Zhang, G.-J.; He, C.-L.; Huang, L.-Y.; Hu, J.-L.; Chen, J.; et al. Longitudinal Dynamics of the Neutralizing Antibody Response to Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) Infection. Clin. Infect. Dis. 2021, 73, e531–e539. [Google Scholar] [CrossRef]
  49. Ibarrondo, F.J.; Fulcher, J.A.; Goodman-Meza, D.; Elliott, J.; Hofmann, C.; Hausner, M.A.; Ferbas, K.G.; Tobin, N.H.; Aldrovandi, G.M.; Yang, O.O. Rapid Decay of Anti–SARS-CoV-2 Antibodies in Persons with Mild Covid-19. N. Engl. J. Med. 2020, 383, 1085–1087. [Google Scholar] [CrossRef]
  50. Patel, M.M.; Thornburg, N.J.; Stubblefield, W.B.; Talbot, H.K.; Coughlin, M.M.; Feldstein, L.R.; Self, W.H. Change in Antibodies to SARS-CoV-2 Over 60 Days Among Health Care Personnel in Nashville, Tennessee. JAMA 2020, 324, 1781. [Google Scholar] [CrossRef] [PubMed]
  51. Long, Q.-X.; Tang, X.-J.; Shi, Q.-L.; Li, Q.; Deng, H.-J.; Yuan, J.; Hu, J.-L.; Xu, W.; Zhang, Y.; Lv, F.-J.; et al. Clinical and Immunological Assessment of Asymptomatic SARS-CoV-2 Infections. Nat. Med. 2020, 26, 1200–1204. [Google Scholar] [CrossRef] [PubMed]
  52. Meyer, B.; Reimerink, J.; Torriani, G.; Brouwer, F.; Godeke, G.-J.; Yerly, S.; Hoogerwerf, M.; Vuilleumier, N.; Kaiser, L.; Eckerle, I.; et al. Validation and Clinical Evaluation of a SARS-CoV-2 Surrogate Virus Neutralisation Test (SVNT). Emerg. Microbes Infect. 2020, 9, 2394–2403. [Google Scholar] [CrossRef] [PubMed]
Figure 1. Samples selected to determine thresholds of anti-S eCLIA and sVNT values predictive of neutralizing activity using a cVNT as a gold standard.
Figure 1. Samples selected to determine thresholds of anti-S eCLIA and sVNT values predictive of neutralizing activity using a cVNT as a gold standard.
Viruses 15 00426 g001
Figure 2. Samples selected to compare anti-S eCLIA and sVNT results. * Including 3 other HcoV, ** HCW—healthcare workers.
Figure 2. Samples selected to compare anti-S eCLIA and sVNT results. * Including 3 other HcoV, ** HCW—healthcare workers.
Viruses 15 00426 g002
Figure 3. Receiver operating characteristic curve (ROC) for prediction of neutralization titers ≥1:16 by cVNT based on the anti-S eCLIA and sVNT, AUC (area under the curve); anti-S eCLIA: 0.958 (95% CI 0.889–1.011); sVNT: 0.870 (95% CI 0.752–0.988) (p < 0.0001).
Figure 3. Receiver operating characteristic curve (ROC) for prediction of neutralization titers ≥1:16 by cVNT based on the anti-S eCLIA and sVNT, AUC (area under the curve); anti-S eCLIA: 0.958 (95% CI 0.889–1.011); sVNT: 0.870 (95% CI 0.752–0.988) (p < 0.0001).
Viruses 15 00426 g003
Figure 4. Anti-S eCLIA (black dots) vs. sVNT (grey dots) results. Cut-offs provided by manufacturers and cut-offs established with cVNTs are represented in full and dotted lines, respectively. Means of titers for each population and each assay are represented by a dash (*p < 0.05; ** p < 0.01).
Figure 4. Anti-S eCLIA (black dots) vs. sVNT (grey dots) results. Cut-offs provided by manufacturers and cut-offs established with cVNTs are represented in full and dotted lines, respectively. Means of titers for each population and each assay are represented by a dash (*p < 0.05; ** p < 0.01).
Viruses 15 00426 g004
Table 1. Comparison between sVNT and anti-S eCLIA titers in different HCW populations.
Table 1. Comparison between sVNT and anti-S eCLIA titers in different HCW populations.
sVNTAnti-S Abs
Mean Titer
(Positive Cut-Off: >24 IU/mL)
N° and %
(>24 IU/mL)
N° and %
(>50 IU/mL)
Mean titer (Positive Cut-Off: >0.8 U/mL)N° and %
(>0.8 U/mL)
N° and %
(>75 U/mL)
Naïve pre-vaccination 10.82 ± 5 0 (0%)0 (0%)<0.40 (0%)0 (0%)
14 days post 1st Dose 86.26 ± 19821 (58%)12 (33%)64.78 ± 9028 (78%)10 (28%)
7 days post 2nd Dose 36,611,419.09
± 82,643,261
51 (86%)50 (85%)201.58 ± 9855 (93%)47 (80%)
Infected 9–12 months 394 ± 75564 (88%)42 (58%)169 ± 9272 (99%)56 (77%)
Infected 7 days post 1st Dose 31,306,349.36
± 85,874,735
23 (100%)22 (96%)250 ± 4123 (100%)22 (96%)
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Mouna, L.; Razazian, M.; Duquesne, S.; Roque-Afonso, A.-M.; Vauloup-Fellous, C. Validation of a SARS-CoV-2 Surrogate Virus Neutralization Test in Recovered and Vaccinated Healthcare Workers. Viruses 2023, 15, 426. https://0-doi-org.brum.beds.ac.uk/10.3390/v15020426

AMA Style

Mouna L, Razazian M, Duquesne S, Roque-Afonso A-M, Vauloup-Fellous C. Validation of a SARS-CoV-2 Surrogate Virus Neutralization Test in Recovered and Vaccinated Healthcare Workers. Viruses. 2023; 15(2):426. https://0-doi-org.brum.beds.ac.uk/10.3390/v15020426

Chicago/Turabian Style

Mouna, Lina, Mehdi Razazian, Sandra Duquesne, Anne-Marie Roque-Afonso, and Christelle Vauloup-Fellous. 2023. "Validation of a SARS-CoV-2 Surrogate Virus Neutralization Test in Recovered and Vaccinated Healthcare Workers" Viruses 15, no. 2: 426. https://0-doi-org.brum.beds.ac.uk/10.3390/v15020426

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop