Next Article in Journal
City Information Modeling: State of the Art
Next Article in Special Issue
Gill Histopathology as a Biomarker for Discriminating Seasonal Variations in Water Quality
Previous Article in Journal
Current Status of Optical Systems for Measuring Lycopene Content in Fruits: Review
Previous Article in Special Issue
The Nasal Epithelium as a Route of Infection and Clinical Signs Changes, in Rainbow Trout (Oncorhynchus mykiss) Fingerlings Infected with Aeromonas spp
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Lipoxygenase Enzymes, Oligosaccharides (Raffinose and Stachyose) and 11sA4 and A5 Globulins of Glycinin Present in Soybean Meal Are Not Drivers of Enteritis in Juvenile Atlantic Salmon (Salmo salar)

1
Agriculture & Food, Livestock & Aquaculture Program, Bribie Island Research Centre, CSIRO, Woorim, QLD 4507, Australia
2
Agriculture & Food, Crops Program, Queensland Bioscience Precinct, CSIRO, St. Lucia, QLD 4067, Australia
3
Agriculture & Food, Livestock & Aquaculture Program, Queensland Bioscience Precinct, CSIRO, St. Lucia, QLD 4067, Australia
*
Author to whom correspondence should be addressed.
Submission received: 10 September 2021 / Revised: 1 October 2021 / Accepted: 2 October 2021 / Published: 8 October 2021
(This article belongs to the Special Issue Histopathology of Aquatic Animals)

Abstract

:
Soybean meal has been largely investigated and commercially used in fish nutrition. However, its inclusion levels have been carefully considered due to the presence of antinutritional factors, which depending on a series of factors might induce gut inflammation damaging the mucosal integrity and causing enteritis. Several strategies including genetic engineering have been applied attempting to reduce or eliminate some of the antinutritional factors. Accordingly, we assessed the intestinal health of juvenile Atlantic salmon fed high levels of speciality soybean genotypes with reduced-to-no content amounts of lipoxygenases, altered glycinin profile and reduced levels of oligosaccharides. No major signs of enteritis, only indication of enteritis progression, was noticed in the soybean meal-based diets illustrated by mild changes in distal intestine morphology. Whereas fish, fed fishmeal control feeds, displayed normal distal intestine integrity. Speciality soybean types did not improve intestinal health of juvenile Atlantic salmon suggesting these antinutrients are not drivers of the intestinal inflammatory process in this species. No additional benefits in terms of production performance or blood biochemistry were noticed in the speciality soybean types compared to the traditional soybean.

1. Introduction

Soybean meal (SBM) has been largely investigated and commercially used in fish nutrition. Although SBM offers several advantages to the aquafeed industry including worldwide availability, competitive pricing, consistent nutritional quality, and an acceptable amino acid profile it also displays some constraints, mainly associated with antinutritional factors [1,2]. There is a long list of antinutritional factors including saponins, lectins, phytic acid, oligosaccharides, isoflavones, and allergens, among others [1]. These antinutrients are known to impair feed intake, palatability, growth performance, digestive enzymes and in some instances induce gut inflammation damaging the mucosal integrity and causing enteritis [1,3,4,5,6,7,8]. The degree of physiological impairments induced by dietary SBM is linked to the SBM inclusion level, blends of raw materials, duration of feeding SBM-based feeds, and the species sensitivity to antinutritional factors [3,4,5,6,9,10]. Intestinal damage induced by SBM has been reported mostly in distal intestine and liver tissues across several fish species including Atlantic salmon Salmo salar [3,4,9,10], Totoaba Totoaba macdonaldi [5,6], Seriola spp Seriola lalandi, Seriola dorsalis [7,11,12,13], common Carp Cyprinus carpio [14,15], and Largemouth Bass Micropterus salmoides [8]. Distal intestine histology shows changes in the length of the mucosal fold, reduction in the number of supranuclear vacuoles of the enterocytes and thickness of the lamina propria, among other pathohistological modifications [3,5,6,10,12,16,17].
Among the fish species, salmon appears to be one of the most sensitive to the antinutritional factors presented in SBM developing a condition known as soybean meal-induced enteropathy, which exhibits similar changes as those described above [3,4,9,17]. Some salmon species such as pink salmon Oncorhynchus gorbuscha appears to be more resistant to antinutritional factors present in dietary SBM than chinook O. tshawytscha and Atlantic salmon S. salar [18]. In the early 2000’s, Buttle et al. [19], suggested the binding mechanism of soybean agglutinin (lectin) to Atlantic salmon intestinal epithelium as a primary contributor to pathological changes in this tissue. Saponins are other top candidates of key antinutritional factors present in soybean meal. A dose-response study reported increasing inflammatory process in Atlantic salmon distal intestine with greater dietary soybean saponins [20].
Several strategies have been applied attempting to reduce or eliminate some of the antinutritional factors in SBM including extrusion, fermentation, pre-processing techniques, and genetic engineering. For example, extrusion with shorter barrel retention times and higher temperatures improved utilization of SBM-rich diets (52% SBM) in salmonids [21]. Another commercial strategy is to increase the protein fraction of SBM through concentration (SPC) or isolation (SPI). A relatively small effort has been done in the genetic space focusing on selecting non-GM SBM for specific genotypes in aquafeeds. Recently, the removal of trypsin inhibitor, lectin and allergen P34/Gly m Bd 30 k from a soybean cultivar failed to alleviate inflammatory processes in Atlantic salmon [9]. Collectively, these studies suggest the challenge in identifying specific antinutritional factors present in soybean responsible for enteritis induction and highlight the complexity of interactions with compounds present in other plant ingredients largely used in aquafeed formulations. As a result, the salmon aquafeed industry has adopted ingredient inclusion limits and more processed soy protein products such as soy protein concentrate and isolate. However, from a cost-effective perspective, finding approaches to minimize or eliminate the soybean-induced enteritis in fish nutrition is worthwhile.
As part of the Australian Soybean improvement program, CSIRO has developed speciality SBM genotypes with reduced-to-no content amounts of lipoxygenases [22], altered glycinin profile [23], and reduced levels of oligosaccharides for human consumption. Accordingly, we assessed the intestinal health of juvenile Atlantic salmon fed high levels of these speciality SBM genotypes.

2. Materials and Methods

2.1. Formulations and Feed Manufacture

Dietary treatments are presented in Table 1. A fish meal-based diet (45%) was used as a control treatment, whereas the experimental diets contained 29% of fishmeal and 30% of SBM from three distinct genotypes of similar genetic background and matched as closely as possible for protein content: standard soybean meal (STD SBM); a soybean genotype homozygous for the gy4 allele conditioning null 11sA4 and 11sA5 globulins of Glycinin, homozygous for the l × 1, l × 2 and l × 3 alleles conditioning absence of seed lipoxygenases and homozygous for the rs2 allele conditioning near absence of seed raffinose and stachyose (TLP SBM); and a soybean genotype homozygous for the gy4 allele conditioning null 11sA4 and 11sA5 globulins of Glycinin (11sA4 null SBM).
All macro ingredients were milled to <750 μm, and well mixed with the remaining dry ingredients, and then extruded through a Baker-Perkins MPV24 twin-screw extruder. Each feed was manufactured using a 1.5 mm Ø die (~2.5 mm Ø pellets) using standard CSIRO Extrusion protocols (Table 2). The pellets were dried at 60 °C for 12 h, after which they were vacuum infused with their specific allocation of oil. All feeds were kept in frozen storage (−20 °C) throughout the feeding trial. All uneaten feed was removed from the collection tank of all treatments 1 h following feeding, and the collected waste feed was then dried to allow calculation of apparent feed intake and feed conversion ratio.

2.2. Experimental System and Feeding Trial

Sixteen tanks with 300 L at Bribie Island Research Centre (BIRC, Queensland, Australia) were used in this experiment with ~3 L/min flow of continuously aerated, recirculating freshwater at 15 °C ± 0.16 (mean ± SEM). Photoperiod was set at 12L:12D. Twenty-five salmon of 37.3 g ± 0.42 (mean ± SD) were randomly allocated to each tank. Fish were fed to apparent satiation twice daily for 56 days. This research was approved by the CSIRO Queensland Animal Ethics Committee—AEC Number: 2018-44.

2.3. Analytical Methods

Chemical analyses were carried out to confirm proximate composition of dietary treatments, and main ingredients (Table 1 and Table 3) [24]. Samples were dried at 105 °C for 12 h to determine gravimetrically dry matter and followed by ashing at 550 °C for 12 h. Total nitrogen was measured by combustion (CHNS auto-analyzer, Leco Corp., St. Joseph, MI, USA) and crude protein was calculated by nitrogen conversion (%N × 6.25). Total lipid was gravimetrically determined via Folch extraction [25]. Finally, gross energy was measured via an adiabatic bomb calorimeter (Parr 6200, Par Instrument Company, Moline, IL, USA).
Total amino acid (TAA) quantification was performed by mass detection following high performance reverse-phase liquid chromatography with pre-column derivatization with 6-aminoquinolyl-N-hydroxysuccinimidyl (AQC). Analyses were undertaken on a Shimadzu Nexera X2 series UHPLC (Shimadzu Corporation, Kyoto, Japan), coupled with a Shimadzu 8030 Mass Spectrometer using a modification of the Waters AccQ-tag system (Waters Corporation, Milford, MA). Bovine serum albumin (BSA) (ICN), milk powder (NIST SRM 1549a) and a well characterized aquafeed were used as reference materials. Samples or reference materials were hydrolyzed using phenolic 6N HCl at 112 °C according to the protocol for complex feed samples outlined by Waters Corp. (1996).

2.4. Production Parameters

The following production parameters were calculated for the 28 and 56 days of the feeding trial; feed conversion ratio (FCR), hepatosomatic index (HSI), and condition factor (K). No differences in production performance parameters were observed among the dietary treatments throughout the feeding trial.
FCR = FI WG
HSI = W Liver W Fish 100
K = 100 W Fish L Fork 3
where FCR = feed conversion ratio, FI = apparent feed intake (g), WG = weight gain (g), HSI = hepatosomatic index, WLiver = wet weight of liver (g), WFish = fish whole weight (g), K = condition factor, LFork = fish fork length (mm)

2.5. Histology

Upon completion of the feeding trial, three fish per tank were randomly selected and euthanized via AQIS overdose (~70 ppm). The distal intestine, i.e., last 2 cm section of the intestine, of each fish was sampled and stored in Davidson’s solution for 24 h, and then transferred to 70% ethanol until further analysis. For each fish sample, the distal intestine was divided into three sections and gradually dehydrated in ethanol, clarified in benzene and embedded in paraffin. As a result, a complete intestinal annular ring from each fish (nine per treatment) was cut into three sections (n = 36) and mounted onto individual glass slides for histological assessment.
Transversal sections of 3 μm were cut using a rotary microtome (Leica RM2245), stained with hematoxylin and eosin (H&E). The slides were blind examined after randomization, using the Zeiss Axiocam light microscope. The pictures were taken using the camera function on the Zeiss Axiocam microscope and then processed and analyzed using Zeiss Zen Light (Version 3.1) image analysis software.
To assess the degree of intestinal damage, a semiquantitative scoring system was used. In this scoring system, three parameters were quantified independently based on [10]: (1) the appearance and length of mucosal folds (MF); (2) the degree of widening of the lamina propria (LP); (3) the abundance of goblet cells (GC). For each of these parameters a score was given on a scale of 1 to 5. An increasing score value represents a greater degree of intestinal damage.

2.6. Statistical Analysis

All data were analyzed for normality and equality by Levene’s tests, respectively, and then subjected to one-way ANOVA (analysis of variance, NCSS 12.0). When significant effects were identified, the post-hoc Tukey’s HSD pairwise comparison test was used to determine difference among means with a significance level of 0.05.

3. Results

Juvenile Atlantic salmon fed the fishmeal control dietary treatment displayed normal distal intestine integrity (Figure 1A). There were no major signs of enteritis, only an indication of enteritis progression was noticed in the SBM-based diets illustrated by mild changes in distal intestine morphology, including reduced number and length of mucosal folds, enlargement of the apical zone of mucosal folds, thickening of lamina propria, and changes in abundance of goblet cells (Figure 1B–D). The removal of lipoxygenases, 11sA4 and A5 globulins of glycinin, and oligosaccharides from SBM failed to prevent morphological changes linked to the inflammatory process. Histology scoring demonstrated statistically higher scores of goblet cells and lamina propria in the SBM treatments than the fishmeal control (Table 4). Scoring of mucosal folds was higher in the speciality SBM groups compared to the control group.
Blood biochemistry was largely unaffected by the dietary treatments. Out of the sixteen parameters analyzed, only albumin and protein were statistically higher in TLP SBM than STD SBM (Table 5).
Survival was high across the dietary treatments (97–100%; Table 6). No differences in production performance parameters, including CV, final weight, FCR, K, and HSI were noticed between the fishmeal control group and the SBM-based groups at day 28 and day of feeding trial.

4. Discussion

Soybean meal-induced enteritis was not observed in juvenile Atlantic salmon, despite the high dietary SBM inclusion level of 30%. Nevertheless, intestinal health impairment was characterized by mild changes in distal intestine morphology (i.e., changes in length, shape, and number of mucosal folds, changes in thickness of lamina propria and changes in number of goblet cells), indicating enteritis progression. Speciality soybean types lacking 11sA4 and A5 globulins of Glycinin, lipoxygenases and oligosaccharides did not further reduce the intestinal inflammatory process compared to STD SBM, although inflammation was mild overall. Similarly, a recent thorough study removing three proteinaceous antinutrients, namely trypsin inhibitor, lectin and the allergen P34/Gly m Bd 30 k, from soybean meal did not mitigate enteritis in Atlantic salmon [9]. The authors suggested extrusion technology inactivated these compounds during feed manufacturing. Although soybean agglutinin, a type of lectin, has been reported to bind to Atlantic salmon intestinal epithelium contributing to pathological changes in gut health, the study did not describe the feed manufacturing procedure [19]. It is unlikely that extrusion technology inactivated the compounds investigated in the present study due to their chemical compositions and exposure to lower mechanical energy during extrusion conditions compared to those reported by [9] (specific mechanical energy69–80 vs. 1872–3060 kJ/kg, respectively).
Soybean-induced enteritis in Atlantic salmon has been widely reported. However, the main drivers of enteritis and their positive and negative interactions with other compounds present in feed formulations remain unclear. A summary of seventeen studies around soybean-induced enteritis in Atlantic salmon is provided in Table 7, Table 8 and Table 9. Due to the complexity of the matter, there is a wide breadth of experimental designs with different stocking densities, fish size, duration, water temperature, salinity, feeding ration and experimental systems. For example, most literature is focused on smaller fish ranging from 41–442 g and some on larger fish 500–927 g. Similarly, soybean type and pre-processing and dietary composition varies throughout the literature. Soybeans are from different parts of the globe and are under several pre-processing conditions such as dehulled, toasted, defatted, and solvent-extracted, included at various levels 8–34% in feeds containing crude protein of 35–47%, total lipids 23–31%, and gross energy 15–24 MJ/kg. Regardless of this variation in the experimental design, most histology analyses are focused on the distal intestine describing morphological changes of standardized parameters including mucosal folds, supranuclear vacuoles, lamina propria, eosinophilic granulocytes, sub-epithelial mucosa, and connective tissue, and continue to be one of the most reliable tools to detect this histopathological condition. Other parameters such as reduced feed intake and growth commonly used as indicators of enteritis are not as reliable with contradicting findings throughout the literature. Most studies highlight the inflammatory process in the intestine of Atlantic salmon fed soybean-based feeds leading to enteritis; however, there are exceptions where only minor to mild intestinal damage were reported, including the present findings. Collectively, these studies illustrate the challenge and high complexity in tackling this issue in fish nutrition. Understandably, the aquafeed industry does not prioritize this research topic and adopt a more conservative approach of using moderate inclusion levels of plant ingredients avoiding any potential intestinal health impairments caused by antinutritional factors. It is likely extremely difficult to identify the key antinutritional factors in the major plant ingredients and their interactions with compounds from other components of the formulations having the effect. Interestingly, the intestinal health research of fish fed soybean-based feeds has gone beyond the traditional highly carnivorous species reaching a wide range of species from different feed, salinity and water temperature preferences, including for example common carp [14,15], grass carp [26], kingfish [7,12], totoaba [5,6], and largemouth bass [8].
This is not the first study where animal growth was not impaired by dietary SBM displaying equivalent performance as the fishmeal SBM-free feeds. As Krogdahl et al. [27] has demonstrated, no detrimental growth effects in feeding 20% SBM to juvenile Atlantic salmon are present; however, more aggressive inclusion levels of 40% reduced growth are evident as compared to the fishmeal control feeds. Indeed, this pattern has been described with other fish species, including California yellowtail Seriola dorsalis [7]. Removal of certain antinutritional factors from SBM also did not affect Atlantic salmon growth response compared to the standard SBM [9]. Conservative inclusion levels of standard SBM at the expense of fishmeal appears to be suitable as long as no intestinal health impairment is noticed.
High levels of dietary SBM resulted in mild intestinal inflammation indicating enteritis progression. Speciality soybean types lacking lipoxygenases, altered glycinin profile and oligosaccharides did not improve intestinal health of juvenile Atlantic salmon suggesting these antinutrients are not drivers of the intestinal inflammatory process in this species. No additional benefits in terms of production performance or blood biochemistry were noticed in the speciality soybean types compared to the traditional soybean. The present findings contribute to and summarize the growing literature in the antinutrient space, providing more insights to future research.

Author Contributions

Conceptualization, A.N.R., A.T.J. and C.J.S.; Data curation, A.N.R., D.B. and T.N.; Formal analysis, A.N.R.; Funding acquisition, A.T.J. and C.J.S.; Methodology, D.B. and T.N.; Writing—original draft, A.N.R.; Writing—review & editing, D.B., A.T.J. and C.J.S. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by Commonwealth Scientific and Industrial Research Organisation grant number SIP331.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

This research was approved by the CSIRO Queensland Animal Ethics Committee—AEC Number: 2018-44.

Data Availability Statement

Data available upon request.

Acknowledgments

The authors thank Russell McCulloch and Roger Chong for assistance with the histology analysis and interpretation. We also thank Barney Hines, Nicholas Bourne and our late colleague Aijun Yang for the chemical analyses.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Gatlin, D.M.; Barrows, F.T.; Brown, P.; Dabrowski, K.; Gaylord, T.G.; Hardy, R.W.; Herman, E.; Hu, G.; Krogdahl, A.; Nelson, R.; et al. Expanding the utilization of sustainable plant products in aquafeeds: A review. Aquac. Res. 2007, 38, 551–579. [Google Scholar] [CrossRef]
  2. NRC (National Research Council). Nutrient Requirements of Fish and Shrimp; Natlional Academic Press: Cambridge, MA, USA, 2011. [Google Scholar] [CrossRef]
  3. Bakke-McKellep, A.M.; Press, C.M.; Baeverfjord, G.; Krogdahl, A.; Landsverk, T. Changes in immune and enzyme histochemical phenotypes of cells in the intestinal mucosa of Atlantic salmon, Salmo salar L., with soybean meal-induced enteritis. J. Fish Dis. 2000, 23, 115–127. [Google Scholar] [CrossRef]
  4. Krogdahl, A.; Bakke-McKellep, A.M.; Baeverfjord, G. Effects of graded levels of standard soybean meal on intestinal structure, mucosal enzyme activities, and pancreatic responses in Atlantic salmon (Salmo salar L.). Aquac. Nutr. 2003, 9, 361–371. [Google Scholar] [CrossRef]
  5. Fuentes-Quesada, J.P.; Viana, M.T.; Rombenso, A.N.; Guerrero-Renteria, Y.; Nomura-Solis, M.; Gomez-Calle, V.; Lazo, J.P.; Mata-Sotres, J.A. Enteritis induction by soybean meal in Totoaba macdonaldi diets: Effects on growth performance, digestive capacity, immune response and distal intestine integrity. Aquaculture 2018, 495, 78–89. [Google Scholar] [CrossRef]
  6. Fuentes-Quesada, J.P.; Cornejo-Granados, F.; Mata-Sotres, J.A.; Ochoa-Romo, J.P.; Rombenso, A.N.; Gurerrero-Renteria, Y.; Lazo, J.P.; Pohlenz, C.; Ochoa-Leyva, A.; Viana, M.T. Prebiotic agavin in juvenile tototaba, Totoaba macdonaldi diets, to relieve soybean meal-induced enteritis: Growth performance, gut histology and microbiota. Aquac. Nutr. 2020. [Google Scholar] [CrossRef]
  7. Viana, M.T.; Rombenso, A.N.; Del Rio-Zaragoza, O.B.; Nomura, M.; Diaz-Arguello, R.; Mata-Sotres, J.A. Intestinal impairment of the California yellowtail, Seriola dorsalis, using soybean meal in the diet. Aquaculture 2019, 513, 734443. [Google Scholar] [CrossRef]
  8. Rossi, W., Jr.; Allen, K.M.; Habte-Tsion, H.-M.; Meesala, K.-M. Supplementation of glycine, prebiotic, and nucleotides in soybean meal-based diets for largemouth bass (Micropterus salmoides): Effects on production performance, whole-body nutrient composition and retention, and intestinal histopathology. Aquaculture 2021, 532, 736031. [Google Scholar] [CrossRef]
  9. Krogdahl, A.; Kortner, T.M.; Jaramillo-Torres, A.; Gamil, A.A.A.; Chikwati, E.; Li, Y.; Schmidt, M.; Herman, E.; Hymowitz, T.; Teimouri, S.; et al. Removal of three proteinaceous antinutrients from soybean does not mitigate soybean-induced enteritis in Atlantic salmon (Salmo salar, L.). Aquaculture 2020, 514, 734495. [Google Scholar] [CrossRef]
  10. Uran, P.A.; Schrama, J.W.; Rombout, J.H.W.M.; Obach, A.; Jensen, L.; Koppe, W.; Verreth, J.A.J. Soybean meal-induced enteritis in Atlantic salmon (Salmo salar L.) at different temperatures. Aquac. Nutr. 2008, 14, 324–330. [Google Scholar] [CrossRef]
  11. Mata-Sotres, J.A.; Viana, M.T.; Tinajero, A.; Del Rio-Zaragoza, O.; Skrzynska, A. Estrés nutricional en peces y los nuevos desafíos de la nutrición. In Revista Bio Ciencias–Coloquio de Nutrigenomica y Biotecnología Acuicola 2020; Universidad Autónoma de Nayarit: Tepic, Mexico, 2021; p. e1132. [Google Scholar] [CrossRef]
  12. Stone, D.A.J.; Bellgrove, E.J.; Forder, R.E.A.; Howarth, G.S.; Bansemer, M.A. Inducing subacute enteritis in yellowtail kingfish Seriola lalandi: The effect of dietary inclusion of soybean meal and grape seed extract on hindgut morphology and inflammation. N. Am. J. Aquac. 2018, 80, 59–68. [Google Scholar] [CrossRef]
  13. Legrand, T.P.R.A.; Wynne, J.W.; Weyrich, L.S.; Oxley, A.P.A. Investigation both mucosal immunity and microbiota in response to gut enteritis in yellowtail kingfish. Microorganisms 2020, 8, 1267. [Google Scholar] [CrossRef]
  14. Uran, P.A.; Goncalvez, A.A.; Taverne-Thiele, J.J.; Schrama, J.W.; Verreth, J.A.J.; Rombout, J.H.W.M. Soybean meal induces intestinal inflammation in common carp (Cyprinus carpio L.). Fish Shellfish Immunol. 2008, 25, 751–760. [Google Scholar] [CrossRef] [PubMed]
  15. Van Der Marel, M.; Propsting, M.J.; Battermann, F.; Jung-Schroers, V.; Hubner, A.; Rombout, J.H.W.M.; Steinhagen, D. Differences between intestinal segments and soybean meal induced changes in intestinal mucus composition of common carp Cyprinus carpio L. Aquac. Nutr. 2014, 20, 12–24. [Google Scholar] [CrossRef]
  16. Van Den Ingh, T.S.G.A.M.; Krogdahl, A.; Olli, J.J.; Hendriks, H.G.C.J.M.; Koninkx, J.G.J.F. Effects of soybean-containing diets on the proximal and distal intestine in Atlantic salmon (Salmo salar) L. a morphological study. Aquaculture 1991, 94, 297–305. [Google Scholar] [CrossRef]
  17. Baeverfjord, G.; Krogdahl, A. Development and regression of soybean meal induced enteritis in Atlantic salmon, Salmo salar L.; distal intestine: A comparison with the intestines of fasted fish. J. Fish Dis. 1996, 19, 375–387. [Google Scholar] [CrossRef]
  18. Booman, M.; Forster, I.; Vederas, J.C.; Groman, D.B.; Jones, S.R.M. Soybean meal-induced enteritis in Atlantic salmon (Salmo salar) and Chinook salmon (Oncorhynchus tshawystscha) but not in pink salmon (O. gorbuscha). Aquaculture 2018, 483, 238–243. [Google Scholar] [CrossRef]
  19. Buttle, L.G.; Burrells, A.C.; Good, J.E.; Williams, P.D.; Southgate, P.J.; Burrells, C. The binding of soybean agglutinin (SBA) to the intestinal epithelium of Atlantic salmon, Salmo salar and Rainbow trout, Oncorhynchus mykiss, fed high levels of soybean meal. Vet. Immunol. Immunopathol. 2001, 80, 237–244. [Google Scholar] [CrossRef]
  20. Krogdahl, A.; Gajardo, K.; Kortner, T.M.; Penn, M.; Gu, M.; Berge, G.M.; Bakke, A.M. Soya saponins induce enteritis in Atlantic salmon (Salmo salar L.). J. Agric. Food Chem. 2015, 63, 3887–3902. [Google Scholar] [CrossRef] [PubMed]
  21. Barrows, F.T.; Stone, D.A.J.; Hardy, R.W. The effects of extrusion conditions on the nutritional value of soybean meal for rainbow trout (Oncorhynchus mykiss). Aquaculture 2007, 265, 244–252. [Google Scholar] [CrossRef]
  22. Yang, A.; Smyth, H.; Chalija, M.; James, A. Sensory quality of soymilk and tofu from soybeans lacking lipoxygenases. Food Sci. Nutr. 2016, 4, 207–215. [Google Scholar] [CrossRef]
  23. James, A.T.; Yang, A. Interactions of protein content and globulin subunit composition of soybean proteins in relation to tofu gel properties. Food Chem. 2016, 194, 284–289. [Google Scholar] [CrossRef]
  24. AOAC. Official Methods of Analysis; Association of Analytical Chemists: Arlington, VA, USA, 2005. [Google Scholar]
  25. Folch, J.; Less, M.; Sloane-Stanley, G.H. A simple method for the isolation and purification of total lipid from animal tissues. J. Biol. Chem. 1957, 276, 497–507. [Google Scholar] [CrossRef]
  26. Feng, L.; Ni, P.-J.; Jiang, W.-D.; Wu, P.; Liu, Y.; Jiang, J.; Kuang, S.-Y.; Tang, L.; Tang, W.-N.; Zhang, Y.-A.; et al. Decreased enteritis resistance ability by dietary low or excess levels of lipids through impairing the intestinal physical and immune barriers function of young grass carp (Ctenopharyngodon Idella). Fish Shellfish Immunol. 2017, 67, 493–512. [Google Scholar] [CrossRef] [PubMed]
  27. Olli, J.J.; Krogdahl, A.; Vabano, A. Dehulled solvent-extracted soybean meal as a protein source in diets for Atlantic salmon, Salmo salar L. Aquac. Res. 1995, 26, 167–174. [Google Scholar] [CrossRef]
  28. Knudsen, D.; Uran, P.; Arnous, A.; Koppe, W.; Frokler, H. Saponing-containing subfractions of soybean molasses induce enteritis in the distal intestine of Atlantic salmon. J. Agric. Food Chem. 2007, 55, 2261–2267. [Google Scholar] [CrossRef] [PubMed]
  29. Uran, P.A.; Schrama, J.W.; Jaafari, S.; Baardsen, G.; Rombout, J.H.W.M.; Koppe, W.; Verreth, J.A.J. Variation in commercial sources of soybean meal influences the severity of enteritis in Atlantic salmon (Salmo salar L.). Aquac. Nutr. 2009, 15, 492–499. [Google Scholar] [CrossRef]
  30. Chikwati, E.M.; Sahlmann, C.; Holm, H.; Penn, M.H.; Krogdahl, A.; Bakke, A.M. Alterations in digestive enzyme activities during the development of diet-induced enteritis in Atlantic salmon, Salmo salar L. Aquaculture 2013, 402–403, 28–37. [Google Scholar] [CrossRef]
  31. Sahlmann, C.; Sutherland, B.J.G.; Kortner, T.M.; Koop, B.F.; Krogdahl, A.; Bakke, A.M. Early response of gene expression in the distal intestine of Atlantic salmon (Salmo salar L.) during the development of soybean meal induced enteritis. Fish Shellfish Immunol. 2013, 34, 599–609. [Google Scholar] [CrossRef]
  32. Silva, P.F.; McGurk, C.; Knudsen, D.L.; Adams, A.; Thompson, K.D.; Bron, J.E. Histological evaluation of soya bean-induced enteritis in Atlantic salmon (Salmo salar L.): Quantitative image analysis vs. semi-quantitative visual scoring. Aquaculture 2015, 445, 42–56. [Google Scholar] [CrossRef]
  33. Jacobsen, H.J.; Samuelsen, T.A.; Girons, A.; Kousoulaki, K. Different enzyme incorporation strategies in Atlantic salmon diet containing soybean meal: Effects on feed quality, fish performance, nutrient digestibility and distal intestinal morphology. Aquaculture 2018, 491, 302–309. [Google Scholar] [CrossRef]
  34. Refstie, S.; Korsoen, O.J.; Storebakken, T.; Baeverfjord, G.; Lein, I.; Roem, A.J. Differing nutritional responses to dietary soybean meal in rainbow trout (Oncorhynchus mykiss) and Atlantic salmon (Salmo salar). Aquaculture 2000, 190, 49–63. [Google Scholar] [CrossRef]
Figure 1. Light microscopic images illustrating morphological changes in the distal intestine associated with inflammatory process in Atlantic salmon fed different soybean meal types for 56 days ((A)—control fishmeal, (B)—STD SBM—standard soybean meal, (C)—TLP SBM – triple null soybean meal absent of seed lipoxygenases and homozygous for the rs2 allele conditioning and near absent of seed raffinose and stachyose, and (D)—11sA4 null SBM—soybean meal conditioning null 11sA4 and 11sA5 globulins of Glycinin).
Figure 1. Light microscopic images illustrating morphological changes in the distal intestine associated with inflammatory process in Atlantic salmon fed different soybean meal types for 56 days ((A)—control fishmeal, (B)—STD SBM—standard soybean meal, (C)—TLP SBM – triple null soybean meal absent of seed lipoxygenases and homozygous for the rs2 allele conditioning and near absent of seed raffinose and stachyose, and (D)—11sA4 null SBM—soybean meal conditioning null 11sA4 and 11sA5 globulins of Glycinin).
Applsci 11 09327 g001
Table 1. Dietary formulation and proximate composition.
Table 1. Dietary formulation and proximate composition.
Ingredients g kg−1FM
Control
STD SBMTLP
SBM
11sA4 Null SBM
Fishmeal a450.0290.0290.0290.0
Wheat flour223.0110.093.0106.0
Wheat gluten120.089.0106.093.0
Blood meal a60.060.060.060.0
Fish oil a70.070.070.070.0
Poultry oil a70.070.070.070.0
Stay-C 35% b1.01.01.01.0
Vitamin mineral premix c6.06.06.06.0
Standard soybean meal (STD SBM) d0.0300.00.00.0
Soybean meal triple lipoxygenase plus (TLP SBM) d0.00.0300.00.0
Soybean meal 11sA4 null (11sA4 null SBM) d0.00.00.0300.0
Methionine e0.03.03.03.0
Taurine f0.01.01.01.0
Proximate composition (g kg−1)
Dry matter959960954954
Protein506492494505
Lipid166171179170
Ash68636262
Gross energy (kJ g−1)23.724.224.324.3
Amino acid (g kg−1)
ASP39434543
SER20222321
GLU88879389
GLY26242623
HIS16151515
ARG22252526
THR19192019
ALA25242424
PRO29282928
CYS5555
TYR14192019
VAL23232423
MET11111110
LYS26282828
ILE17181918
LEU38383938
PHE22232523
TAU3333
a Ridley, Aquafeeds, Queensland, Australia. b DSM, Heerlen, Netherlands. c Rabar Pty Ltd., Queensland, Australia. d CSIRO, Australia. e Redox, Queensland, Australia. f Bulk Nutrients, Tasmania, Australia.
Table 2. Dietary extrusion parameters.
Table 2. Dietary extrusion parameters.
ParameterFM ControlSTD SBMTLP
SBM
11sA4 Null SBM
RPM220220220220
Feed (g min−1)72625953
H2O (g min−1)16222020
Torque (%)8666
SME (kJ kg−1)99697380
Zone 1 (Cone)120120120120
Zone 295959595
Zone 385858585
Zone 4 (Inlet)70707070
Die Diameter (mm)1.51.51.51.5
RPM = revolutions per minute. SME = specific mechanical energy.
Table 3. Key ingredients proximate and amino acid composition (g kg−1).
Table 3. Key ingredients proximate and amino acid composition (g kg−1).
FishmealSTD SBMTLP
SBM
11sA4 null SBM
Proximate composition (g kg−1)
Dry matter911949918926
Protein719495521462
Lipid117119116118
Ash126545258
Amino acid (g kg−1)
ASP67.746.048.548.7
SER29.819.620.821.6
GLU97.469.573.469.1
GLY45.216.016.316.7
HIS16.29.110.29.3
ARG39.724.626.025.0
THR30.814.315.216.3
ALA40.716.317.517.6
PRO31.619.320.519.7
CYS6.55.55.25.3
TYR24.411.713.913.2
VAL34.416.717.516.7
MET22.82.83.03.1
LYS58.921.123.723.4
ILE29.015.816.717.5
LEU52.627.630.429.8
PHE30.118.218.820.4
TAU7.8NDNDND
ND = not detected.
Table 4. Semiquantitative scoring system (mean ± SD) of three parameters based on [10]: (1) the appearance and length of mucosal folds (MF); (2) the degree of widening of the lamina propria (LP); (3) the abundance of goblet cells (GC). For each of these parameters a score was given on a scale of 1 to 5. An increasing score value represents a greater degree of intestinal damage.
Table 4. Semiquantitative scoring system (mean ± SD) of three parameters based on [10]: (1) the appearance and length of mucosal folds (MF); (2) the degree of widening of the lamina propria (LP); (3) the abundance of goblet cells (GC). For each of these parameters a score was given on a scale of 1 to 5. An increasing score value represents a greater degree of intestinal damage.
ScoringFM
Control
STD
SBM
TLP
SBM
11sA4 Null SBMp-Value
Goblet cells1.9 ± 0.3 b3.2 ± 0.4 a3.4 ± 0.5 a3.7 ± 0.5 a<0.001
Lamina propria1.8 ± 0.4 b3.2 ± 0.4 a3.5 ± 0.5 a3.6 ± 0.5 a<0.001
Mucosal fold1.9 ± 0.5 b2.7 ± 0.7 ab2.9 ± 0.6 a3.3 ± 0.7 a0.001
Table 5. Sixteen blood chemistry parameters of juvenile Atlantic salmon fed a fishmeal control diet (FM control) and different soybean meal types-based diets (STD SBM—standard soybean meal, C—TLP SBM—triple null soybean meal absent of seed lipoxygenases and homozygous for the rs2 allele conditioning and near absent of seed raffinose and stachyose, and D—11sA4 null SBM—soybean meal conditioning null 11sA4 and 11sA5 globulins of Glycinin) for 56 days.
Table 5. Sixteen blood chemistry parameters of juvenile Atlantic salmon fed a fishmeal control diet (FM control) and different soybean meal types-based diets (STD SBM—standard soybean meal, C—TLP SBM—triple null soybean meal absent of seed lipoxygenases and homozygous for the rs2 allele conditioning and near absent of seed raffinose and stachyose, and D—11sA4 null SBM—soybean meal conditioning null 11sA4 and 11sA5 globulins of Glycinin) for 56 days.
Blood Chemistry ParametersFM
Control
STD
SBM
TLP
SBM
11sA4 Null SBMp-Value
Albumin (g L−1)20.3 ± 0.6 ab20.0 ± 1.0 b22.3 ± 1.1 a20.7 ± 0.6 ab0.044
Alkaline phosphatase (U L−1)21.3 ± 6.118.7 ± 4.724.3 ± 2.115.7 ± 5.70.242
Anion gap (mmol L−1)59.3 ± 3.556.0 1.053.0 ± 2.658.0 ± 8.70.463
AST (U L−1)301.3 ± 53.6301.7 ± 28.7359.5 ± 2.1315.7 ± 31.50.361
Bicarbonate (mmol L−1)6.3 ± 0.65.3 ± 1.15.7 ± 0.66.7 ± 2.10.577
Chloride (mmol L−1)123.3 ± 8.7120.3 ± 1.5125.3 ± 5.1120.0 ± 1.00.561
Cholesterol (mmol L−1)13.2 ± 2.99.8 ± 1.112.4 ± 0.911.9 ± 1.40.236
Creatine kinase (U L−1)7286 ±14257992 ± 11987446 ± 17198089 ± 16740.890
Globulin (g L−1)18.7 ± 2.118.7 ± 1.121.0 ± 1.020.3 ± 0.60.142
Glucose (mmol L−1)5.2 ± 0.84.2 ± 0.34.2 ± 0.24.0 ± 0.40.055
Phosphate (mmol L−1)3.1 ± 0.32.7 ± 0.32.6 ± 0.32.5 ± 0.30.103
Potassium (µm mmol−1)4.2 ± 0.64.1 ±0.64.1 ± 0.5 4.0 ± 0.40.963
Protein (g L−1)39.0 ± 2.6 ab38.7 ± 1.5 b43.3 ± 1.5 a41.0 ± 1.0 ab0.042
Ratio1.1 ± 0.11.1 ± 0.11.0 ± 0.11.0 ± 0.00.344
Sodium (mmol L−1)184.7 ± 11.6177.7 ± 0.6179.7 ± 3.0 180.3 ± 5.10.683
Triglyceride (mmol L−1)2.9 ± 0.43.5 ± 0.53.1 ± 0.83.5 ± 0.30.377
Table 6. Production performance at 28 and 56 days of juvenile Atlantic salmon fed a fishmeal control diet (FM control) and different soybean meal types-based diets (STD SBM—standard soybean meal, C—TLP SBM—triple null soybean meal absent of seed lipoxygenases and homozygous for the rs2 allele conditioning and near absent of seed raffinose and stachyose, and D—11sA4 null SBM—soybean meal conditioning null 11sA4 and 11sA5 globulins of Glycinin).
Table 6. Production performance at 28 and 56 days of juvenile Atlantic salmon fed a fishmeal control diet (FM control) and different soybean meal types-based diets (STD SBM—standard soybean meal, C—TLP SBM—triple null soybean meal absent of seed lipoxygenases and homozygous for the rs2 allele conditioning and near absent of seed raffinose and stachyose, and D—11sA4 null SBM—soybean meal conditioning null 11sA4 and 11sA5 globulins of Glycinin).
Production PerformanceFM
Control
STD
SBM
TLP
SBM
11sA4 Null SBM
0 day
Initial weight (g) 37.3 ± 0.037.4 ± 0.037.3 ± 0.137.3 ± 0.0
Initial CV (%)12.1 ± 0.515.7 ± 3.411.1 ± 0.811.5 ± 0.5
28 days
Survival (%)99 ± 0.097 ± 0.099 ± 0.0100
CV (%)12.3 ± 1.313.0 ± 0.512.1 ± 0.713.1 ± 0.9
Mid weight (g)49.6 ± 0.652.1 ± 0.749.8 ± 0.751.4 ± 0.6
FCR1.0 ± 0.01.0 ± 0.01.1 ± 0.11.0 ± 0.0
K1.3 ± 0.071.4 ± 0.091.5 ± 0.091.4 ± 0.01
HSI1.8 ± 0.161.7 ± 0.171.6 ± 0.191.9 ± 0.40
56 days
Survival (%)98.5 ± 1.597.0 ± 1.598.5 ± 1.5100
CV (%)15.2 ± 1.515.5 ± 0.814.6 ± 1.015.2 ± 0.7
Final weight (g)59.4 ± 1.566.1 ± 1.265.1 ± 2.164.4 ± 0.7
FCR1.0 ± 0.00.9 ± 0.01.0 ± 0.00.9 ± 0.0
K1.5 ± 0.171.4 ± 0.021.4 ± 0.041.3 ± 0.03
HSI1.6 ± 0.191.4 ± 0.111.4 ± 0.071.6 ± 0.25
* CV = coefficient of variance. FCR = feed conversion ratio. K = condition factor. HSI = hepatosomatic index.
Table 7. Summary of the experimental design of soybean meal-induced enteritis studies with Atlantic salmon (Salmo salar).
Table 7. Summary of the experimental design of soybean meal-induced enteritis studies with Atlantic salmon (Salmo salar).
Experimental Design
IBW (g)SGR (% BW)FCRTank Volume
(L)
Stocking Density (Fish/Tank)Duration (Days)SalinityTemp. (°C)FeedingSystemRefeed to ControlRef.
927 20002521FW12–13 Semi-RAS [3]
2800.94–1.050.81–0.944505460SW8.4 [4]
41 0.725055 56FW14 RAS [9]
300 4005020SW8 and 12 120%RAS [10]
550 27,000 120 3.25 120%Net pens [16]
550 27,0008042SW10.8 and 8.2120%Net pensYes[17]
535|140|166 * 1900|650|85050 21SW9 1% BW Flow-through [18]
54 100020 7FW15 [19]
442 100022 SW12–9 RAS [20]
900 125 000300 300SW8.2 Net pens [27]
213|202 400|10030|2062|44SW8.3|9 120%RAS [28]
396 4002528SW12 110%RAS [29]
500–600 10002521SW9 RAS [30]
500–600 100025–3021SW9 RAS [31]
80 40070 53SW8.6 120%Flow-through [32]
601.36–1.470.71–0.7450075 93SW10.9120%Flow-through [33]
2070.7–0.90.92–1.1760050 84FW7 115%Flow-through [34]
IBW = initial body weight in g; SGR = specific growth rates in percentage of body weight; FCR = feed conversion ratio; Temp. = temperature; Ref. = references; RAS = recirculating aquaculture systems; SW = seawater; FW = freshwater; * numbers represent the following species Atlantic salmon, Chinook salmon and pink salmon.
Table 8. Summary of soybean types and dietary treatments of soybean meal-induced enteritis studies with Atlantic salmon (Salmo salar).
Table 8. Summary of soybean types and dietary treatments of soybean meal-induced enteritis studies with Atlantic salmon (Salmo salar).
SBM InformationDietary Treatments
LocationTypeCP (%)Pellet TypeCP (%)TL (%)GE
(MJ /kg)
SBM Type/Inclusion (%)SBM Type/Inclusion (%)SBO Inclusion (%)Ref.
NorwayWith hulls, toasted and extractedExtruded3528 SBM 30 [3]
NorwaySolvent-extracted, toastedExtruded40–4422–2422–23SBM—8, 12, 15, 19, 27 [4]
USATriple null and standardExtruded402624SBM 25Triple null SBM 27[9]
NetherlandsExtracted Extruded4530 SBM 20 [10]
432015FFSBM 30SPC 280–10[16]
SBM 33 0–8.5[17]
Pelleted3723 SBM 20 [18]
Soybean agglutinin Soybean agglutinin 3.5 [19]
Soya saponins 42–4429–3024Soy saponins—0, 2, 4, 6, 10 [20]
Dehulled solvent extracted SBMExtruded4022 SBM 17 and 34 [27]
NorwayDefatted Extruded DSBM 20|molasses [28]
NA, EU, SA44–49Extruded42252320 [29]
Extracted 432824SBM 20 [30]
Extracted 432824SBM 20 [31]
NorwayDefatted Extruded472623SBM 25 + soy saponins 0.17Lupins + soy saponin concentrate 0.17 + soy saponins 0.11[32]
Switzerland Extruded462524SBM 20Pre-processed SBM 20[33]
Dehulled defatted SBMPelleted403124SBM 30 [34]
Ref. = references; NA = North America, EU = Europe, SA = South America; FFSBM = full-fat soybean meal; SBM = soybean meal; DSBM—defatted soybean meal; SPC = soy protein concentrate; SBO = soybean oil.
Table 9. Summary of the main findings and parameters investigated of soybean meal-induced enteritis studies with Atlantic salmon (Salmo salar).
Table 9. Summary of the main findings and parameters investigated of soybean meal-induced enteritis studies with Atlantic salmon (Salmo salar).
Growth ImpairmentFeed IntakeLevel of DI InflammationEnzymesTime Sampling (Days)Tissues AnalyzedParameters Ref.
SBM yes
SPC no
SBM +++ PI and DIMSA/LSC, ESA/LSC, LPSA/LSC, ESA/LPSA, GC/E, LM[16]
Yes +++ 2, 7, 14 and 21DIMF[17]
++5′ N, Mg-ATPase, ALP, ACP, NSE, LAP, AAP21 MI and DI [3]
Low SBM no
High SBM yes
No changes+, ++ and +++ALP, LAP, maltase, isomaltase, lactase and sucraseMI and DIMF, SNV, LP, leycocyte[4]
+, ++ and +++ DIMF, SNV, LP, CT[28]
++ and +++ DIMF, GC, LP SNV, EG, SM[29]
++ and +++ DIMF, GC, LP SNV, EG, SM[10]
+, ++, and +++Pancreatic (trypsin, chymotrypsin, elastase, and lipase), chyme (LAP), brush border membrane (LAP and maltase)0, 1, 2, 3, 5, 7, 10, 14, 17, and 21PI, MI and DI [30]
+, ++, and +++ 1, 2, 3, 5 and 7DI [31]
Low saponins no
Mid-high saponins yes
Low saponins no
Mid-high saponins yes
+, ++, and +++Trypsin activity, bile acids, brush border membrane enzyme activity (LAP)PI and DIMF, LP, enterocyte vacuolization, GC, nucleos position within the enterocytes[20]
+++ DIMF, SNV, LP, CT[32]
+, ++, and +++ 7, 14 and 21DIInflammation score, SM and microbiome[18]
NoNo changes DIMF, GC, LP, SNV, EG, SM[33]
NoNo changes+ and ++Brush border LAP, trypsin activityDIMF, SNV, SM, LP, microbiota, gene expression[9]
+ and ++ PI, MI and DIMF, SNV, LP, CT,[19]
SBM 17 no
SBM 34 yes
Body composition and blood biochemistry[27]
YesNo changes++ and +++ DIMF, SNV, LP, CT,[34]
Ref. = references; Level of enteritis: + mild, ++ moderate, and +++ high; 5′ N = 5′-nucleotidase; Mg-ATPase = Mg2+ dependent adenosine triphosphatase; ALP = alkaline phosphatase; ACP = acid phosphatase; NSE = non-specific esterase; LAP = leucine aminopeptidase, AAP = alanine aminopeptidase; PI = proximal intestine; MI = mid-intestine; DI = distal intestine; MF = mucosal fold; MSA = mucosal surface area; LSC = length mucosal stratum compactum; ESA = epithelial surface area; LPSA = lamina propria surface area; GC = goblet cells; E = 100 um epithelium; LM = length microvilli; SNV = supranuclear vacuoles; LP = lamina propria; EG = eosinophilic granulocytes; CT = connective tissue; SM = sub-epithelial mucosa.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Rombenso, A.N.; Blyth, D.; James, A.T.; Nikolaou, T.; Simon, C.J. Lipoxygenase Enzymes, Oligosaccharides (Raffinose and Stachyose) and 11sA4 and A5 Globulins of Glycinin Present in Soybean Meal Are Not Drivers of Enteritis in Juvenile Atlantic Salmon (Salmo salar). Appl. Sci. 2021, 11, 9327. https://0-doi-org.brum.beds.ac.uk/10.3390/app11199327

AMA Style

Rombenso AN, Blyth D, James AT, Nikolaou T, Simon CJ. Lipoxygenase Enzymes, Oligosaccharides (Raffinose and Stachyose) and 11sA4 and A5 Globulins of Glycinin Present in Soybean Meal Are Not Drivers of Enteritis in Juvenile Atlantic Salmon (Salmo salar). Applied Sciences. 2021; 11(19):9327. https://0-doi-org.brum.beds.ac.uk/10.3390/app11199327

Chicago/Turabian Style

Rombenso, Artur N., David Blyth, Andrew T. James, Teisha Nikolaou, and Cedric J. Simon. 2021. "Lipoxygenase Enzymes, Oligosaccharides (Raffinose and Stachyose) and 11sA4 and A5 Globulins of Glycinin Present in Soybean Meal Are Not Drivers of Enteritis in Juvenile Atlantic Salmon (Salmo salar)" Applied Sciences 11, no. 19: 9327. https://0-doi-org.brum.beds.ac.uk/10.3390/app11199327

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop