Analysis of Evaluation Methods of Sustainable Supply Chain Management in Production Engineering Journals with High Impact

Abstract

Nowadays, scholars need to know the trends for specific themes and how the main methods are applied to solve the gaps. The research problem for this work is the assessment of methods used in the sustainable supply chain, published in journals with a high impact factor and especially in the production engineering field. The methodology utilized was an adapted version of the “Methodi Ordinatio”, in structured stages to select a limited class of papers of high scientific relevance, to show how the methods are being used. The search steps included the filtering of journals in the research field with relevant impact factors and selecting databases and keywords; reading titles and abstracts; classifying the number of citations; and critical reading of all papers listed in the bibliographic portfolio. The research results and analysis of frequency of papers in the portfolio were used to identify and describe the most commonly used evaluation methods, the correlated theories, advantages, and disadvantages of each researched work.

1. Introduction

According to Bai and Sarkis [1], environmental concerns of consumers, companies, and governments have increased in recent years. Sustainable supply chain management enables companies to increase their cost efficiency and improve productivity and product quality, resulting in competitive advantage [2].

The evaluation methods used to measure the performance of sustainable supply chains help organizations choose the best investments in programs and initiatives to generate the best return and support the development of environmentally responsible suppliers [1]. To accompany this trend, the availability of scientific works on this subject in databases is steadily growing. However, researchers are currently struggling to select papers considered prestigious and renowned by the scientific community [3]. A simple search of scientific papers to create a literary base for research without a well-structured process merely creates a vague stage that lacks scientific effectiveness.

The increasing of journals and publications about methods applied in several contexts, generates a necessity to study how these methods are really being developed and used in specific fields, so as to balance the principles of the triple bottom line (social, environmental and financial) [1,3].

There is a great number of works in traditional bibliographic research. However, in order to know current trends in a theme, the present methodologies do not have accessible tools for performing a more specific analysis.

This study focuses on an adapted methodology to search the main methods published in journals from the production engineering field [4]. This covers a large number of themes such as public administration and business, accounting and tourism, as well as production engineering. Thus, by the increased number of journals existing in this field of interest, the purpose of this study is to analyze the sustainable supply chain management evaluation models currently used in the literature, by applying a structured methodology capable of selecting specific and relevant scientific articles to answer questions in the production engineering field.

The relevance of this development is the generation of a capacity to concentrate efforts in the investigation of specific points to answer questions about only one field of interest, and not an extensive literature review that covers other irrelevant material.

Some literature review studies have already studied the topic of supply chain management and its applications, each with its own specific characteristics, but none have yet addressed the theme of sustainable supply chain management in specific production engineering journals. Kouvelis et al. [5] for example conducted a literature review focused on supply chain management that was published in a journal of production and operations management (POM). As a result, the articles found addressed topics such as supply chain design, uncertainty and whip effect, supply chain contracts, decisions on capacity and supply, applications and practices, and supply chain management education. Zimmer et al. [6] reviewed the sustainable supplier management (SSM) literature, focusing on decision-making support models in the selection, monitoring and development of sustainable suppliers. The justification for this study lies in the assumption that environmental management indicators can be the most important performance evaluation methods for companies to increase their competitive advantage in the near future [2].

Moreover, studies on evaluation methods that support the environmental management of the company and the development of suppliers are still limited in the literature [1,7]. The theoretical discussion of this study is based on the relevance of analyses, by bringing a discussion on methods applied in production engineering, which grows exponentially. The articles produced in the theme of sustainable supply chain management, in general, are quite broad and so researchers need to treat and filter the results that interest them. Therefore, this article seeks to overcome this difficulty, motivating a debate more directed to the scholars of this specific field.

Consequently, the methods that make it possible to evaluate sustainable supply chain management in high impact factor journals can benefit stakeholders, both in companies and universities, and serve as a reference for future research. Thus, the question that this study examines is: Which models for assessing sustainable supply chain management are being used in papers published in journals with a high impact factor?

2. Materials and Methods

To create a bibliographic portfolio, 12 stages were adopted as follows:

In the first stage, the filtering of periodicals for the interested field was performed (in this case, production engineering). The second stage checked the impact factor score (IF) of the journals and sorted them in descending order. The IF is calculated annually by the Institute for Scientific Information for the journals in its database, and published by the Journal Citations Reports (JCR) [8]. The journals with an IF value under 1 were excluded since they represent journals with a low IF to the field. Three journals with this characteristic were excluded, resulting in 154 journals.

In the third stage, another filtering of journals took place, focused on production engineering. In our application from 154 journals in the production engineering category (with an IF greater than 1), 115 were found aligned with this theme. For filtering, each of the 154 journals were searched in the databases that describe the journals and their scientific indicators [9,10].

The fourth stage addresses the selection of databases to conduct the search for articles. Given our familiarity with tools created in previous studies, we worked with the databases Scopus and Science Direct in the practice stage of this work. However, there is a wide range of databases available, such as Science Direct, Scientific Electronic Library Online (SciELO.), Scopus, and Web of Science ISI (Institute for Scientific Information) [3,11,12].

The fifth stage consists of keyword selection that will guide the search. It should be noted that the keywords in scientific papers are used for the indexes and search engines to find relevant documents and, thus, provide a description of the subjects in discussion [13]. Seven keywords were selected for the “indicators” (assessment, evaluation, measurement, performance indicators, measures, metrics), two for “supply chain” (supply chain, supplier), and four for “sustainable” (green, sustainability, sustainable, triple bottom line). Figure 1 shows a diagram with the bibliographic research methodology structure.

The main search filters used in the databases were: (a) the Boolean word “AND”, used between the keywords to filter papers with the three combinations of keywords in at least one of the fields of title, abstract, and keyword; (b) the keywords were searched in English to ensure searches in international journals; (c) the search period was defined as 2010 to 2019 to guarantee the inclusion of non-obsolete materials [3]; and, (d) in terms of file type, only “papers” were included to select specific documents for researchers. The search generated 56 keyword combinations. These combinations led to 35,842 papers, of which 6760 were found in the Scopus database and 29,082 in the Science Direct database.

The sixth stage consisted of exporting papers to Endnote in order to manage the gross bibliographic references. The seventh stage was defined to exclude papers that did not belong to the journals selected in stage three. This stage led to the elimination of 32,485 papers, since they did not belong to any of the 115 journals in the production engineering theme.

In the eighth stage, the exclusion of duplicate papers was performed. Of the 3357 papers, 2838 were duplicates and therefore were excluded from the research database. The remaining non-duplicated papers totaled 519 in the database for analysis. The ninth stage consisted of reading titles of the papers to see if they were associated with the objective of the research. After reading the papers, 147 titles were deleted because the title was not related to the research subject. The remaining papers totaled 372 for summary analysis.

This stage introduces a subjective aspect of research; however, as recommended by Tranfield, et al., [14] this stage should be carried out by more than one reviewer to minimize the subjectivity of the research.

Stage ten is organized in two steps. This stage consisted of reading the abstracts to check the consistency with the purpose of research. Of the 372 papers, 183 had abstracts that did not match the subject of research and were excluded. The abstracts of 189 papers were consistent with the research subject and were maintained in the database.

Stage eleven consisted of verifying the number of citations of each of the 189 papers in Google Scholar database. These papers were classified according to the decreasing number of citations. The cutoff point of the most cited and least cited articles was based on the Pareto’s rule (80/20) considering that approximately 20% of the most cited articles represents 80% of the total citations, totaling 14 articles. The 14 most cited articles were part of Repository A, while the 175 least cited papers were subjected to a differentiated analysis. It was found that the authors of the 175 least cited papers were part of the 14 most cited papers. Thus, 29 of the least cited papers had authors who were among the most cited, and were therefore reclassified to Repository B. The remaining 146 were eliminated from the study because they did not meet the proposed requirements.

Stage twelve consisted of adding the papers of repositories A and B to Repository C (with 43 papers). These 43 papers had high impact factor on sustainable supply chain and applied to production engineering. The papers are listed in alphabetical order in Box 1.

Box 1. Bibliographic portfolio of papers with high impact factor about the sustainable supply chain applied to Production Engineering.

• AZEVEDO, S. G.; CARVALHO, H.; MACHADO, V. C. The influence of green practices on supply chain performance: a case study approach. Transportation research part E: logistics and transportation review, v. 47, n. 6, p. 850-871, 2011. [15]
• BAI, C.; DHAVALE, D.; SARKIS, J. Complex investment decisions using rough set and fuzzy c-means: An example of investment in green supply chains. European Journal of Operational Research, v. 248, n. 2, p. 507-521, 2016. [16]
• BAI, C.; SARKIS, J. Green supplier development: analytical evaluation using rough set theory. Journal of Cleaner Production, v. 18, n. 12, p. 1200-1210, 2010. [1]
• CHIOU, T.; CHAN, H. K.; LETTICE, F.; CHUNG, S. H. The influence of greening the suppliers and green innovation on environmental performance and competitive advantage in Taiwan. Transportation Research Part E: Logistics and Transportation Review, v. 47, n. 6, p. 822-836, 2011. [2]
• DENG, H.; LUO, F.; WIBOWO, S. Multi-Criteria Group Decision Making for Green Supply Chain Management under Uncertainty. Sustainability, v. 10, p. 3150-3163, 2018. [17]
• GOVINDAN, K.; KHODAVERDI, R.; JAFARIAN, A. A fuzzy multi criteria approach for measuring sustainability performance of a supplier based on triple bottom line approach. Journal of Cleaner Production, v. 47, p. 345-354, 2013. [18]
• GREEN JR, K. W.; ZELBST, P. J.; MEACHAM, J.; BHADAURIA, V. S. Green supply chain management practices: impact on performance. Supply Chain Management: An International Journal, v. 17, n. 3, p. 290-305, 2012. [19]
• HSU, C.; KUO, T.; CHEN, S.; HU, A. H. Using DEMATEL to develop a carbon management model of supplier selection in green supply chain management. Journal of Cleaner Production, v. 56, p. 164-172, 2013. [20]
• KUO, R. J.; WANG, Y. C.; TIEN, F. C. Integration of artificial neural network and MADA methods for green supplier selection. Journal of Cleaner Production, v. 18, n. 12, p. 1161-1170, 2010. [21]
• KUSI-SARPONG, S.; GUPTA, H.; SARKIS, J. A supply chain sustainability innovation framework and evaluation methodology. International Journal of Production Research, v. 57, n. 7, p. 1990-2008, 2019. [22]
• LIN, R. Using fuzzy DEMATEL to evaluate the green supply chain management practices. Journal of Cleaner Production, v. 40, p. 32-39, 2013. [23]
• LIN, Y.; TSENG, M. Assessing the competitive priorities within sustainable supply chain management under uncertainty. Journal of Cleaner Production, v. 112, p. 2133-2144, 2016. [24]
• LOZANO, R.; HUISINGH, D. Inter-linking issues and dimensions in sustainability reporting. Journal of Cleaner Production, v. 19, n. 2, p. 99-107, 2011. [25]
• LUTHRA, S.; MANGLA, S. K.; XU, L.; DIABAT, A. Using AHP to evaluate barriers in adopting sustainable consumption and production initiatives in a supply chain. International Journal of Production Economics, v. 181, p. 342-349, 2016. [26]
• MANGLA, S. K.; GOVINDAN, K.; LUTHRA, S. Critical success factors for reverse logistics in Indian industries: a structural model. Journal of Cleaner Production, v. 129, p. 608-621, 2016. [27]
• PATCHARA, P.; CHUNQIAO, T. An integrated multi-criteria decision-making model based on prospect theory for green supplier selection under uncertain environment: A case study of the Thailand palm oil products industry. Sustainability, v. 11, p.1871-1894, 2019. [28]
• TSENG, M.; CHIU, A. Evaluating firm’s green supply chain management in linguistic preferences. Journal of cleaner production, v. 40, p. 22-31, 2013. [29]
• TSENG, M. L.; LIMB, M. K.; WONG, W. P.; CHEN, Y. C.; ZHAN, Y. A framework for evaluating the performance of sustainable service supply chain management under uncertainty. International Journal of Production Economics, v. 195, p. 359-372, 2018. [30]
• PATHAK, D. K.; THAKUR, L. S.; RAHMAN, S. Performance evaluation framework for sustainable freight transportation systems, International Journal of Production Research, v. 57, n. 19, p. 6202-6222, 2019. [31]
• CHOI, S. B.; MIN, H.; JOO, H. Y.; CHOI, H. B. Assessing the impact of green supply chain practices on firm performance in the Korean manufacturing industry, International Journal of Logistics Research and Applications, v. 20, n. 2, p. 129-145, 2016. [32]
• SHAFIQ, A.; JOHNSON, P.; KLASSEN, R.; AWAYSHEH, A. Exploring the implications of supply risk on sustainability performance, International Journal of Operations & Production Management, v. 37 N. 10, p. 1386-1407 2017. [33]
• GANDHI, S.; MANGLA, S. K.; KUMAR, P.; KUMAR, D. A combined approach using AHP and DEMATEL for evaluating success factors in implementation of green supply chain management in Indian manufacturing industries, International Journal of Logistics Research and Applications, v. 19, n. 6, p. 537-561, 2016. [34]
• LUN, Y. H. V. Green management practices and firm performance: A case of container terminal operations, Resources, Conservation and Recycling, v. 55, n. 6, p. 559-566, 2011. [35]
• SHENC, L.; OLFAT, L.; GOVINDANB, K.; KHODAVERDIA, R. DIABAT, A. A fuzzy multi criteria approach for evaluating green supplier’s performance in green supply chain with linguistic preferences, Resources, Conservation and Recycling, v. 74, p. 170 – 179, 2013. [36]
• KIM, J.-S.; SHIN, N. The Impact of Blockchain Technology Application on Supply Chain Partnership and Performance. Sustainability, v. 11, n 21, p. 6181, 2019. [37]
• SUBIC, A.; SHABANI, B.; HEDAYATI, M.; CROSSIN, E. Performance Analysis of the Capability Assessment Tool for Sustainable Manufacturing. Sustainability, v. 5, n. 8, p. 3543-3561, 2013. [38]
• BARATA, J. F. F.; QUELHAS, O. L. G.; COSTA, H. G.; GUTIERREZ, R. H.; DE JESUS LAMEIRA, V.; MEIRIÑO, M. J. Multi-Criteria Indicator for Sustainability Rating in Suppliers of the Oil and Gas Industries in Brazil. Sustainability, v. 6, n. 3, p. 1107-1128, 2014. [39]
• VALIDI, S., BHATTACHARYA, A.; BYRNE, P. A Case Analysis of a Sustainable Food Supply Chain Distribution System - A Multi-Objective Approach. International Journal Of Production Economics, v. 152, p. 71-87, 2014. [40]
• HAYAMI, H.; NAKAMURA, M.; NAKAMURA, A. O. Economic performance and supply chains: The impact of upstream firms׳ waste output on downstream firms׳ performance in Japan, International Journal of Production Economics, v. 160, p. 47-65, 2015. [41]
• ESFAHBODI, A.; ZHANG, Y.; WATSON, G. Sustainable supply chain management in emerging economies: Trade-offs between environmental and cost performance’, International Journal of Production Economics, v. 181, p. 350-366, 2016. [42]
• KATIYAR, R.; MEENA, P. L.; BARUA, M.; TIBREWALA, R.; KUMAR, G. Impact of sustainability and manufacturing practices on supply chain performance: Findings from an emerging economy, International Journal of Production Economics, v. 197, p. 303-316, 2018. [43]
• WANG, X.; CHAN, H. K. A hierarchical fuzzy TOPSIS approach to assess improvement areas when implementing green supply chain initiatives, International Journal of Production Research, v. 51, n. 10, p. 3117-3130, 2013. [44]
• LI, C. An integrated approach to evaluating the production system in closed-loop supply chains, International Journal of Production Research, v. 51, n. 13, p. 4045-4069, 2013. [45]
• TSUI, C, W.; TZENG, G. H.; WEN, U. P. A hybrid MCDM approach for improving the performance of green suppliers in the TFT-LCD industry, International Journal of Production Research, v. 53, n. 21, p. 6436-6454, 2014. [46]
• GOSWAMI, M.; GHADGE. A. A supplier performance evaluation framework using single and bi-objective DEA efficiency modelling approach: individual and cross-efficiency perspective, International Journal of Production Research, p. 1-24, 2019. [47]
• FENG, M.; YU, W.; WANG, W.; WONG, C. Y.; XU, M.; XIAO, Z. Green supply chain management and financial performance: The mediating roles of operational and environmental performance, Business Strategy and the Environment, v. 27, n. 7, p. 811-824, 2018. [48]
• BLASS, V.; CORBETT, C. J. Same Supply Chain, Different Models Integrating Perspectives from Life Cycle Assessment and Supply Chain Management, Journal of Industrial Ecology, v. 22, n. 1, p. 18-30, 2018. [49]
• BAG, S. Green strategy, supplier relationship building and supply chain performance: total interpretive structural modelling approach. International Journal of Procurement Management, v. 9, n. 4, p. 398-426, 2016. [50]
• ARAMPANTZI, C.; MINIS, I. A new model for designing sustainable supply chain networks and its application to a global manufacturer. Journal of Cleaner Production, v. 156, p. 276-292, 2017. [51]
• NOYA, I.; VASILAKI, V.; STOJCESKA, V.; GONZÁLEZ-GARCÍA, S.; KLEYNHANS, C.; TASSOU, S.; MOREIRA, M. T.; KATSOU, E. An environmental evaluation of food supply chain using life cycle assessment: a case study on gluten free biscuit products, Journal of Cleaner Production, v. 170, p. 451-461, 2017. [52]
• CHATTERJEE, K.; PAMUCAR, D.; ZAVADSKAS, E. K. Evaluating the performance of suppliers based on using the R’AMATEL-MAIRCA method for green supply chain implementation in electronics industry, Journal of Cleaner Production, v. 184, p. 101-129, 2018. [53]
• DAS, D. The impact of Sustainable Supply Chain Management practices on firm performance: Lessons from Indian organizations, Journal of Cleaner Production, v. 203, p. 179-196, 2018. [54]
• KHAN, S.A; KUSI-SARPONG, S.; KOW ARHIN, F.; KUSI-SARPONG, H. Supplier sustainability performance evaluation and selection: A framework and methodology, Journal of Cleaner Production, v. 205, p. 964-979, 2018. [55]

In sequence, the bibliometric and systematic analysis of the 43 papers were performed.

3. Results

3.1. Bibliometric Analysis

3.1.1. Scientific Recognition of Articles

In order to verify the scientific recognition of the articles belonging to the bibliographic portfolio, a search was carried out to verify the number of citations of each article [56]. The data were sorted according to the decreasing number of citations.

Table 1. Scientific relevance of articles.

Paper	Year	N° Citations	% of Citations
Green Jr et al. [17]	2012	656	10.81
Govindan et al. [16]	2013	549	9.04
Chiou et al. [2]	2011	504	8.30
Kuo et al. [19]	2010	477	7.86
Bai Sarkis [1]	2010	436	7.18
Azevedo et al. [13]	2011	431	7.10
Lozano Huisingh [22]	2011	430	7.08
Hsu et al. [18]	2013	380	6.26
Lin [20]	2013	352	5.80
Shen et al. [36]	2013	319	5.25
Tseng Chiu [25]	2013	276	4.55
Validi et al. [40]	2014	181	2.98
Lun [35]	2011	137	2.26
Wang Chan [44]	2013	105	1.73
Luthra et al. [26]	2016	77	1.27
Bai et al. [14]	2016	76	1.25
Lin Tseng [21]	2016	75	1.24
Chatterjee et al. [53]	2018	71	1.17
Gandhi et al. [34]	2016	60	0.99
Mangla et al. [23]	2016	58	0.96
Tsui et al. [46]	2014	46	0.76
Tseng et al. [30]	2018	38	0.63
Arampantzi minis [51]	2017	37	0.61
Li [45]	2013	31	0.51
Bag [50]	2016	29	0.48
Kusi-Sarpong et al. [22]	2019	26	0.43
Katiyar et al. [43]	2019	24	0.40
Feng et al. [48]	2018	23	0.38
Hayami et al. [41]	2015	22	0.36
Blass Corbett [49]	2018	20	0.33
Choi et al. [32]	2016	19	0.31
Barata et al. [39]	2014	19	0.31
Shafiq et al. [33]	2017	18	0.30
Kan et al. [55]	2018	18	0.30
Noya et al. [52]	2017	16	0.26
Das [54]	2018	13	0.21
Subic et al. [38]	2013	12	0.20
Deng et al. [17]	2018	3	0.05
Pathak et al. [31]	2019	3	0.05
Patchara Chunqiao [24]	2019	1	0.02
Kin Shin [37]	2019	1	0.02
Esfahbodi et al. [42]	2016	1	0.02
Goswami Ghadge [47]	2019	1	0.02
Lozano Huisingh [22]	2011	431	7.10

shows the results of the citations of the 43 articles that comprise the bibliographic portfolio. It should be noted that the most recent articles have a low number of citations in comparison to the older ones, as they have had less time to be cited [3].

Table 1 shows the number of article citations collected from the Google Scholar database [57]. In addition, it should be noted that the 6 articles with a greater number of citations represent 50.29% of the total citations. Two articles with the lowest number of citations that belong to Repository B are redundant articles, but nonetheless, closely aligned with the research objective.

3.1.2. Relevance of Journals

Figure 2 shows the relevance of periodicals present in the portfolio. This analysis seeks to verify which journals publish the most on the sustainable supply chain subject, according to the delimitations of the research.

According to Figure 2, the journal that most published articles on sustainable supply chain in the field of production engineering was the Journal of Cleaner Production, with 14 articles, corresponding to a 32.55% of the portfolio. The International Journal of Production Economics and International Journal of Production Research, each have 6 articles, representing 13.95% of the total articles in the bibliographic portfolio, while Sustainability has 5 articles, representing 11.62% of the total articles. The International Journal of Logistics Research and Applications, Resources, Conservation and Recycling and Transportation Research Part E: Logistics and Transportation Review, each have 2 articles, representing 11.62% of the total number of articles. The other journals have only one article in the bibliographic portfolio. It is important to highlight the relevance of the Journal of Cleaner Production to the bibliographic portfolio, as it represents 32.44% of the total articles and 48.17% of citations.

3.1.3. Relevance of Authors

As shown in Figure 3, one author has four articles, as the main author or co-authored with others, three authors have three articles, five authors have two articles in the bibliographic portfolio in authorship or co-authorship and the other authors have just one article. For the analysis under consideration, this may also be an indication that the sustainable supply chain issue is of interest to various universities and researchers.

This analysis demonstrates that there is a plurality of researchers on this topic. Therefore, this scope of research may demonstrate that the topic is still under development in the research universe and that there are still gaps to fill.

3.1.4. Publications for Years

Figure 4 shows the number of publications for each of the 10 years surveyed. It should be noted that the year 2019 is also included in the research, although only until October, when the data collection for the bibliographic portfolio were finalized.

According to the analysis performed, and shown in Figure 4, every year surveyed added articles to the bibliographic portfolio. The years 2013, 2016 and 2018 have the largest number of articles published in the bibliographic portfolio, with 8 articles each year, representing 18.60% of the total articles each year.

The year 2019, even though it is not being considered in its entirety, has 5 articles, demonstrating that this year may surpass the others, while the years 2011, 2014 and 2017 have four publications, representing 9.30% each year, respectively.

The fourth largest number of publications is concentrated in 2010 and 2015, with two publications each year, and 2012 has only one publication, the lowest number found.

This analysis shows a growing trend of research development in this thematic area, because, with a small variability, the number of publications increases over the last years researched [35]. From 2016, the number of publications in the bibliographic portfolio remained high, ranging from 4 to 8 publications.

3.2. Systemic Analysis

After reading the 189 papers categorized in stage eleven, we selected 43 articles to create the final bibliographic portfolio, listed in stage twelve, for full reading and analysis, due to their greater relevance and impact in the area of research.

As shown in Table 1, the following methods for evaluating the sustainable supply chain were identified in the papers: rough set theory; data envelopment analysis (DEA) combined with the analytic network process (ANP), artificial neural network (ANN) and multi-attribute decision analysis (MADA); structural equation modeling; grounded theory; exploratory case study; decision-making trial and evaluation laboratory (DEMATEL); fuzzy multi-criteria; fuzzy multi-criteria combined with technique for order of preference by similarity to ideal solution (TOPSIS); fuzzy multi-criteria combined with DEMATEL; fuzzy multi-criteria + ANP, BW-MCDM, AHP, multivariate statistics, sustainable production capability assessment tool (CMAT), Tism + Fahp, blockchain technology (BCT), cluster analysis + one way ANOVA, ELECTRE TRI, PROMETHEE, AHP+TOPSIS and, finally, the analytic hierarchy process (AHP) combined with DEMATEL.

Table 2. Evaluation methods used in the papers of the bibliographic portfolio.

Evaluation Method	No. of Papers	%
Fuzzy Multi-Criteria	6	13.95%
Multivariate Statistics	6	13.95%
Exploratory Case Study	5	11.63%
AHP + DEMATEL	3	6.98%
DEA combined with ANP, RNA, and MADA	2	4.65%
Fuzzy Multi-Criteria + DEMATEL	2	4.65%
Fuzzy Multi-Criteria + TOPSIS	2	4.65%
Structural Equation Modelling	2	4.65%
Sust. Manuf. Capability Assessment Tool (CMAT)	2	4.65%
TISM + FAHP	2	4.65%
AHP	1	2.33%
Blockchain Technology (BCT)	1	2.33%
BW-MCDM	1	2.33%
Cluster Analysis + One Way ANOVA	1	2.33%
DEMATEL	1	2.33%
ELECTRE TRI	1	2.33%
Fuzzy Multi-Criteria + ANP	1	2.33%
Grounded Theory	1	2.33%
PROMETHEE	1	2.33%
Rough Set Theory	1	2.33%
AHP + TOPSIS	1	2.33%
Total	43	100%

shows that the ten first methods with 32 occurrences represent 74.41% of the total of occurrences. These ten main methods found used in the developed models were: fuzzy model (six occurrences); multivariate statistics (six occurrences); exploratory case study (five occurrences); AHP combined with DEMATEL (three occurrences); DEA combined with ANP, RNA and MADA (two occurrences); fuzzy multi-criteria + DEMATEL (two occurrences); fuzzy multi-criteria combined with TOPSIS (two occurrences); structural equation modeling (two occurrences); sustainable manufacturing capacity assessment tool (CMAT) (two occurrences) and TISM + FAHP (two occurrences). In the remaining studies (11 studies), evaluation methods appear only once, as follows: block chain technology (BCT) [37]; cluster analysis + one way ANOVA [32]; ELECTRE TRI [39]; PROMETHEE [46]; DEMATEL [58]; AHP [26]; BW-MCDM [22]; fuzzy multi-criteria + ANP [30]; rough set theory [1]; grounded theory [25] and AHP + TOPSIS [40].

4. Discussion

In this section we analyze the models/methods/methodologies found in the articles listed in the bibliographic portfolio. The evaluation methodologies are discussed, as well as advantages and disadvantages, exploring the trends and possible deficiencies of the application and integration of these methods.

4.1. Fuzzy Multi-Criteria

A fuzzy number transforms a linguistic variable into a real number between 0 and 1, unlike the Boolean theory, which only considers the exact values 0 and 1. Thus, a variable can be partially true or false, thereby creating degrees of relevance between the variables of a study [16,17,24,29,36,55].

The advantage of the fuzzy theory is that it can be applied in many different scenarios and using several possible criteria. However, a fuzzy number depends strongly on the judgment of each decision-maker, which carries a more subjective aspect into the study [16]. In some cases it can lose some refined and important information in precise scales, for example. Thus, some studies might be failing in performing a coherent application of this theory [16,17,18,23,24,29]. The fuzzy theory must be used in cases wherein the decision-maker is not certain about preferences or might be confused with the possible admitted variations by the nature of the problem itself. When it is not observed systematically, the application could present questionable results.

4.2. Multivariate Statistics

Multivariate statistics analyzes a group of variables together. The main advantage of this method is the considerable number of analyses for variables that represent the problem in question. With this, it is possible to provide a series of interpretations and interaction possibilities to the specialists, since these multivariate statistical methods make it possible to analyze the same phenomenon from different angles and to evaluate the chain of analysis [33,42,52].

One of the disadvantages of multivariate statistics for assessing sustainable supply chain performance is that it uses a large number of variables to solve one unique problem, because it is not enough to know isolated statistical data. Rather, it is important to understand all the information and the relationships present among them [45,54]. Because it is a purely mathematical method, the information extracted from the analysis needs to be validated by experts in the field of study, thus demonstrating vulnerability in this analysis model. These disadvantages make the interpretation of the phenomenon difficult using the chosen variables. On the other hand, in the case that it is organized in a systematic way, this method then brings important information to the decision-maker about the studied problem or necessity. Multivariate statistics analyses provide an understanding of the phenomena in a statistical way, considering multiple variables in the analysis. This technique, correctly integrated with other multi-criteria methods, can help to obtain more efficient results.

4.3. Exploratory Case Study

This method is used at the beginning of a study when the data that must be tested to form the theory are not accurately known. The researcher must define the research problem and the hypotheses to be tested, and go into the field [15,28,35,43,48].

The advantage is that it can initially be applied in any study. The downside of the theory is the impossibility of inferring results to be generalized for other scenarios. The study must be specific to the scenario in which it is applied. In the case of where a case study is conducted to study a success case (TPM—total productive maintenance theory from Toyota, for example) it contributes an interesting learning opportunity for the industries, in order to adapt the formed theory and the best practices found in the case study.

4.4. AHP Combined with DEMATEL

The AHP method is used to establish the priorities of the critical success factors of enterprises. The DEMATEL approach categorizes the causal relationships between these factors [27,34,53]. The advantage is that it is easy to apply and review, and can be applied to qualitative and quantitative variables. The downside is that the results cannot be used in other scenarios. The study must be specific for the scenario in which it was applied.

However, the use of similar multi-criteria methods, such as AHP and DEMATEL, may present deficiencies in the reliability and validity of the constructs used for the study. The problems arising from the misapplication of these methods reveal the tendency of the use of human factor preferences for decision making, which is the opinion of decision makers [56]. The AHP, for example, uses averages in its formulations. This could be a problem for data with great variation.

4.5. DEA Combined with ANP, RNA, and MADA

In the works found, this methodology was used for determining the indicators of green supplier selection, predict the performance measurement values of each indicator, and determine the weight of indicators. After obtaining the performance values for each supplier, the next stage is to conduct a data envelopment analysis to make a final assessment of the suppliers [21,47].

The advantages of this integration are that the data generate better results by evaluating the performance of suppliers. Moreover, the number of efficient suppliers is much lower than if the techniques had been applied to the best suppliers with this methodology. As for the disadvantages, after conducting the study with the suppliers, these suppliers should be classified, since the theory does not perform this task.

4.6. Fuzzy Multi-Criteria Combined with DEMATEL

The linguistic variables are translated into numbers using the fuzzy technique, followed by an analysis of the criteria of cause and effect between the variables found in the study [23,55]. The advantage is that this methodology does not require many variables for its application. The main disadvantage is that the results cannot be replicated in other studies since each model is specific for each company. The particularities of the fuzzy theory mean that it is completely dependent on the opinions of decision-makers.

4.7. Fuzzy Multi-Criteria Combined with TOPSIS

Triangular fuzzy numbers are used to express linguistic values of the subjective preferences of reviewers. These values are transformed into quantifiable numbers, after which the TOPSIS technique is proposed to find the ranking of suppliers [18,44].

The advantages of the theory are that it allows for the assessment of an unlimited range of criteria and performance attributes. Moreover, when the qualitative data are transformed into numbers, they are more easily understood and analyzed. The disadvantage of this theory is that the availability of data is critical to implement the model, and data are not always available to operationalize the study. Also the TOPSIS method presents some problems related to the reversion of ranking, which is also present in the major part of multi-criteria methods, but in the original version of TOPSIS it is more evident. There are efforts and variations of the TOPSIS method trying to minimize this problem.

4.8. Structural Equation Modelling (SEM)

This method is used to study the correlation between variables. It focuses on quantifying how quantifiable variables reflect the unobserved variables (constructs) [2,19]. The advantages of this theory include practices used by businesses to develop their sustainable supply chain to enhance environmental performance of the focal firm (measurable variable) and, consequently, improve organizational performance (construct). It also demonstrates advantage in the exploration of a statistical technique that brings reliability to the results, through the study of the correlation among variables.

One of the disadvantages of the theory is that the model can vary significantly with every company. There is no way of inferring whether the observable variables in an enterprise will generate the same constructs in another corporation, because each organization is unique.

4.9. Sustainable Manufacturing Capacity Assessment Tool (CMAT)

This method uses questionnaires for data collection, using the Likert scale to quantify qualitative data, and later uses statistical analysis for information processing. As advantages, this method presents an analysis of 170 resources linked to the supply chain, and can segment them to gain greater understanding of the demands of the researched environment [38]. The use of expert managers to answer the data collection questionnaire enables a deeper discussion of the critical elements belonging to the supply chain of the studied environment. Moreover, the method makes it possible to carry out a complete evaluation and subsequently propose continuous improvement initiatives through the proposed stages: measure, compare, relate, classify, document and propose improvements [51].

Disadvantages include the limitation of the method in analyzing the other aspects of the triple bottom line, as the method is designed to recognize only environmental aspects linked to the sustainable supply chain, showing difficulties in recognizing social and economic aspects. In addition, the questionnaires must be systematically constructed in order to collect the real information about the problem. It can be a problem if it is not made appropriately. However, this method can undergo adaptations and improvements to meet other aspects not recognized by the original model, which is the case for economic and social aspects.

4.10. TISM + FAHP

The advantages declared by the authors in the literature about the integration of the FAHP (fuzzy analytical hierarchy process) technique with TISM (total interpretive structural modeling) techniques can be defined by providing more sensitive results than using one of the techniques individually [31,50]. Another factor reported is that the integration of these methods makes it possible to perform an assessment with qualitative aspects, using TISM, and quantitative aspects from a nine-point numerical scale (Saaty Scale), using the FAHP method. The TISM method facilitates the transformation of inaccurate and limited articulated data from a complex system into a simple and well-defined model. This method assists in providing interpretation in a structural model [31]. These integrated methods are agile for data execution and processing. Conversely, the integration of these methods generates disadvantages and wrong applications. All TISM methodology has limitations, since it uses purely the expert opinion, which is qualitative in nature [50]. To minimize limitations, integration with another technique can be performed for statistical validation as well as for introducing other variables to the problem.

In this sense, studies [59,60] proposed to solve two research gaps (institutional barriers and resource constraints), combining TISM with SEM (Structural Equation Modeling) and MICMAC analysis. The TISM method uses the opinion of expert groups to understand the relationships between the studied variables and the SEM method with the aid of PLS (partial least squares method) which allows for the evaluation of exogenous variables. The results showed that the use of TISM associated with other methods of this nature can generate variability of data of qualitative nature, therefore the use of PLS was necessary. In this case the integration of the TISM method with the FAHP minimized this variability, because this second method uses weights based on the decision makers’ preference.

According to [60], the synthesis of the TISM structure and MICMAC analyses could be considered an alternative methodology for theory building research. This methodology shows how cross-sectional data could be treated to establish causality.

4.11. AHP

The AHP method is typically used to make selections of better alternative choices. Decisions are based on the evaluation of experts and decision makers. The main benefit of the method is that the judgment values of parity comparisons are based on experience, AHP can deal with both qualitative and quantitative aspects of a decision [26]. It enables the decision maker to gain insight into the system as a whole and its components, as well as the interactions of these components and their impacts on the whole.

The disadvantage of the method is related to its inadequate application, which can occur in unfavorable environments, when the decision makers make excessive simplifications of the analyzed context. In the bibliographic portfolio only one article [26] applies the AHP method without the use of auxiliary methods. The great problem in the use of the AHP method is the formulations based on averages that, in some cases, can bring unreliable results.

4.12. Blockchain Technology (BCT)

The advantages of BCT technology in the sustainable supply chain management partnership, is that it is a set of methods embedded in software development providing a range of interaction possibilities. BCT technology is now considered a next generation information technology, a tool for sustainable supply chain management growth [37].

However, being a method unique to a technological development, it is restricted to its initial configurations. In this way the analyses are linked to the existing software resources, and it is not possible to quickly modify for custom applications.

4.13. BW-MCDM

The best–worst method (BM-MCDA) can be applied to evaluations using scales, categories or paired comparisons to obtain preferences and importance for each evaluated element [22]. The main advantage of BM-MCDA is that this method requires less data for interactions, and results in more consistent comparisons for safer results. The disadvantage of the method is that decision makers need to choose between the best and the worst item in a data set, thus generating some subjective evaluations.

4.14. Cluster Analysis + One Way ANOVA

The use of cluster analysis combined with the one way ANOVA statistical method has advantages because it is possible to classify supply chain forming clusters according to specific characteristics defined by researchers within each type of practice [32]. ANOVA analysis of variance helps to verify if there is any difference in the company’s performance, because this method is able to treat various structures of experiments, besides estimating the variance with more accuracy and precision, extracting more information about the data.

As disadvantages of using these methods, it can be highlighted that the analysis of variance applied alone can limit the information, it is necessary to test sensitivity using alternative data analysis as confirmatory factor analysis.

4.15. DEMATEL

The technique studies the structure of cause and effect, analyses the relationship element “x” exerts on element “y”, and the relation “y” receives over element “x” [21]. The advantage of this theory is that it does not require many variables to evaluate the relationship between the elements [58]. The downside is that it only considers the qualitative aspect of data. In this specific context [21,58] the DEMATEL method may not represent all its usefulness, since the reliability of the data becomes questionable due to the number of decision makers involved.

4.16. ELECTRE TRI

It is a method that integrates indicators for strategic environmental assessment. This method can also integrate functions that support the decision maker in the resolution process, thereby reducing the cognitive effort required in the problem modeling and analysis phase [39].

However, as it is an exclusive method of non-compensatory classification, the ELECTRE TRI is restricted to this application, requiring complementary methods to search for analysis with other objectives. These constraints may promote simpler and more limited interpretations, and often the integration of other multi-criteria or statistical methods is necessary to expand the possibilities for analysis and development of more robust results.

4.17. Fuzzy Multi-Criteria+ANP

The method consists of determining the selection criteria and their respective weights. The combination of fuzzy multi-criteria + ANP methods has advantages related to simplifying data variability.

Similar to the application of fuzzy multi-criteria combined with DEMATEL, the disadvantage is that the results cannot be replicated in other studies because the models are independent and specific to each application context [30]. In the bibliographic portfolio, only one article [30] applies the fuzzy multi-criteria + ANP method in a combined manner.

4.18. Grounded Theory

The theory is based on deriving conceptual categories grounded in evidence to collect data. The information obtained from the range of data and their peculiarities is gathered to reach the conclusion of the study [25].

The advantage lies in the possibility of analyzing whether there are correlations between the studied theory and the collected data. In contrast, the disadvantage is that this theory calls for operationalization or the need to study literature, formulate a theory, and enter the field to test the theory.

4.19. PROMETHEE

The PROMETHEE method resembles the ELECTRE method in that it originates from the same theoretical basis. In the field of sustainable supply chain management evaluation, the PROMETHEE method has advantages in formally structuring the problems, allowing for simplification of peer comparison. Another noteworthy factor is that this method allows for checking the consistencies of the assigned weights, making its application more versatile, and there is also some application flexibility in situations where numerical ranges are used to represent priorities [46].

However, PROMETHEE also has some disadvantages in the application in these contexts. One of these limitations is related to the need to change from qualitative to numerical scale, as it can significantly change the result. Other disadvantages that can occur with this method are the problems with ranking reversal in the insertion of new alternatives and data mismatch in situations with large amounts of criteria.

4.20. Rough Set Theory

The rough set theory deduces that the information about a particular subject is represented in a data set to reach the conditional performance relations [1]. The advantages of the theory include the provision of extensive relations between the environmental attributes of suppliers and business performance; the theory does not need any preliminary data or additional information about the variables; and it can be used to complement the AHP/ANP techniques. The disadvantages are that the variables for supplier development can be broad, and effective filtering of the variables is recommended to restrict the decision-making rules to a minimum.

4.21. AHP Combined with TOPSIS

The AHP method is used to set priorities for making alternative best choice selections through a set of weights assigned by decision makers. Decision analysis is based on expert and decision-maker evaluation. One of the main advantages of the method is that the judgment values of parity comparisons are based on the experiences of the agents involved. The AHP recognizes both the qualitative and quantitative aspects of a decision. The integration of the AHP result with TOPSIS is used to find the sustainable supply chain ranking [40].

However, this integration of methods may be related to their inadequate application, because by the ease of application of the methods, decision makers can generalize and oversimplify the analyzed context. For both these and other methods, the data needs to be available and reliable for application so that the result is not compromised at the end of the analysis.

This presentation of the evaluation methods used in the 43 papers of the bibliographic portfolio used to assess sustainable supply chain management reveals that there is no best or worst model for companies. Each model has its advantages and disadvantages. Some authors of the listed studies applied more than one evaluation model to reduce the negativity of a technique [18,21,23,27].

The papers of the portfolio also revealed that for the evaluation model to be successful, the information of the focal company and its suppliers and customers must be available and easily accessible [2,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29]. The methods found in the papers of the portfolio complement each other, suggesting that future studies can analyze the integration of the assessment models fuzzy multi-criteria combined with AHP or fuzzy multi-criteria combined with ANP.

The analyzed studies show a tendency to use expert opinion together with the application of mathematical methods to produce optimal results [55]. When this combination occurs, the results are more relevant, because in the application of purely mathematical methods the preferences of the decision makers are not admitted, but when the use of this opinion occurs, the decisions can be modified, even if the mathematical methods are applied correctly.

Some studies suffer limitations regarding the application of the proposed methods [22,23,27,34]. The integration of some methods can cause problems in the research results, such as data reliability, validity of the constructs used for the study. The analysis of the articles in the bibliographic portfolio of this study helps to identify the success points of the integration of some methods, which are the advantages discussed in each method and the shortcomings of these applications.

Related to the increased need for food production to supply population growth and saturation of arable land, future studies could be developed to assess sustainable food supply chains in the agricultural environment [61]. Studies in other contexts can also be developed using combined methods that recognize the preferences of experts [55].

5. Conclusions

From the application of the proposed methodology, the results show a balance between methodological rigor and scientific relevance. Minimizing research problems indicated by Tranfield et al. [14], which classify three types of recurring problems, research with low rigor and high relevance that are called “popular science”. “Pedant science” is high in rigor but low in relevance; in turn “child science” is neither rigorous nor relevant.

This work answered the initial research question and achieved the overall objective of developing a methodology to search papers in a structured way. The specific objectives were also achieved, namely: (i) to describe each stage of this research methodology in the research procedures; (ii) to present and discuss the results using a table and observations, and subsequently provide an overview of the subject analysis of methods to evaluate environmental supply chain management applied to production engineering in journals with a high impact factor; and, (iii) by means of these results and discussions, contribute to literature on the subject of this work by listing the subjects surveyed over the last nine years on methods to evaluate environmental supply chain management with their advantages and disadvantages.

The results of this paper are valid only for the specific context in which the research was conducted, that is, scientific production of models to assess sustainable supply chain management applied to Production Engineering in Engineering III journals with the highest JCR in this discipline from January 2010 to June 2019. Thus, the results of this research cannot be used in other contexts. The research methodology, however, can be replicated in other studies using other subject matters and other keywords to create a bibliographic portfolio.

The findings of this article corroborate the development of further discussions to fill the gap highlighted by the study by Luo et al. [60] that indicated the need for theory building in sustainable supply chains. It also helps researchers describe the opportunities and challenges of applying a bibliometric methodology to sustainable supply chain management. The analysis provides a range of results that demonstrate the main methods used in the researched context, thus assisting in the development of new research and the deepening of the concepts already researched.

The theoretical contribution of this study could be highlighted by the methodology to focus on the research of methods applied in a specific research field. The articles already produced in the theme of sustainable supply chain management, bring a generalist approach so that many researchers on this subject in the field of production engineering need to treat and filter the results that interest them. Therefore, this article seeks to overcome this difficulty, motivating a debate more directed to the scholars of this specific field.

The present study assists the literature in discussing how the methods are being applied in sustainable supply chain management. One deficiency found, which needs further study, is the misapplication of analysis methods, which are applied without the judgment of the decision makers. The results showed that the integration of the subjective view of the actors involved, together with the application of mathematical models, complement each other, making the results more relevant and effective.

The present study developed its results in order to provide a directed discussion to the many researchers in the field of production engineering, avoiding the fact that these researchers need to access generalist results, which still need to be filtered to meet their interests. The study is not intended to replace any previous theory, but rather, to add analysis to the scientific debates related to sustainable supply chain performance assessment.

Another relevant contribution of this study is related to the identification of the tendency to use methods that help in understanding the preferences of the human factor in the decision process. Expert opinion motivates the reduction of deficiencies in the application of purely mathematical methods, which may differ due to the use of quantitative vision.