Multi-Criteria Methods Applied in the Studies of Barriers Identified in the Implementation of Reverse Logistics of E-Waste: A Research Agenda

Abstract

Reverse logistics (RL) has emerged as a form of disposal and a conscious way to use products at the end of their use through the refurbishment, reuse, and recycling processes. Mainly in developing countries, there are still several barriers during the process of implementing reverse logistics, so decision-makers need to know what barriers to prioritize to propose a solution. This paper aims to investigate which multicriteria decision aid (MCDA) methods are most applied in the studies on barriers during the implementation of reverse logistics. For this purpose, we conducted a systematic literature review using a well-known and detailed protocol, considering various papers published in journals from the databases Science Direct, Emerald, Web of Science, and Google Scholar in the period from 2008 to 2019. As a result, a few studies discussing multicriteria methods were identified, analyzing the existing barriers to the implementation of reverse logistics. The main methods and criteria were gathered from the literature and were systematized and discussed to identify a research agenda. This paper contributes to the work carried out by researchers and practitioners in this field and can guide them in their choice of methods to approach decision-making related to reverse logistics.

1. Introduction

The discussions related to poor and deficient waste management have increased in the last decades, mainly in developing countries. As a result, some countries have passed legislations related to waste management and reverse logistics. These legislations aid to reduce the improper disposal of solid waste by imposing some restrictions and sanctions on companies and final consumers. The amount of waste is recognized as an environmental problem to be solved, mainly related to types of waste with a high level of toxicity, such as the residues of electronic equipment. In addition, it is estimated that the growth of e-waste will be between 200% and 500% in key emerging markets, such as Brazil, China, India, and South Africa in the next few years, as reported in a study from the United Nations Environment Program (UNEP). According to the report “The Global E-waste Monitor 2017,” 1.5 million tons of electronic waste (e-waste) was generated in Brazil during 2016 (7.1 kg per inhabitant). This amount is similar to that of other developing countries like Indonesia, India, and Russia [1]. Even among developed countries that are part of the European Union, the generation of waste of electrical and electronic equipment (WEEE), or e-waste, is high, although the EU has guidelines on restricting the use of hazardous substances in consumer electronics [2]. For example, in 2016, France generated 21.3 kg of waste per inhabitant, or 1.3 million tons in total, and Germany generated 1.88 million tons (22.8 kg per inhabitant).

The lack of electrical power and inadequate management of e-waste is causing the annihilation of some raw materials, mainly precious metals, which are scarce in the environment and valuable [3,4]. WEEE contains substances such as aluminum, barium, lead, and mercury, among others, all of which are carcinogenic [3,5]. The use of these types of substances entails exposure to three possible types of risks: the contamination of consumers, contamination of the environment, and an increased contribution to global warming [6].

Furthermore, this increase has enabled the emergence of reverse logistics as a means to manage the return of waste or, in the last case, disposal in an adequate place [7]. Reverse logistics (RL) is a process which recaptures the value of a used product or finds an appropriate final destination for it (post-consumption), or enables the product not to be needed in the first place (post-sale), aiming at the proper destination of the main operation of RL [8].

Some developed countries have already developed legislation related to waste management; however, this is not the case for developing countries, where implementation of reverse logistics is currently in a state of infancy [9]. In Latin America, for example, Brazil was the first country to implement waste management legislation. The Brazilian Policy of Solid Waste Management (BPSW) instituted the principle of shared responsibility for a product’s entire life cycle and made the implementation of reverse logistics mandatory to make it feasible [10]. However, the implementation of an expanded responsibility to the product by mandating processes such as electronic waste management is still seen as costly, and it has various barriers [11]. These barriers in the management of e-waste can be classified [12] as: economic and financial, legal, environmental, policy (rules), management, knowledge, market, technique, and technology-related.

There are many topics of study related to reverse logistics being developed, mainly in the last decade. Some of them deal with the decision to make in-house or outsource reverse logistics activities, while others prefer to select the most appropriate third party logistics provider to carry out these kinds of activities; some choose to send relevant products to the most appropriate destinations (recycling, reconditioning, remanufacturing, incineration, composting, or still, final destination), while others set up sectoral agreements to make the implementation of reverse logistics feasible [11,12,13,14].

Nevertheless, few studies deal with another type of problem related to reverse logistics, which arises when the implementation of reverse logistics faces some restrictions that prevent it from being successful, mainly in developing countries. This fact occurs considering that the context of decisions in developing countries is embedded in a lack of resources and conflicting interests. This category is classified as the study of barriers related to reverse logistics. Very often, the barriers are identified and classified by the level of prioritization through a multiple criteria decision aid (MCDA) method [15]. In addition, some authors have identified the main barriers related to reverse logistics and have provided a ranking of solutions aimed at overcoming the barriers [16], also through a MCDA method. However, in a systematic review of the application of multicriteria decision support modeling on reverse logistics, it was highlighted that the use of MCDA methods is still very limited in the context of reverse logistics and more attention is needed [17]. A study from the UNEP also listed some barriers to the transfer of sustainable e-waste recycling in South Africa, Morocco, Colombia, Mexico, and Brazil, such as: (i) a lack of policy and legislation, (ii) technology and skills, and (iii) business and financing. The main barrier in these countries is the charging of a fee to encourage or fund the reverse logistics process, which is considered a very unpopular measure [13].

Considering the above-mentioned, this study aims to analyze the current literature framework of the application of multicriteria decision aid methods (MCDA) in studies related to barriers during the implementation of reverse logistics through a systematic literature review, following the protocol proposed by Cronin, Ryan, and Coughlan [18]. With this study, we intend to answer the question: What are the main methods used in the studies of the barriers encountered during the implementation of reverse logistics, according to international literature.

The main contribution of this paper is the systematization of the knowledge related to the prioritization of barriers in adopting reverse logistics under the perspective of the MCDA approach, which can aid researchers and practitioners acting in this field. The results gathered from specialized well-known literature can also be useful to aid policymakers (mainly in developing countries) to propose policies to enable the full adoption of reverse logistics.

This article is structured into five sections. Section 2 presents the theoretical framework with the main theoretical constructs: reverse logistics, barriers of reverse logistics, and multiple criteria decision aids. The research method and the protocol used are described in Section 3. The results and discussion are presented in Section 4, and finally, in Section 5 are the conclusions, limitations, and indications of future studies.

2. Theoretical Background

2.1. Reverse Logistics

Reverse logistics (RL) is in the management of return of products after use, both in the post-sale (trade disagreement, excess inventory, and others) and post-consumption (industrial waste, end-of-use products, and others) situations [7]. This return is made possible by increasing the quantities and variety of products that must be processed or returned at the end of their life cycle, in accordance with environmental requirements [19,20].

Many developed countries have implemented reverse logistics processes to deal with several types of waste. However, in developing countries, the full implementation of reverse logistics is not a reality yet [12]. In Brazil, after 20 years of discussion in the National Congress, the legislation related to the Brazilian Policy of Solid Waste Management (BPSW) was passed in 2020. The BPSW is the instrument of economic and social development aiming at the recovering of solid waste for its reuse or for its proper disposal. This legislation determines the implementation of reverse logistics for several types of residues as mandatory, such as tires, lamps, batteries, lubricant oils, agrochemicals and their packaging, electro-electronic equipment and its residues, and packaging in general [10]. The implementation of an environmental regulation is seen as important, considering that national regulations have a strong impact on the performance of WEEE collection, as identified by Isernia et al. [21].

Even recognizing that the BPSW can be considered as an advance of Brazilian legislation related to waste management, during its 10 years of existence, few actions have been implemented. Some agreements have enabled the socio-productive insertion of waste pickers and the closing of dumps. In addition, some other sectoral agreements are in the process of implementation, such as those related to general packaging, lamps, and more recently, related to WEEE. The sectoral agreement related to reverse logistics of WEEE was signed in December of 2019 [12]. The preferential destination of waste in the BPSW follows the well-known sustainability hierarchy, in which first of all, it would be better try to reduce the consumption; secondly, the reuse; thirdly, the recycling; fourthly, the incineration for generation of energy; and finally, when there is no other possibility of revalorization, the final destination. Thus, when properly destined, the product at the end of its life cycle can be reused, recycled, reconditioned, and/or recovered [10,20].

The waste can be collected for associations (cooperatives) of waste pickers of recycling materials, thus promoting the social inclusion of vulnerable workers who are devalued in the process [12,22,23]. In developing countries, such as in Brazil, the presence of waste pickers is responsible for about 90% of recycling materials of households, [13,22,23]. However, besides the inclusion of waste pickers, during the implementation of reverse logistics, several other barriers cross the various dimensions to be faced by the actors in the supply chain involved [24].

Whether organizations face complications and barriers during the implementation of reverse logistics, the total quantity of waste generated cannot be reused or have recovered value at the end of the product’s life [25]. Many managers consider reverse logistics as being the last phase of the product’s life cycle. However, RL should be considered for the entire life cycle of the product, including as part of the product design process, to be operational. The consideration of RL in the earlier processes will enhance the recycling and reuse possibilities [13]. This idea is similar to the circular economy concept, from which reverse logistics can be considered as a part, from the point of collection of waste. The barriers of RL can be grouped into nine different types, namely [12]: (i) economic and financial, (ii) legal, (iii) environmental, (iv) policy (rules), (v) management, (vi) knowledge, (vii) relations of the market, (viii) technique, and (ix) technology.

Besides, some other authors have classified barriers according to four main factors (management, financial, infrastructure, and policies employed). Thus, it is clear that the barriers vary among authors due to the perspective analysis [9,15,26,27,28] and the product recovery (electronic, automotive oil). Analyzing the critical factors for the success of the implementation of reverse logistics practices [25], some authors have identified that understanding and analyzing is important for developing strengths and competitive advantages, overcoming most parts of highlighted barriers, and offering solutions.

2.2. Multiple Criteria Decision Aid—MCDA

A multiple criteria decision problem is summed up as a situation that involves at least two criteria and two alternatives to choose from, and this decision is driven by the desire to satisfy several objectives, which may have conflicting relationships [29,30]. Thus, the multiple criteria decision aid (MCDA) or multiple criteria decision making (MCDM) comprises the analysis of decision problems presenting different points of views to be considered. The results do not aim at presenting only one unique solution for decision-makers, which optimizes or dominates all the others, and consequently stands in opposition to the optimization concept commonly used in hard operational research (OR) approaches [29,30].

Methods from operational research (OR) can be set in two main categories: (i) hard OR and (ii) soft OR. In general, the hard OR uses mathematical programming as the core of its methods. It considers the concept of optimization, in which a dominant solution dominates all the possible others. In the case of soft OR, it is possible to consider the imprecision and subjectivity of the judgments and perceptions from decision-makers. For this purpose, the soft OR uses mainly the problem structuring methods (PSM). The main methods from PSM are: value focused thinking (VFT), strategic options development analysis (SODA), soft system methodologies (SSM), and others [18,31].

On the other hand, the main approaches from MCDA are: (i) multi-attribute utility methods (American school), (ii) outranking methods (French school), and (iii) iterative methods [30]. The outranking approach arose in opposition to the American school, which used utility methods of multi-attributes [30]. The outranking approach presupposes that, not always, the decision-maker is willing to make a decision. Sometimes, the decision-maker even does not have knowledge on all decision elements. In this type of situation, relationships of incomparability are accepted [30]. These methods also can provide more balanced decisions since they do not include trade-offs (compensation) between criteria [18,31,32].

Table 1. Description of multicriteria decision support approaches.

Approaches	Description	Methods
Multi-attribute Utility	Preferences are aggregated into a unique utility value, which is additively measured (with trade-offs). A score of each alternative is generated based on the performance presented on each criterion.	MAUT, SMART, TOPSIS, AHP, TODIM, DEMATEL
Iterative	They are characterized by having computational steps and being interactive, allowing trade-offs between alternatives.	STEM, TRIMAP, ICW, ISM
Outranking	It has controls for binary relationships representing preferences between criteria and does not have score alternatives (without trade-offs between criteria).	ELECTRE and PROMETHÉE, CPP

Source: Adapted [30].

presents the description of the approaches and the most applied method in each one.

The choice of the method depends on four factors: (a) problem, (b) examined context, (c) decision-makers´ essential preferences, and (d) problematic involved [30]. There is no consensus among several authors on the best method to be employed [31,33]. Besides that, the rationality (whether compensatory or non-compensatory) of the decision-maker should also be considered when choosing the method, allowing trade-off relationships within the criteria or not [31,33].

To understand the preferences of decision-makers, it is necessary to assess their subjectivity and individual perceptions, besides identifying the context of the decision, the existence of other stakeholders, and the common or conflicting points of view [30]. The rationality of decision-makers can be additive, allowing trade-offs between criteria, or non-additive, in which trade-offs are not allowed. In this case, instead of a score, alternatives are evaluated based on strong preference, weak preference, indifference, and incomparability relations [31,33], which are characteristic of the outranking methods. There are still types of problems requiring a specific MCDA problematic to deal with them; in general, they can be categorized as: selection (decreasing to a smaller set of alternatives), ranking (up or down), sorting (aggregating and sorting alternatives in groups by similarity), and description (describes the problem in detail) [33].

3. Materials and Methods

3.1. Research Characterization

This was a descriptive and qualitative study, considering that the intention was to describe and analyze the knowledge body related to the prioritization of barriers of reverse logistics under the perspective of a MCDA approach [34]. Regarding the objective of the research, the methodological framework is characterized as exploratory because of its investigative nature and the purpose of detailing the characteristics of the papers analyzed. Regarding the technical procedure, the paper used a systematic literature review (SLR). There are two main types of literature review, the narrative one, in which the criteria to select and filter the papers under analysis are not described, and the systematic one, in which the criteria chosen are described in detail to allow its replicability [18].

Systematic literature reviews (SLR), in general, follow a protocol to describe the criteria used to select and filter the papers. In this paper, the protocol of Cronin, Ryan, and Coughlan was chosen [18]. The steps of this SLR are detailed in the following section. As a technique of analysis of the papers selected, we used the technique of content analysis proposed by Bardin [35]. This technique systematizes the results in categories, which in this study were made “a posteriori” [35] to detail and demonstrate: (i) instruments and methods used in the studies, (ii) distribution of studies by country, (iii) multicriteria decision aid methods identified in the studies, (iv) prioritized barriers, and (v) main research gaps found.

3.2. Research Procedure

The protocol defined by Cronin, Ryan, and Coughlan [18] consists of five steps: (a) formulation of the research question; (b) establishment of inclusion and exclusion criteria; (c) selection and access to literature; (d) quality assessment of literature; and (e) analysis, synthesis, and dissemination of results. These steps are described as follows.

a. 

Formulation of the research question: What are the multicriteria decision aid methods applied to prioritize barriers in the implementation of e-waste reverse logistics in an international scope?

b. 

Establishment of inclusion and exclusion criteria: The criteria included the scientific databases used, the time frame, types of articles, keywords, and Boolean operators used. We delimited the search based on: Science Direct—it has more than 769 periodicals in the fields of social sciences and humanities. Emerald: it has more than 290 periodicals, books, and online resources, Web of Science: it presents over 12,000 periodicals and provides access to references and abstracts in all fields of knowledge, and Google Scholar: it has wider access to articles. The period of publication of articles was defined as eleven years, between 2008 and 2019. The keywords used were “reverse logistics,” stakeholders, barrier*, “multiple criteria” and its synonym multicriteria using the Boolean operators, as shown in

Table 2. Search results organized by database and keyword combination.

Keywords	Database
	Science Direct	Web of Science	Emerald	Google Scholar
“reverse logistics” AND stakeholders AND barrier*	Result: 435 Time: 2008–2019 Types of articles: research articles, data articles, replication studies. Title, abstract and keyword reading: 15	Result: 20 Time: 2008–2019 Filter: English, Title or Topic Title, abstract and keyword reading:4	Result: 395 Time: 2008–2019 Filter: All Emerald content, All field and All content Title, abstract and keyword reading: 9	Result: 1.500 Time: 2008–2019 Filter: English, do not include, patent do not include citation Title, abstract and keyword reading: 15
“reverse logistics” AND stakeholders AND “multiple criteria” AND barrier*	Result: 33 Time: 2008–2019 Types of articles: research articles, data articles, replication studies. Title, abstract and keyword reading:4	Result: 6 Time: 2008–2019 Filter: English, Title or Topic Title, abstract and keyword reading: 0	Result: 19 Time: 2008–2019 Filter: All Emerald content, All field and All content Title, abstract and keyword reading: 0	Result: 479 Time: 2008–2019 Filter: English, do not include, patent do not include citation Title, abstract and keyword reading: 5
“reverse logistics” AND stakeholders AND multicriteria AND barrier*	Result: 12 Time: 2008–2019 Types of articles: research articles, data articles, replication studies. Title, abstract and keyword reading: 2	Result: 0 Time: 2008–2019 Filter: English, Title or Topic Title, abstract and keyword reading: 0	Result: 0 Time: 2008–2019 Filter: All Emerald content, All field and All content Title, abstract and keyword reading: 0	Result: 1.500 Time: 2008–2019 Filter: English, do not include, patent do not include citation Title, abstract and keyword reading: 15

. Studies using “green supply chain management,” “green supply chain,” and “circular economy” were not included because they are outside of the scope of our study. Regarding the type of articles, we selected research articles, data articles, and replication studies; thus, we excluded review articles, books, and book chapters.

c. 

Literature selection, filtering, and access: In all databases, three exploratory searches were performed: in the first, we used the combination of “reverse logistics” AND stakeholder AND multicriteria AND barrier, thus obtaining two articles in Science Direct and 15 articles in Google Scholar; in the other bases (Emerald and Web of Science), the combination did not identify any papers. Then, by carrying out a more specific search, we obtained two articles in Science Direct and five in Google, while in other bases, the combination did not identify any papers. Then, we chose to use the synonym “multiple criteria”, finding 4, 6, 1, and 5 articles, in Science Direct, Emerald, Web of Science, and Google Scholar, respectively. Finally, encompassing the search with “reverse logistics” and stakeholders and barrier*, we found 13, 9, 4, and 15 articles, respectively.

d. 

Evaluation of the literature quality: In Science Direct, 21 articles were identified: five were duplicates, and thus the number remaining for analysis was 12. By reading the introduction and conclusion, 12 of those articles were eliminated, as the main focus of the study was not related to barriers of RL. In Emerald, nine studies were selected, but six were excluded because they did not apply MCDA, and two applied MCDA in “green supply chain management,” resulting as “out of scope.” We excluded two studies found in Web of Science for the same reason. In Google Scholar, 35 articles were selected, but 26 were duplicates and three did not apply the MCDA approach (“out of scope”). Therefore, an amount of 10 articles from the three databases were fully analyzed.

e. 

Analysis, synthesis, and dissemination of results: The last step of the protocol consisted of a detailed analysis of the 10 articles focusing on the authors, the journals, the concepts covered, the methods, the period of publication, the suggestions, and the main results. Later, the results were compiled in graphs and tables presented in the results section.

Table 2 shows an overview of the protocol used in the systematic literature review, allowing us to know the details obtained in each search.

4. Results and Discussion

This section presents the results obtained from the systematic literature review. Initially, it shows the characterization of the studies, the methods of multiple criteria decision aid applied in studies, and, finally, the research agenda.

4.1. Description

The majority of the five studies analyzed, three of them, were published in 2018. The journals that published the studies were: Computers and Industrial Engineering (1), International Journal of Manufacturing Systems (1), Sustainable Production and Consumption (1), Granular Computing (1), Electronic Commerce Research (1), Annals of Operations Research (1), Global Journal of Flexible Systems Management (1), Journal of Cleaner Production (1), and Resources, Conservation and Recycling (2). The latter concentrated the publications, publishing two articles from the same group of authors.

Regarding the methods, the instruments for data collection used were documentary surveys, questionnaires, and interviews; some studies used more than one method and one study did not report which method was used. The analysis techniques were content analysis (5) and combined statistics (4). Most studies used more than one analysis technique, characterizing mixed approaches. The combination of the analysis techniques can be explained by the fact that the ten studies started with a literature review to identify the barriers, validating them and checking their levels of importance through questionnaires, then applying the multicriteria modeling to prioritize the barriers, as shown in Figure 1.

As shown in Figure 1 and also Figure 2, the countries that applied multiple criteria decision aid methods in studies on barriers to implement reverse logistics were Bangladesh [36,37,38,39], Brazil [15,29], India [27,40,41], and Thailand [16,42]. Figure 2 shows the number of publications from each country. The countries in gray did not present publications, considering the results of this review.

As seen in Figure 2, Bangladesh and India have the most concentrated of the studies. All studies analyzed were conducted in developing countries as their locus and all used the ISM method. The study by Abdulrahman, Gunasekaran, and Subramanian [9] used as justification for conducting studies in these countries the fact that most previous studies have been concentrated in developed countries [15,16,27,29]. There was also a repetition of the citation of the studies conducted by Zhang et al. and Sarkis et al. to show that studies on reverse logistics in developing countries are not surprising because they are in the early stages of reverse logistics implementation.

Several studies have also used, in their theoretical base, the concepts of reverse logistics citing the authors Rogers and Tibben-Lembke [9,15,16,27,29,41], which is justified considering that it was the first book published on this subject. Only Bouzon, Govindan, and Rodriguez [28] introduced the theory of stakeholders in their references, thus demonstrating a lack of the use of organizational theories on studies related to reverse logistics barriers.

4.2. Multicriteria Decision Aid (MCDA) Methods

The MCDA methods, as previously mentioned, are derived from the soft operational research (OR) area, which is considered as an alternative to the logic of the hard OR. Methods from the hard OR are considered well-structured, the variables are well-known, and a single “best solution” dominates all the other possible solutions derived from the calculations. In addition, the methods from the hard OR are concentrated in the use of mathematical programming and its variations. On the other hand, the MCDA methods are not based on mathematical programming because they do not use the concept of optimization; instead, the approach is aimed to ill-structured decision problems (the variables are not known) and it presents a set of adequate solutions instead of a single “best solution.” The main reason is because the MCDA methods consider the inherent subjectivity and hesitation present in the decision-making process, while the hard OR is classified as more objective and based on rational decisions [13,41]. Variations in the multicriteria methods identified were grey-DEMATEL [28], interpretive structural modelling (ISM) [38,40], and fuzzy—ANP [40]. There were also fuzzy Delphi and analytic hierarchy process—AHP combinations [15], and studies with the same fuzzy AHP-TOPSIS combination from different authors [16,27]. Studies also applied only the combination fuzzy-AHP [37,41] and SWARA and WASPAS [37].

Table 3. Multicriteria decision support methods identified in the studies.

Method	Goal	Focus	Reference
Interpretative Structural Modeling (ISM)	Applied to identify the interrelationship between barriers and model them to identify the most significant barriers.	Automotive oil	[38]
	Analyze the relationship and interactions between barriers.	Computer network	[42]
Decision Making Trial and Evaluation Laboratory Method (DEMATEL)	Combined with the grey theory to investigate the causal relationship between barriers.	Electro-electronics	[28]
Analytic Hierarchy Process (AHP)	Applied to prioritize barriers. Used with the fuzzy method to determine weights, decision-makers’ preferences.	Electro-electronics	[15,16,26]
	Identifying the most common barriers and assessing them in the context of the country.	Leather footwear industry	[36]
Analytic Network Process (ANP)	Propose a method that has strategies to overcome barriers.	Palm oil industry	[40]
Technique for Order Preference by Similarity to Ideal Solution (TOPSIS)	Combined with the fuzzy method to generate a ranking of the proposed solutions.	Electro-electronics	[16,26]

presents the methods used, their purpose in the applied study, and the focus.

As shown in the framework, ISM is one of the interactive methods and the AHP, ANP, and TOPSIS are characteristic of the single synthesis criterion method. As a result, no outranking methods were found in studies of barriers in the implementation of RL. Regarding the problem, the studies decided to use ranking when presenting the barriers in importance ranking and/or sorting when classifying the barriers as structural, financial, or political, among others.

Decision-making trial and evaluation laboratory (DEMATEL) is more versatile than ISM because it has greater insight into the measures used and allows the identification of a two-way relationship [28]. However, two studies applied ISM and only one the DEMATEL. The DEMATEL was combined with the grey theory to remove the uncertainty caused by the lack of information. The DEMATEL method was also used to construct a network of interrelationships among decision criteria, which can result in the identification of the most influential criteria to be employed [43].

The studies used the combination of MCDA methods and fuzzy logic to reduce the inaccuracies and uncertainty in decision-making and the presence of subjectivity [16,27,37,41,42]. The combination of fuzzy-AHP and fuzzy-TOPSIS improves the multicriteria decision making process by integrating more than one method [26]. This is corroborated by the presence of indication of solutions for overcoming the barriers of RL that were considered a limitation in other studies by only presenting the prioritization of barriers without proposing a solution [15].

The other study applying AHP used it after using fuzzy–Delphi to identify the main barriers [15]. By analyzing the focus, the three studies applying AHP analyzed electronic waste. There are mainly three reasons for this: (1) environmental policies that make reverse logistics of WEEE mandatory in Brazil and the approval of the Brazilian Policy of Solid Waste, (2) the importance of electronic equipment in the world, and (3) increased generation of electronic waste [15].

Two studies combined MCDA methods with other methods, as they wanted to test the model. Phochanikorn et al. [40] applied VIKOR (Vise Kriterijumska Optimizacija I Kompromisno Resenje) to assess the different standards of various competing projects, such as changes in weights between evaluations.

As presented previously, the Brazilian Policy of Solid Waste (BPSW) has made RL mandatory for electronics and other kinds of waste [10]. In the legislation, there is no definition of WEEE; however, the Brazilian RoHS Working Group created by the Ministry of Environment is in the process of developing a regulation that determines the restrictions of hazardous materials present in electronic equipment (EEE), using as a basis the directives of the European Union and the Brazilian legislations, and the equipment that will be covered by the normative [44].

The studies applied MCDA by analyzing a group of decision makers from more than one perspective, as was the study by Bouzon et al. [28], who analyzed it from the perspective of government, organization, and the final consumers.

As demonstrated, some methods were applied in different countries and with different combinations.

Table 4. Prioritized barriers in each multicriteria decision aid (MCDA).

MCDA	Main Barrier (Top 1#)	Country
Grey-DEMATEL	Organizational	Brazil
Fuzzy AHP-TOPSIS	Technological	Thailand
	Infrastructural	India
Delphi fuzzy (FDM) and AHP	Economy	Brazil
Fuzzy ANP and VIKOR	Market related	Thailand
Fuzzy AHP	Economy	India
	Knowledge and support	Bangladesh
SWARA and WASPAS	Organizational	Bangladesh

shows which categories of barriers (economic, organizational, technological) were identified as priorities and in which country the study was applied. That way, we can compare the methods.

As seen in Table 4, the results differ between the methods. In addition, the same method generated different priorities, such as fuzzy AHP-TOPSIS, that was applied in the context of Thailand and India, and can be justified with Bouzon et al. [15], which shows that the context can influence the priority of each barrier.

The nomenclature and categorization of barriers may differ between studies, which could generate different results, such as the studies by Prajapati et al. [37] and Phochanikorn et al. [40], who applied two different methods and combinations in addition to different focuses (electrical manufacturing industry and palm oil industry), both in Bangladesh, and the main barriers were of different categories. However, when analyzing the barriers of the study by Phochanikorn et al. [40], it is possible to observe that the barrier “lack of interest and support from high-level management” can be classified as an organizational or management barrier, demonstrating similarities. It can be inferred that the same results, even though they are different combinations, come from the focus: industries in Bangladesh.

As Abdulrahman et al. [9] indicated the barriers can be related to internal and external factors in organizations, what stands out when analyzing Table 4 is that the main barriers are internal factors.

4.3. Main Research Gaps Based in RSL

Based on the suggestions of future studies, each article that presented a research agenda was elaborated, and suggestions were categorized by method, weights, sample, and locus, as shown in

Table 5. Research agenda based on the studies analyzed.

Focus	Gaps	Reference
MCDA Method	- Exploration of the study by Prakah and Barua using other approaches: ANP electre, PROMETHE, and approximate set theory; - analyzing list of barriers proposed by Bouzon et al. using different tools from MCDA; - using structural equation modeling to validate the results of the ISM approach; and - reapplication of the study of Sirisawat and Kiatcharoenpol using other methods of multi decision making criteria such as fuzzy Vikor, fuzzy ANP, and fuzzy ELETRE.	[15,16,26,38,39,40]
Weights	- Apply the study by Gardas et al. [38] in other sectors since the weights of specialties vary significantly from one sector to another. - Apply the model by Phochanikorn et al. [40] in other industries and even in other sectors to later carry out a comparison between studies.	[38,40]
Sample	- Include other stakeholder perspectives, such as customer perspectives or investor perspectives to analyze RL barriers. - In addition, include managers’ views regarding the most important barriers to RL that can be controlled and managed by companies; research on RL barriers should include further consideration of small and medium enterprises (SMEs). - Identify the incentive factors to implement an RL strategy for the leather footwear industry using the total interpretative structural modeling (TISM) approach	[15,28,40]
Locus	- Research empirical studies to investigate knowledge about the influential forces and the implementation of RL processes in emerging economies. - Examine and compare the contexts of different companies using data envelopment analysis (DEA). - Continue the study by Prajapati et al. with electrical industries in other regions and make a comparison.	[28,37,39]

.

As seen in Table 5, most authors indicate more studies applying MCDA, which reinforces the findings of our research. There are a few studies prioritizing the barriers of RL, and when changing the country of application of the study, the results can change, as shown in Table 4. Therefore, it is possible to perceive that most authors suggest study of the prioritization of barriers considering other methods from the MCDA approach, and the observation of other industries and sectors.

We corroborated these findings, highlighting that the application of the methods from the outranking approach does not allow the trade-offs between criteria. This is an important point to be stressed mainly because the studies related to reverse logistics involve conflicting objectives, such as the reduction of costs and the maximization of environmental preservation. Clearly, these two specific objectives have different directions, and while the preservation of the environment is increased, the organizations had to increase their costs, which did not meet the objective to reduce costs. In this context, studies applying methods from Electre and Promethèe approaches can be useful because they can provide more balanced results. Another aspect related to the application of the classification methods refers to the consideration of the inherent subjectivity involved in the decision-making process, in which some decision makers cannot feel comfortable establishing discrete values to the alternatives under the evaluation of each criteria. Some of these methods use fuzzy logic to allow a range of evaluation in an interval of the values, which also allows the consideration of the incomparability between alternatives. As such, future studies can adopt these kinds of approaches in the analysis of prioritization of barriers to implement reverse logistics.

Another venue for future studies can be the use of MCDA methods prioritizing barriers of reverse logistics in the analysis and proposition of public policies to overcome the bottlenecks and challenges. Studies with systematized methods dealing with barriers to implement reverse logistics can give some input to policy makers in public and private organizations, since they consider several elements of the decision-making process.

Study related to barriers to implement reverse logistics in developing countries in comparison with the best practices of developed countries can also be an opportunity for future studies. This kind of analysis can bring light to the discussion, verifying whether the barriers prioritized in developing countries are aligned with the actions implemented in them. This can be helpful to policy makers, managers, researchers, and practitioners acting in this area.

Finally, further studies can also analyze the implementation of reverse logistics in developing countries under the perspective of some organizational theories, such as institutional theory and stakeholders theory to analyze if the rules, beliefs, and concerns from the external environment in which the organizations are inserted motivate the adoption of measures to overcome the barriers. Another theory that can be used is the stakeholders theory, which can enable the analysis of conflicting points of view and interests of different stakeholders involved in this context of decision-making, such as governments; private companies (manufacturing, retailers); recycling companies; third party logistics providers; consumers; community; research centers; final consumers; waste pickers; and others.

5. Concluding Remarks

By analyzing the methods used in the articles published in international journals, a low number of articles applying any of the multiple criteria decision aid methods was found, corroborating the study by Rezaei [17]. The authors already highlighted the limited use of MCDA methods in reverse logistics. Although there is a high number of articles dealing with reverse logistics barriers, in this study, they were excluded because there is no application of MCDA methods. In addition, some studies indicated the use of MCDA methods as research gaps.

We verified that all studies conducted a literature review to identify barriers as a first step, then, the most used instrument found in the studies was the questionnaire, used in order to validate the barriers identified with experts in the field, and only then, applying the multicriteria methods to prioritize the barriers. There is also a predominance of a combination of methods in four studies using the fuzzy theory or grey theory to remove the impression or lack of information [15]. The application of fuzzy occurred due to the presence of inaccuracy, subjectivity, and vague information during the barrier verification process. Thus, there are ways to use MCDA methods when there are context-generated inaccuracies.

Based on the literature review, we identified only studies in developing countries (Brazil, Bangladeshi, India, and Thailand), which can be explained because reverse logistics is in its early stages in these countries, considering the lack of infrastructure of some countries. As a consequence, the barriers are still present, as found in Brazil, considering that the legislation making reverse logistics mandatory was passed just in 2010 [10,18].

Despite the contributions, we should highlight some limitations of this paper. As shown in the protocol, the review was limited to articles published in the period from 2008 to 2019 in the databases Science Direct, Emerald, Web of Science, and Google Scholar. Therefore, other criteria considered in the protocol can result in different findings. This study also did not address the terms “green supply chain management,” “green supply chain,” and “circular economy,” but we recognize that they are broader terms and maybe could cover some discussion related to reverse logistics; however, not as a main focus. As presented by Ritzén and Sandström [45], RL can be a barrier to the circular economy.

Future studies could investigate the barriers to the implementation of the concept of the circular economy and green supply chain management, and increase the databases for Scopus, Springer, and Taylor and Francis, who also publish articles in the fields of reverse logistics. Studies on the barriers of RL with empirical data could apply some method from the outranking approach, such as ELECTRE and PROMETHEE, or non-compensatory methods such as CPP to prioritize the barriers of reverse logistics, or still, other additive and compensatory methods such as AHP or TOPSIS.

The main contribution of this paper is aimed to researchers and practitioners in this field. By synthetizing the main studies focusing on barriers of reverse logistics under a perspective of MCDA, this study can be used as a guide in the choice of methods to approach this decision-making context. In addition, it can be useful for policymakers to propose some solutions to overcome these barriers.