How to Assess Reverse Logistics of e-Waste Considering a Multicriteria Perspective? A Model Proposition

Abstract

As e-waste exponentially increases on a global scale, some legislation and plans to implement reverse logistics have arisen in some countries. While reverse logistics is already well consolidated in developed countries, it is still in a state of infancy in developing countries. In this article, we propose a theoretical model to assess reverse logistics, based on a multicriteria decision aid perspective. This was done while considering a context in which the implementation of the reverse logistics of e-waste is at the beginning. To show the main characteristics of the literature and obtain the model’s inputs, we conducted a systematic literature review to help us understand the main criteria and methods used for this purpose. Then, we illustrated the use of the proposed model with a numerical example. This paper contributes to the current body of research, and aims to assist practitioners within the field of reverse logistics by systematizing the knowledge related to this issue, and offering a model that can guide the decision-making. This model differentiates from others already published because it uses the probabilistic composition of preferences (PCP) method, which does not require the assignment of weights. The model can also be used while considering the preferences of several decision-makers, therefore covering several points of view and providing a more holistic view of the decision problem.

1. Introduction

People from all over the world are increasingly being integrated into the information society and the digital economy, and benefiting from the opportunities arising from it. Examples of this include the delivery of new applications and services at very high speed in a wide range of areas from health and education to entertainment and commerce [1].

This integration is possible because of technological development and its consequent diffusion as a result of the efforts of large corporations to become global. Despite the undeniable advances that this brings, the same development is used as an instrument of consumption control. This can be observed in the increase in the consumption levels of durable goods such as electronic equipment [2].

As a result, the generation of residues from electro-electronic equipment (commonly referred to in the scientific literature as waste electrical electronic equipment (WEEE, or e-waste) has exponentially increased worldwide [3,4,5]. This specific type of waste corresponds to 5% of all urban waste produced worldwide, with the highest growth rate among the other categories [6]. For standardization, we adopted the term ‘e-waste’ in this paper.

In response to the scenario of increased waste generation, and aggravated by the toxicity of its components, governments in several countries have sought legal and regulatory mechanisms to transfer their responsibility of mitigating the environmental impact of e-waste. An example of this mitigation is through management tools that aggregate all links of the productive chain [7].

The purpose of reverse logistics is to manage the flow of goods (from the point of consumption to the point of origin). Its objective is to recapture the item’s value or provide an adequate destination [8]. It appears to be a viable and sustainable measure, and complies with the pillars of enterprise sustainability from the triple bottom line (TBL) model (social, economic, and environmental), as proposed by Elkington [9].

The potential of reverse logistics is now widely recognized, and has become part of the environmental agendas of governments and companies. Thus, this new area of multidisciplinary studies was formed, and strives to understand its nuances, and develop tools and strategies for applying reverse logistics to the management of e-waste.

Given the above, it has become essential to understand the level of discussions about the subject, identifying gaps, advances, and good practices that contribute to the consolidation of effective and effective management techniques. This paper proposes a theoretical model to assess the implementation of the reverse logistics of e-waste, based on a multicriteria decision aid (MCDA) perspective. The methodological procedures to achieve this objective were two-fold: (i) a systematic literature review was conducted using the protocol proposed by [10] to show the main characteristics of literature, and obtain the model’s inputs (such as the main criteria, alternatives, and methods used for this purpose); and (ii) modeling was used to illustrate the use of the proposed model with a numerical example, considering a probabilistic composition of preferences (PCP) model [11].

This model differentiates from previously published studies because it uses the PCP method, which does not require the assignment of weights. This is one of the most contested steps by some researchers, because the weights tend to be arbitrary.

This paper contributes to the current body of research, and aims to assist practitioners within the field of reverse logistics by systematizing the knowledge related to the reverse logistics of e-waste. We considered multiple criteria and variables of decisions made in previous years, and offer a model that can guide decision-making in this context.

2. Literature Review

2.1. Reverse Logistics of e-Waste

The literature presents several definitions and concepts of what reverse logistics is. Reverse logistics is characterized by logistical management skills and activities, related to reduction, management, and disposal of waste [12]. Five factors lead to its application [13]: (I) Economic factors related to production costs, use of raw materials, and environmental costs; (II) governmental factors that involve legislation and policies related to the environment; (III) corporate responsibility, which translates into the collection of waste generated after the end of the useful life of the products; (IV) technological factors related to the advances of recycling and green design, where the products are designed to facilitate the reverse process; and, finally (V) logistic factors, which deal with reverse channels and other aspects.

Reverse logistics is defined as a process of planning, implementing, and controlling an efficient flow of low-cost raw materials, in-process inventory, finished product value, and related information from the point of consumption to the point of origin. It seeks to recover the value of the initial materials or facilitate the proper disposal of the product [8].

Additionally, reverse logistics is the process in which a manufacturer systematically accepts the return of previously forwarded products (or part of them) to recycle, remanufacture, or dispose of them [14]. Some authors have argued that reverse logistics includes the logistics activities of used products that no longer serve the users as well as the replacement of products still usable in the market [15]. For the same authors, reverse logistics of e-waste also involves a process of material recovery, since many of these wastes contain valuable and scarce metals.

The expansion of studies related to reverse logistics shows its current importance, aiming at aspects of efficiency and use of industrial processes as well as the maximization of consumer satisfaction. Thus, when the concerns related to sustainability and the environment in the generation and disposal of solid waste became an object of public concern, subject to regularization and specific regulations, reverse logistics gained more prominence [4].

e-waste covers the products of the disposal of electrical and electronic equipment, parts and pieces that have lost their value to their owners, reached the end of their useful life, or whose use has been discontinued [16,17].

Electronic devices are configured as equipment whose operation depends on electric currents or electromagnetic fields as well as the equipment for the generation, transmission, transformation, and measurement of these currents and fields, being able to be of domestic or industrial use [17].

The European Parliament’s Waste Electrical and Electronic Equipment (WEEE) Directive suggests that electronic equipment should be classified into ten categories: (i) large appliances; (ii) small household appliances; (iii) computer and telecommunications equipment; (iv) consumer equipment; (v) lighting equipment; (vi) electrical and electronic tools; (vii) toys and sports and leisure equipment; (viii) medical apparatus; (ix) monitoring and control instruments; and (x) automatic vending machines [18].

The special attention given to e-waste is mainly due to two factors: the existence of toxic substances in discarded material that can cause damage to the environment and human health if inappropriately arranged, and the presence of metals of high economic value in their composition such as silver, gold, and platinum [19], which are highly valued in the urban mining strategy.

In the search for mechanisms of treatment and revalorization of these residues, reverse logistics appears as a viable alternative, since it allows the recovery of the value and benefits both the company through the reduction of costs with the reuse of materials, for example, the environment, with the reduction of the extraction of raw materials and the depletion of the mineral resources of the planet [17,18,20].

2.2. Multiple Criteria Decision Aid (MCDA)

The multicriteria decision aid approach aims to provide decision makers with some tools to enable them to move forward in solving decision problems, in which several—often contradictory—points of view must be taken into account [21].

A multicriteria decision problem consists of a situation where there are at least two alternatives of action to choose. The process of the choice of the most adequate alternatives or courses of actions occurs through the desire to meet multiple goals that often have conflicting relationships [21,22,23]. These objectives have associated variables that represent them and allow each alternative to be evaluated based on each objective, which can be called criteria, attributes, or dimensions [22].

To identify the system of preferences of a decision-maker, first it is necessary to consider the subjectivity of the decision-makers, the individual perceptions [21,24]; second, it is necessary to structure the problem according to the decision-maker or the group of decision makers’ points of view [24,25]. It is important to point out that the multicriteria decision aid analysis process involves a few steps, which are presented below [24].

• Identification and structuring of the problem. Before any analysis begins, the various stakeholders including facilitators and technical analysts, need to develop a common understanding of the problem, the objectives, the decisions to be made, and the criteria by which those decisions are judged and evaluated;
• Construction and use of the model. A primary feature of the multicriteria approach is the development of formal models of decision-maker preferences, valuing trade-offs, goals, among others, to allow those alternatives and policies or actions under consideration to be compared and related to each systematically and transparently; and
• Development of action plans. The analysis does not solve the decision problem. All management science, and multicriteria decision analysis in particular, also concerns the implementation of results, translating analysis into specific action plans.

These steps should be considered to deal with the decision elements that involve the preference relations of the decision maker; the conflicting objectives; the criteria that represent them, when alternatives are evaluated; the rationality related to compensation or not between criteria, and the choice of multicriteria decision support approach to be used, which involves an extensive set of methods. These methods analyze the decision-making elements related to the consequences of the problem in question, providing an appraisal of the possible options considered in the decision-making process. Nevertheless, it is important to point out that there is no consensus amongst specialists in which the best method is to use, because this choice will depend on the features of the decision context approached [21,25]. Thus, the first movement is to know the characteristics of the decision problem and the preferences of decision-makers, to identify the elements of decision, and to make feasible the most appropriate method to choose.

3. Methods and Techniques

Considering the intention of analyzing how to assess the implementation of the reverse logistics of e-waste, considering a multiple criteria approach, we conducted two methodological procedures. First, to obtain the model’s inputs, we conducted a systematic literature review (SLR) to understand the main characteristics of publications on this topic and to get the main criteria and proper methods to use in this context; and second, multicriteria modeling was used to propose the model, which was illustrated with a numerical example based on inputs from the SLR.

In the next sections, we present the two procedures used in this study. Section 3.1 presents the protocol and procedures followed to conduct the SLR and Section 3.2 describes the multicriteria modeling steps.

3.1. Systematic Literature Review

The literature reviews presented two perspectives: (i) revisions of topics that have accumulated knowledge and require analysis and synthesis; and (ii) revisions of emerging issues, whose contribution is the exposure of potential theoretical foundations, which is usually not as extensive as the first. This research fits the first perspective [26].

There are two types of literature review: (1) narrative or traditional literature reviews, and (2) a systematic review of the literature [27]. According to the authors, the narrative review condenses the literature, but does not make explicit to the reader the selection criteria of the sources used. On the other hand, the systematic literature review is characterized by well-defined steps and structured criteria in the procedure of searching through and analyzing the information related to the investigated matter.

This process guarantees greater rigor, robustness, and replicability to the research [28]. The rigor of this procedure originated from the medical area, in which it is extensively used [29,30]. The Cochrane Handbook was the first publication to formalize this methodology [29,30]. In the area of social sciences, the utilization of the procedure of SLR is considered a recent phenomenon [31].

For the reasons exposed, the procedure chosen for this research was a systematic literature review (SLR). It is also important to emphasize that the SLR considers only papers with empirical results, excluding papers of literature review from the total analyzed.

Due to the increase of the number of publications, concerns have been raised regarding the choice of the most relevant sources to be included in the research and the criteria to eliminate those that are not relevant [10]. There are several RS protocols of SLR. For this paper, the protocol Methodi Ordinatio was chosen [10]. This protocol was selected because it presents well defined steps and a more rigorous procedure to select the papers for a SLR. The authors proposed an adaptation of Proknow—C methodology [32], which is based on a multicriteria decision aid (MCDA) approach. The protocol considers three criteria in ranking articles: the year of publication, the number of citations obtained, and the impact factor of the journal in which it was published. The methodology aids the researchers to select, collect, rank, and systematically read scientific papers published in journals. The Methodi Ordinatio employs an equation to rank papers, the Index Ordinatio (InOrdinatio—IO) [10].

$$\mathrm{InOrdinatio}=\left(\mathrm{IF}/1000\right)+\mathsf{\alpha }\ast \left[10-\left(\mathrm{ResearchYear}-\mathrm{PublishYear}\right)\right]+\left({\displaystyle \sum }\mathrm{Ci}\right)$$

(1)

where

-

IF is the impact factor, which is divided by 1000 (one thousand), aiming to normalize its value concerning the other criteria;

-

α is a weighting factor ranging from 1 to 10, to be attributed by the researcher. The closer the number is to one, the lower the importance the researcher will attribute to the criterion year, while the closer to 10, the higher is the importance;

-

ResearchYear is the year in which the research was developed;

-

PublishYear is the year in which the paper was published, and

-

Ci is the number of times the paper has been cited [10].

After the calculation, the InOrdinatio of each paper is obtained, and from this point, it is possible to rank the papers according to their scientific relevance: the higher the InOrdinatio value, the more relevant the paper. Thus, when the ranking is provided, the researcher can define how many papers they will search for the full version, according to the priorities of the research, which will be analyzed.

The intention of the research was defined as: Analyzing the state-of-art of papers on the reverse logistics of electronic waste considering multiple criteria, published in last ten years, and the following aspects: university of origin; countries involved in research; partnerships between universities and other organizations; research methods; objectives; main criteria approached; utilization of methods from MCDA, PL, and other approaches; main results; the main concepts approached; and suggestions for future research.

A series of databases for research was highlighted: Web of Science, Scopus, Science Direct Online, Emerald, Springer Link, IEEExplore, Academic Search Premier—ASP (EBSCO), Scielo, Sage Pub, Taylor & Francis, Oxford University Press, and Cambridge University Press [33]. Three scientifical bases were selected for this SLR: Science Direct, Web of Science, and Emerald. The bases cover the main leading journals in the social sciences and are multidisciplinary in nature. Another reason for choosing the three bases was that most journals covered contain an impact factor in the Journal Citations Report (JCR^®).

The keywords used in the search for papers were ‘reverse logistics’ AND ‘electronic’ OR ‘e-waste’ AND ‘multiple criteria’ OR ‘multicriteria’. The keywords were searched in all fields of the paper (title, abstract, keywords and text). In both bases, the period of the publication of the papers defined was 2007–2017. Regarding the type of documents, it was limited to “full articles”, published in journals, in the English language.

Considering the choice by the Methodi Ordinatio [10], only journals with impact factors were included in the systematic literature review, as proposed in this paper. Aside from that, the impact factor chosen was the Journal Citations Report (JCR^®), which is a base from the Thomson Reuters Publisher. The JCR is a resource for evaluating and comparing scientific journals using citation data from academic and technical journals and their impact on the scientific community indexed by the Web of Science main collection. It reviews journals from 3300 publishers, about 200 disciplines, and 60 countries and, it is possible to check citation statistics from 2001 to the present.

Considering the criteria of the inclusion of papers to be analyzed, it excluded papers published in the proceedings of events, chapters, patents and books, short survey, short communication, correspondence, letter, discussion, book review, product review, erratum, and editorial. Furthermore, papers published prior to January 2007 and subsequent to March 2017 were not considered. In addition, papers published in journals without JCR were also excluded.

Regarding the subject of the papers excluded, some of them deal with: the footwear industry; retail sector; packaging industry; pharmaceutical sector; food industry; integrated vehicle configuration system; green supplier selection; green supply chain management; eco-efficient supply chains; airline industry; environmental footprint; internet of things; supplier development; military aircraft industry; circular economy; construction industry; apparel industry; responsive supply chain; sustainable supply chain management; global agri-food supply chains; fashion retail supply chain; automotive industry; and healthcare quality innovation.

Considering the above-mentioned, subsequently, the duplicate checking of papers was carried out and these were also eliminated from the analysis.

It is important to emphasize that the JCR report of 2015 was considered related to the categories Sciences (SCIE) and Social Sciences (SSCI). Regarding the number of citations of each article, this was obtained from Google Scholar.

Figure 1 illustrates the results found in terms of the quantities of papers found and the results after the filtering process and the application of Methodi Ordinatio.

As can be noted in Figure 1, the systematic literature review covers two parts: Part 1 and Part 2.

In Part 1, in Science Direct, with the terms reverse logistics AND electronic AND multiple criteria OR multicriteria, 307 results were found. In Web of Science, with the terms (TS = (reverse logistics AND electronics) AND TS = (multiple criteria OR multicriteria)), we found 23 results. In Emerald, with the terms reverse logistics AND electronics OR e-waste AND multiple criteria OR multicriteria, we found 278 papers. The number of articles in two bases was 608 papers.

After the first filtering process, which comprised the title and abstract reading in the bases Science Direct, Web of Science and Emerald, 256 papers, six, and 269 papers were eliminated, respectively, thus 77 papers remained.

At the second filtering process, which covered the introduction and conclusion reading, aside from duplicate checking, 24 papers were eliminated from Science Direct; three papers from Web of Science, and five papers from Emerald. The number of articles remaining was 45.

In Part 2, we applied the formula for calculating the index InOrdinatio (IO), which allows for a ranking of the remaining papers to be obtained, organized from the best to the worst. These 45 papers met all previously defined inclusion criteria in the protocol of the systematic literature review. After the calculation of IO, it was defined by the researchers to establish IO < 80 as the cut level. According to Pagani, Kovaleski, and Resende [10], this cut level should be defined by the researchers according to their experience and sensitivity related to the subject researched, thus three papers were eliminated, resulting in 42 papers analyzed.

3.2. Multicriteria Modeling

Based on the systematic literature review conducted in this paper and considering that, although some papers addressing the decision problem were related to the implementation of reverse logistics of e-waste in a multicriteria context used the PCP such as [11,34], which does not require the assignment of the weights. It is recognized that this is one of the most contested steps by some researchers because the weights tend to be arbitrary and biased, especially when the decision context involves multiple decision-makers.

The probabilistic composition of preferences [11] is a multicriteria decision aid methodology able to take into account the presence of uncertainty in the preference evaluations. In this method, the measure of preference for an alternative is given by the probability of this alternative being chosen.

The PCP is developed in two steps: (1) Define the probability of being the best and/or the worst option; and (2) Carry out the composition of the probabilistic preferences. In the first moment, an ordering of the set of options is established through the determination of preferences according to the type of criteria. After determining this ranking, the maximum (Mij) and/or minimum (mij) probabilities of each option (alternative) i must be calculated with concern to the others, combining the preferences obtained for each criterion j. [35].

At the same time, it is important to point out that realistic situations usually involve multiple decision-makers, with conflicting views. Thus, this method can be used to avoid arbitrariness and bias, incorporated in the final decision. Then, the model is adequate to be used considering the preferences of several decision-makers, covering several points of view, thus providing a more holistic view of the decision problem.

In the earlier stages of the implementation of reverse logistics, in this case related to e-waste, it is necessary to continuously follow the advances, assessing the performance in order to implement changes and some improvements. For this purpose, MCDA methods can be very useful because they allow the systematization of decision elements in well-defined steps and measure the performance of some alternatives or options of action under a set of criteria. As a result, for example, it is possible to obtain information on what options of action should be prioritized to be improved.

The PCP method was used by [36], carrying out the prioritization of the actions considering the set of criteria, which was based on the systematic literature review. In this sense, the contribution of this proposition can be emphasized, the model approach the implementation of reverse logistics of e-waste from the perspective of multiple criteria, which can enable the constant revision of the model chosen to implement reverse logistics, and allows the timely correction of the course of action by decision-makers.

This model differentiates from others already published because it uses a non-compensatory approach to deal with the assessment of alternatives under a set of criteria, the probabilistic composition of preferences (PCP) method. This method has proven to be more appropriate than others from an outranking approach, considering that it does not require the assignment of weights such as Promethèe and Electre. The assignment of weights, in general, tends to be arbitrary and requires great effort in the elicitation process from the decision-makers. Another advantage is related to the evaluation of the criteria (exact values), which are converted into possible values by using a probability distribution, allowing an imprecision to be considered in the evaluation process. Therefore, this method contributes to avoiding mistakes in the elicitation process and tend to be cognitively easier for decision-makers.

4. Presentation of Results of Systematic Literature Review (SLR)

Considering that this paper used two methodological procedures: (i) a systematic literature review (SLR); and (ii) multicriteria modeling, we present Section 4 in two parts: Section 4.1 with the results of SLR and Section 4.2 with the proposition and illustration of the multicriteria model.

4.1. Analysis of the Papers Found in Systematic Literature Review (SLR)

The descriptive analysis of the bibliographic data frame shows the main characteristics of the publications. Therefore, it was possible to identify the evolution of annual scientific production on this topic; the most cited papers; most productive countries; most productive universities, the collaboration between researchers around the world related to this topic; the most relevant sources of publications (journals); the analysis of co-occurrence and co-citation network/the authors coupling; the main criteria or variables considered in the decision-making process of reverse logistics of e-waste, main methods or models used by the structure of these criteria/variables; and the main technical procedure to collect and organize the data. In addition, the qualitative analysis of the complete articles was carried out through content analysis, as proposed by Bardin [37]. In general, we expressed the desire to view the first ten authors or first ten sources. The results are displayed in

Table 1. Descriptive analysis: Top 10 most productive authors.

Authors	No. of Articles	Author	No. of Articles Fractionalized
Ahmed S	2	WAGNER TP	1.333
Aras N	2	Barua Mk	1.000
Barua Mk	2	Gobbi C	1.000
Buyukozkan G	2	Krikke H	1.000
Erol I	2	Prakash C	1.000
Guarnieri P	2	Ravi V	1.000
Korugan A	2	Srivastava Sk	1.000
Prakash C	2	Subramanian N	0.833
Serifoglu Fs	2	Ahmed S	0.667
Subramanian N	2	Guarnieri P	0.583

,

Table 2. Descriptive analysis: Top 10 top manuscripts per citations.

Paper	Total Citations (TC)	TC per Year
Srivastava SK, 2008, Omega-International Journal Of Management Sci	211	19.18
Chi X, 2011, Waste Management	169	21.12
Geyer R, 2010, International Journal of Advanced Manufacturing Technology	98	10.89
Rahman S, 2012, International Journal Of Production Economics	74	10.57
Abdulrahman MD, 2014, International Journal Of Production Economics	63	12.60
Barker TJ, 2011, Omega- International Journal Of Management Sci	62	7.75
Govindan K, 2014, European Journal of Operational Research	56	11.20
Bernon M, 2011, International Journal Physical Distribution Logistics Management	55	6.88
Zoeteman BCJ, 2010, International Journal Of Advanced Manufacturing Technology	54	6.00
Ozceylan E, 2014, Transportation Res Pt E-Logistics Transportation Review	42	8.40

,

Table 3. Descriptive analysis: Top 10 most frequent journals.

Sources	No. of Articles
Journal of Cleaner Production	7
Resources Conservation and Recycling	5
Waste Management	5
International Journal of Production Economics	4
International Journal of Advanced Manufacturing Technology	3
Omega-International Journal of Management Science	3
European Journal of Operational Research	2
International Journal of Logistics Management	2
International Journal of Physical Distribution & Logistics Management	2
Applied Soft Computing	1

,

Table 4. Descriptive analysis: Top 10 most frequent keywords.

Author Keywords (DE)	No. of Articles	Keywords-Plus (ID)	No. of Articles
Reverse Logistics	17	Management	16
Recycling	6	Returns	9
WEEE	6	Model	8
e Waste	5	Performance	7
Supply Chain Management	5	Product Recovery	7
Electronic Waste	3	Recovery	7
Sustainability	3	Electronic Waste	6
AHP	2	Reverse Logistics	6
Barriers	2	Uncertainty	6
Electronics Industry	2	Design	5

,

Table 5. Citation analysis: Top 10 most frequent cited manuscripts.

Cited References	Citations
Fleischmann M, 2001, Prod Oper Manag, V10, P156	8
Jayaraman V, 1999, J Oper Res Soc, V50, P497, Doi 10.1057/Palgrave.Jors.2600716.	8
Fleischmann M, 2000, Omega-Int J Manage S, V28, P653, Doi 10.1016/S0305-0483(00)00022-0.	7
Thierry M, 1995, Calif Manage Rev, V37, P114, Doi 10.2307/41165792.	7
Fleischmann M, 1997, Eur J Oper Res, V103, P1, Doi 10.1016/S0377-2217(97)00230-0.	6
Guide Vdr, 2001, Prod Oper Manag, V10, P142.	6
Guide Vdr, 2009, Oper Res, V57, P10, Doi 10.1287/Opre.1080.0628.	6
Ongondo Fo, 2011, Waste Manage, V31, P714, Doi 10.1016/J.Wasman.2010.10.023.	6
Ravi V, 2005, Comput Ind Eng, V48, P327, Doi 10.1016/J.Cie.2005.01.017.	6
Shih Lh, 2001, Resour Conserv Recy, V32, P55, Doi 10.1016/S0921-3449(00)00098-7.	6

,

Table 6. Citation analysis: Top 10 most cited first authors.

Cited First Author	Citations
Guide Vdr	33
Fleischmann M	28
Jayaraman V	14
Ravi V	14
Anonymous	11
Ilgin Ma	10
Pishvaee Ms	10
Carter Cr	9
Guarnieri P	9
Ongondo Fo	9

,

Table 7. Total citations (TC) per country.

COUNTRIES	TC	TC per Year
India	302	50.3
USA	253	36.1
Australia	243	121.5
United Kingdom	125	25
The Netherlands	120	40
China	95	47.5
Denmark	83	41.5
Turkey	58	19.3
Brazil	41	8.2
Spain	17	17

and

Table 8. Criteria and methods used in publications.

AUTHORS	CRITERIA/VARIABLE CONSIDERED	METHODS/MODELS	TECHNICAL PROCEDURE
[40]	No one identified	No one identified	Survey
[41]	LCA criteria: Door-to-door collection by truck, Door-to-door collection by pickers, Delivery at electrical electronic equipment (EEE) shops collection Delivery at metro stations, Delivery at delivery points, Dismantled by recycling company, Refurbishment by social program, Informal dismantling Informal recycling, Adequately recycled regionally, Social; Enterprises/cooperatives, PWB recycling exported overseas, Landfill; Economic criteria: System feasibility System efficiency, Population awareness and adhesion to reverse logistics, Innovation and generation of new economic activities, Competitiveness of formal products in regard to informal ones; Social criteria: Social inclusion, Formal employment Generation of income, Opportunity for professional development, Health risks and working, Conditions, Access to healthcare, Access to education.	PCP	Modeling
[42]	No one identified	No one identified	Survey
[3]	No one identified	No one identified	Interview
[43]	Legislation; customer demand; strategic cost/benefit; environmental concern; volume and quality; incentive, resource, integration and coordination	Decision making trial and evaluation laboratory (DEMATEL)	Modeling
[44]	Sources, recyclers, smelters, illegal dumping, offshoring, landfills, demand markets	Multitiered e-cycling network model	Modeling
[45]	CRT recycling: plastics recycling, glass-to-lead recycling; Plastics recycling: glass-to-glass recycling, chemical recycling, thermal recycling.	MAGIQ technique	Modeling
[4]	Strategic: General enforcement of the law, qualification of the companies, partnerships, Incentives; Environmental: Promotion of environmental education, creation of natural resources savings policies, creation of policies for the correct disposal of e-waste by companies; Economic: Creation of policies for materials reutilization; savings through the use of recycled materials; revenues from the sales of e-waste; creation of advertising campaigns, installation of collection points for e-waste, creation of tax incentives for companies operating RL of e-waste; Social: Creation of policies for public awareness; creation of employment policies, creation of digital inclusion policies and, creation of qualification policies.	Strategic Options Development Analysis (SODA)	Structuring problem
[46]	Electrical fault, Mechanical fault, Electrical and mechanical faults	No one identified	Interviews
[47]	Reuse, recycling, return incentive, collection and shipping, inspection, sorting, average cost, average revenue	No one identified	Documental analysis
[48]	Clear return policies, recognition of reverse logistics as a factor in creating competitive advantage, performance management system, information technology support, forecasting and planning.	No one identified	Survey and interviews
[49]	Economic viability, environmental soundness, transport issues,	No one identified	Documental analysis
[50]	Economic: Operating cost (transportation and labor); Fixed cost: (capital investment, Operating time); Operational: Handling system capacity, Human resources (number of staff involved); Strategic: Accessibility (proximity to the customer/distance), Flexibility and responsiveness (product flexibility, type of products accepted) System response (time); Social: Stakeholders’ participation/willingness to cooperate.	Fuzzy Analytic Hierarchy Process (AHP)	Modeling
[51]	No one identified	No one identified	Documental analysis
[52]	Knowledge requirements: Accessibility of information, What and how to sort/segregate, Where to bring materials, How to get to drop-off site, When (days and hours) to go, Acceptance requirements, Drop-off procedures, Drop-off fees; Proximity to a collection site: Distance walking, Distance vehicular, Degree of traffic congestion/control, Access to public transportation, Timing of public transportation; Opportunity to drop-off materials: Days open, Hours open, Authorization to drop-off; Draw of the collection site: Retail centers, Non-retail services, EOL fees, Cleanliness, security; Ease of the process: Sorting/segregation requirements, Storage requirements, Processing/cleaning, Storage needs, Collection container type, Physical effort, Physical access, Staff interactions and assistance, Access to vehicle, Drop-off procedure.	No one identified	Documental analysis
[53]	No one identified	No one identified	Survey
[54]	Channel conflict, investment, Closed-loop chains unlikely, control over designs/materials, Decouples design, contracting and oversight, Processes fast and flexible, Allows for recovery of valuable modules, Higher recovery rates, Long-term contracts, Higher variable costs, Closed-loop supply chains, levels of material recovery and learning, access to scarce materials	No one identified	Documental analysis
[55]	Total cost, number of disposed items, material sales revenue, and customers’ satisfaction level	ARTODTO model, linear physical programming model	Modeling
[56]	The types, numbers and locations of storage sites, The types, numbers and locations of recycling facilities, The quantity of product categories to be allocated to storage sites, The quantity of product categories to be allocated to recycling facilities, The network flow of product categories through storage sites, recycling facilities and secondary market, The total cost of the reverse logistics system.	Mixed integer linear programming (MILP)	Modeling
[57]	Specification of a new product, sales quantity, marketing implementation feasibility, production capacity, technical information, information related to quality control,	No one identified	Case study
[58]	Firm performance, resources capacity, service delivery, reverse logistics operation, communication and information technology systems, geographical location, reputation and experience	AHP-VIKOR approach	Modeling and case study
[59]	Convenience, Store type, Store Status, Location, Hours	Regression analysis	Documental analysis
[60]	Profit, quantity, quality and timing of returns on decisions, customer convenience distance constraints and the per unit transportation costs	Mixed Integer Linear Programming (MILP)	Modeling
[61]	Cost savings (Recycled product; Testing; Scrap shipped; Original facility) and business relations (Proprietary knowledge and control; Customer interactions and direct relationships)	AHP	Modeling
[62]	Time delays on asset recovery opportunities, cost of transportation from a collection point to a central processing location, capacitated returns processing center, information sharing between the collection point and the central processing facility,	No one identified	Modeling
[63]	Costs, profits, and requirements for developing the reverse supply chains infrastructure for collecting and recycling end-of-life electronic	MILP	Modeling
[64]	Sales quantity of new devices realized by the retailer during one period; recovery rate; selling price of refurbished devices; refurbishing/remanufacturing cost; inspection/sorting cost; disposal cost for devices not selected for refurbishment; recycling fee to distributor	Analytical model	Modeling
[65]	Mass balance, GER value per type of material, Energy use on the process level, Energy use on the facility level, Product volume and capacity use, Capacity of equipment, Ton-kilometers and fuel consumption trucks WMS of LSP/truck owner, Degree of substitution per recovery option, Conversion factors CO2	MILP	Modeling
[66]	Capacity Criteria (CC); Financial Ability (FA) IT System (IT) Service Quality (SQ); RL Activities (RA) Geographical Location (GL), Partner Image & Experience (PE)	Fuzzy AHP TOPSIS	Modeling
[67]	Management, financial, policy and infrastructure	Factor Analysis	Documental analysis
[68]	Profit	Game Theory	Modeling
[46]	Quality; cost	No one identified	Survey and interviews
[69]	Internal barriers and external barriers	No one identified	Interviews
[70]	Investment; installing capacity; collection network; quantity of discarded products.	MILP	Modeling
[71]	Components of the TPB, socio-demographic and socio-economic variables, the degree of awareness towards the problem, and the personal assessment of the environmental situation of Brazil as predictors of e-waste recycling	Theory of Planned Behavior (TPB); (ANOVA), logit regression	Survey
[72]	Barriers for GSCM Implementation; Drivers for GSCM Implementation; Environmental Performance; Internal Environmental Management; Green Purchasing; Life Cycle Assessment; Eco-Design; Waste Management; Reverse Logistics; Green Manufacturing and Remanufacturing	No one identified	Interview
[73]	Costs of transportation, purchasing, refurbishing, and operating the disassembly workstations	Nonlinear mixed integer programming	Modeling
[74]	Total cost of the network	Nonlinear mixed integer programming	Modeling
[75]	Operational performance, organizational integration and management reporting and control	No one identified	Documental analysis
[76]	Level of expertise; service cost, service capacity; facility capacity; people employed; information system	No one identified	Documental analysis
[77]	Credibility; Confirmability; Control; Transferability	No one identified	Documental analysis
[78]	Primary focus; Recovery process focus; Lead time focus	No one identified	Interviews
[79]	Total profit; Total revenue; Total cost; Total emission cost; Total collection cost; Total transportation Cost; Total fixed facility cost; Total disposal cost; Total processing cost	MILP	Modeling

.

First, Figure 2 shows the annual scientific production and its evolution. Analyzing these data, we can point out that 2009 was the only one without publications, and the greatest number of publications occurred in 2014 with eight papers, followed by 2011 with seven, 2016 with six, and 2015 and 2017 with five papers each. The estimated annual percentage growth rate was 22.28.

Table 1 shows the 10 most productive authors, in which the authors’ frequency distribution was defined by Articles Fractionalized.

Table 2, Table 3, Table 4, Table 5 and Table 6 present the descriptive analysis of the papers more cited, the main journals of publication, main keywords, and the main authors. These results can be useful for researchers acting in this field, in order to identify the most relevant papers and journals.

It is worth mentioning the variety of areas of knowledge related to journals of publications analyzed in this SLR. This observation is an indication that concerns the issue of e-waste impacts and is impacted in many ways: social, environmental, technological, managerial, productive, economical, and others. Thus, their analysis and solution can originate from different theoretical approaches, but also, moreover, from the dialogue between them.

According to Baldé et al. [1], China, the United States of America, Japan, India, Germany, the United Kingdom, Brazil, Russia, France, and Indonesia are respectively the ten economies that generated the most waste in 2016 in absolute numbers per kg/ton. We can observe that four of the ten countries cited had publications in this period. This evidence may point to a relationship between the commitment of academia and the emergence of the problem. In addition, it is important to point out that most of these countries have specific legislation on e-waste management. Table 7 presents the total citations per country.

Figure 3 shows the most productive universities, considering the link of corresponding authors with two or more papers: the Indian Institute of Technology Roorkee, India; Abant Izzet Baysal University, Turkey; Duzce University, Turkey; University of Brasilia, Brazil, Tilburg University, The Netherlands; University of Southern Maine, United States of America (USA); and York University, the United Kingdom (UK).

Figure 3 shows the most productive universities, considering the number of papers per corresponding authors.

Table 9. Criteria to be included in the model based on the SLR.

Group	Criteria
Economic	Costs of transportation, Service cost, Cost of refurbishing and operating the disassembly workstations; System feasibility; System efficiency, Innovation and generation of new economic activities, Competitiveness of formal products in regard to informal ones; Sales quantity; Costs with maintenance of equipment; Costs with workforce; Costs with infrastructure; Strategic cost/benefit; Total cost of the network; Total profit; Total revenue; Total cost; Total emission cost; Total collection cost; Total fixed facility cost; Total disposal cost; Total processing cost; Total Investment; Sales quantity of new devices realized by the retailer during one period; Recovery rate; Selling price of refurbished devices; Refurbishing/remanufacturing cost; Inspection/sorting cost; Disposal cost for devices not selected for refurbishment; Recycling fee to distributor; Creation of policies for materials reutilization; Savings through the use of recycled materials; Revenues from the sales of e-waste; Costs with advertising campaigns; Costs of installation of collection points for e-waste; Incentives for customers return e-waste;
Technical and Managerial	Specification of a new product; Marketing implementation feasibility, Production capacity; Technical information; Quality control; Level of expertise; Facility capacity; Information system; Credibility; Confirmability; Control; Transferability; Sorting/segregation requirements; Storage requirements; Processing/cleaning; Storage needs; Collection container type; Physical effort, Physical access; Staff interactions and assistance; Access to vehicle; Drop-off procedure; Integration and coordination; Volume and quality of waste; Installing capacity; Collection network; Quantity of discarded products; Handling system capacity, Human resources (number of staff involved);
Environmental	Compliance with environmental legislation; Environmental concern from consumers; Population awareness and adhesion to reverse logistics; Promotion of environmental education; Natural resources savings; Policies for the correct disposal of e-waste by customers; Mass balance; Energy use on the process level; Energy use on the facility level; Product volume and capacity use; Ton-kilometers and fuel consumption trucks; Degree of substitution per recovery options; Conversion factors CO2; Carbon footprint; Water footprint;
Social	Social inclusion, Formal employment/Jobs; Generation of income; Opportunity for professional development; Health & Safety; Labor Conditions; Access to healthcare; Access to education; Digital Inclusion; Gender Issues; Inclusion of vulnerable workers; Diversity issues; Socio-demographic variables; Socio-economic variables; Stakeholders’ participation/willingness to cooperate; Engagement in community; Qualification/training programs.

corroborates the results from Figure 3, considering that the most productive countries appear at the top of the ranking of the most cited countries, except for the case of Brazil and Turkey.

In addition, Figure 4 shows the collaboration between researchers, considering their country of affiliation. We can highlight partnerships between researchers from the USA and Thailand; USA and Canada; USA, Spain, and The Netherlands; USA and Malaysia; USA, Canada, and Austria; USA and Turkey; USA and Denmark; Brazil and Portugal; Brazil and China; Australia and Switzerland; and Australia and India, which means that the USA has a greater frequency of partnerships, followed by Australia, Brazil, Canada, and China.

Regarding the most relevant sources of publications, we can point out the Journal of Cleaner Production with seven papers; Resources, Conservation and Recycling and Waste Management journals with five papers; International Journal of Production Economics with four papers; International Journal of Advanced Manufacturing Technology and Omega and International Journal of Management Science with three papers; European Journal of Operational Research, International Journal of Logistics Management, and the International Journal of Physical Distribution & Logistics Management with two publications. The Applied Soft Computing Journal presented only one paper published in this period. Complementarily, regarding the publishers, it was found that 33 journals were from Elsevier; five from Emerald; three from Springer, one from Taylor & Francis, and one from Sage.

The co-occurrence network of the top-thirty terms on research is presented in Figure 5. The circle size of the terms represents the frequency of occurrence in the articles. The thickness of the line between any two terms displays the degree of cooperation. These terms include decision-making, supply-chain management, and design. It should be noted that the keyword co-occurrences analysis identified “China” as the only country.

According to Mo et al. [38], citation is not a perfect index to evaluate the author’s impact in a specific field of research. However, co-citation patterns are found to differ significantly from bibliographic coupling patterns, but to agree generally with patterns of direct citation. Clusters of co-cited papers provide a new way to study the specialty structure of science. They may provide a new approach to indexing and creating SDI profiles. The co-citation frequency of two scientific papers can be determined by comparing lists of citing documents in the Science Citation Index and counting identical entries [39]. Article accessibility is a possible constraint for authors to cite articles. Some authors may have access to a large number of articles, while others may not. Therefore, the overlap between the accessible articles would be the only linkage for a coupling relationship between the authors in these two groups. Moreover, the authors’ preferences for different kinds of articles differ [38].

From Table 3, we can see that most of the top manuscripts per citations were published after 2010. Similarly, the majority of articles presented in the co-citation network were published after 2010 (Figure 5). On the other hand, the most frequently cited manuscripts (Table 5) were published before 2002. This may be attributed to the new emergency trending regarding world sustainability.

Finally, Figure 6 shows the authors’ coupling, where two authors are coupled through a third party, the cited article in this case, and it can be demonstrated by the authors’ publications and articles cited by them. Regarding author publication, it is true that authors need to have a certain number of publications in order to have the chance to be coupled with other authors through cited references [38]. The colors in Figure 6 represent the networks of connections between the authors, differentiating the networks present. As can be seen in Figure 6, some nodes (circles) were larger than others, as there was a greater quote.

Figure 7 shows the authors’ coupling considering the papers analyzed in the SLR. This indicator presents the cumulative references to a common document. In addition, if the authors share some documents, their coupling strength increases.

In addition, Figure 8 shows there are three groups (networks) of links between the most cited authors.

Based in Table 8, we could identify the main criteria or variables considered in the decision problem related to reverse logistics and the variations (which can be encountered in some authors as part of its activities: recycling, reuse, refurbishing, remanufacturing, collection, disposal) involved. Regarding the main methods or models used to deal with these criteria/variables, it was possible to identify mainly methods from linear programming such as mixed integer linear programming (MILP), used in six of 42 papers analyzed in this systematic literature review, and from the multiple criteria decision aid (MCDA) approach such as the analytic hierarchy process (AHP), used in five papers. We could not fail to mention that 20 of 42 papers did not cite any structured or formal method of the analysis of the data. The remaining 11 papers used several other methods, as shown in

Table 10. Assessment of alternatives under the set of criteria from the numerical example.

		Economic							Technical & Management						Environmental					Social
Alternative		C1	C2	C3	C4	C5	C6	C7	C8	C9	C10	C11	C12	C13	C14	C15	C16	C17	C18	C19	C20	C21	C22	C23	C24
A1	Door-to-door collection by truck	2	4	3	2	4	5	4	4	5	4	4	3	3	3	3	4	5	5	4	3	3	2	2	3
A2	Door-to-door collection by waste pickers	4	4	3	4	4	3	3	3	3	4	3	3	4	4	4	5	4	3	5	5	4	4	4	3
A3	Delivery at EEE shops collection	3	3	4	3	4	3	4	2	3	2	2	3	4	3	3	4	3	4	3	2	3	3	3	3
A4	Delivery at metro stations	4	3	3	3	3	4	3	4	3	4	3	4	3	4	4	4	4	4	4	3	4	3	4	4
A5	Delivery at delivery points	4	3	4	4	3	3	3	3	4	3	3	4	4	4	5	4	3	5	5	4	4	4	3	4
A6	Refurbishment by social program	3	3	4	3	4	3	4	2	3	2	2	3	4	3	3	4	3	4	3	2	3	3	3	3
A7	Informal dismantling	2	2	2	2	2	2	2	3	2	3	2	1	2	2	2	2	2	3	3	3	2	3	2	2
A8	Informal recycling	2	2	2	2	2	2	2	3	2	3	2	1	2	2	2	2	2	3	3	3	2	3	2	2
A9	Adequately recycled regionally	4	4	3	4	4	3	3	3	3	4	3	3	4	4	4	5	4	3	5	5	4	4	4	3
A10	Dismantled by recycling company;	4	4	3	4	4	3	3	3	3	4	3	3	4	4	4	5	4	3	5	5	4	4	4	3

. Other important data excerpted from Table 10 are related to technical procedures used to collect and organize the data.

We can also observe that most of the papers, 20 of them, used the modeling as the main procedure; 10 used documental analysis; seven of them used interviews, and another six used survey. The modeling methods were used mainly to create models for structuring, managing, and optimizing reverse logistics chains/networks for e-waste. The papers using other procedures had the purpose of describing reverse activities of e-waste in some regions such as informal recycling in China, cases of industries in Turkey, the barriers to implement reverse logistics, the behavior of disposal of consumers from or the new demands that arise for stakeholders from Brazil due to legal demands. Some papers have analyzed the efforts made by different actors (government, manufacturers, consumers, distributors, third party logistics providers, among others) along the supply chain.

The main contribution of these papers is related to the dissemination of practices that can be reproduced in other contexts and that favor the expansion of the adoption of reverse logistics. Some studies presented data on the production and destination of e-waste with the demographic information. There were also studies dedicated to the evaluation of the efficiency of reverse logistics systems for e-waste. This kind of study, usually qualitative, contributes to the identification of bottlenecks and barriers in the reverse reinforcing the challenges to implement it, mainly in emerging economies, which can include factors related to technology, supply chain process governance, economics, knowledge technical, legislation/policy and market, among others.

Additionally, some papers have evaluated the economic, social, and environmental impacts of the implementation of reverse logistics for e-waste. These studies include comparisons related to the perception of different actors involved in the reverse logistics of e-waste, also related to management techniques and instruments. Another important aspect of the studies related to the reverse logistics of e-waste are the legal and political concerns. These papers describe the measures taken by governments of different countries to implement and control the generation and management of e-waste. It can contribute to the benchmarking of institutionalized practices, the evaluation of the efficiency of policies, and the leveling of national advances in issues related to e-waste. The use of an information system related to the operational process of reverse logistics for the recovery of e-waste can also be found in some papers, citing technologies such as radio frequency identification (RFiD) and enterprise resource planning (ERP). Furthermore, the analysis of consequences of the decisions taken in the development of products considering their life cycle was also encountered in this literature review.

4.2. The Model Proposed to Assess the Implementation of the Reverse Logistics of e-Waste in the Earlier Stages

The model proposed follows the MCDA approach, as derived from the operational research (OR) area. Considering that the modeling of problems, considering a MCDA approach, should be systematized not only in terms of technical inputs and analytical characteristics, but also in terms of the support and insight given to the implementation [24], we preview steps aimed at the elicitation process with decision-makers conducted by an analyst.

Some assumptions of the model, based on the SLR, should be pointed out:

(a)

This model includes several qualitative and quantitative criteria, related to economic, social, and environmental issues, which very often have conflicting relationships [3,4,25,36];

(b)

This context of decision-making comprehends several stakeholders such as government, citizens, waste manager, industries of electrical electronic equipment (EEE), recycling industries of e-waste, third-party logistics providers, small, medium or large size retailers, waste pickers and, non-governmental organizations, and others [3,4,36];

(c)

These stakeholders, very often, present conflicting points of view related to the implementation of reverse logistics of e-waste, considering economic, social, and environmental dimensions, considering their different interests related to the topic [4,36];

(d)

Suppliers should have a balanced performance in all criteria proposed, which means that compensation between criteria is not allowed (a poor performance in a given criterion should not be compensated for by very good performance in another criterion) [4,25,80];

(e)

In realistic situations, decision-makers may have difficulties to decide due to (i) there is subjective or insufficient data; (ii) the analyst may not know the preferences of the decision-makers; and (iii) it is hard, or even arbitrary to assign weights or exact values to the criteria chosen to evaluate the options of actions (alternatives) [36]; and

(f)

It is required that the model incorporates inaccurate, imprecise, uncertain, or ill-determined data and considers the imperfect nature concerning the values of the criterion variables in the decision-making process [4,25,36].

A framework of the model proposed can be seen in Figure 8.

As presented in Figure 8, the model has the following steps:

(1)

Identifying the decision-makers (active participants to be involved in decision making). These people should be selected for their knowledge and experience about various functions in the company, and they do not have the same competence. Since conflicting judgments are unlikely to be resolved by consensus, stakeholders can be weighed according to their power to influence results.

(2)

Validating the set of decision criteria. Identify the significant factors involved in the decision process based on the criteria included in the model (obtained from the literature review). This can be done with brainstorming sessions involving several individuals, performing different functions, and to obtain different points of view. Some of these factors may not be significant in order to achieve the overall objective, so they must be filtered out; if there are conflicts, a group leader must resolve them.

(3)

Identifying the alternatives. The alternatives are the options of action aimed at the implementation of reverse logistics—if any of them is unfeasible, it must be eliminated before applying the method—by mutual agreement between the stakeholders involved;

(4)

Eliciting the preferences of the decision-makers. Assess the alternatives (options of action) through objective/direct values, or, still, through ordinal scales, obtain the evaluation of each action option according to the criteria validated by the decision-makers. if there are different measures, it is necessary to normalize the data before applying the method (dividing the value of each cell in the matrix by the total of the column of which it is part);

(5)

Applying the PCP method. To probabilistically compose the preferences of the decision-makers and prioritize the actions that need immediate actions to improve the process. The proposed methodology seeks to optimize according to all criteria and considers the probabilities of maximization;

(6)

Delivering results. The results can be presented in a ranking organized from the best to worst, or the worst to best, demonstrating what alternatives should be improved in order to reach the objective of the implementation of the reverse logistics of e-waste [p1]; and

(7)

Providing Feedback. Based on the results, it is possible for decision-makers to decide whether to maintain the course of action or whether to implement changes to improve the performance of indicators in a timely and accurate manner.

Additionally, it is also possible to conduct an analysis in two scenarios, using the beta and triangular distribution, to test the validity and robustness of the results.

Numerical Example to Illustrate the Use of the Model Proposed

Step 1: The decision-makers can be the people in charge of the logistics, financial, production and operations sectors, among others.

Step 2: It is expected that the decision-makers analyze the criteria suggested based on the systematic literature review (Table 9), which can be categorized in four major groups: economic, technical and managerial, environmental, and social, and pick those more appropriate for the context.

To illustrate the model, we will use the following criteria:

Economic category: C1—Costs of transportation; C2—Cost of refurbishing, and operating the disassembly workstations; C3—Total revenue; C4—Total Investment; C5 Selling price of refurbished devices; C6—Costs of installation of collection points for e-waste; and C7—Incentives for customers return e-waste.

Technical & Management: C8—Production capacity; C9—Quality control; C10—Sorting/segregation requirements; C11—Storage requirements; C12—Volume and quality of waste; and C13—Human resources (number of staff involved).

Environmental: C14—Promotion of environmental education; C15—Natural resources savings; C16—Energy use on the process level; C17—Carbon footprint; and C18—Water footprint.

Social: C19—Formal employment/Jobs; C20—Generation of income; C21—Health & Safety of employees; C22—Labor Conditions; C23—Digital inclusion; and C24—Engagement in community.

In Step 3, there are some alternatives in the literature [37], which will be used for the purposes of the illustration of the application of the model:

• A1—Door-to-door collection by truck;
• A2—Door-to-door collection by waste pickers;
• A3—Delivery at EEE shops collection;
• A4—Delivery at metro stations;
• A5—Delivery at delivery points;
• A6—Refurbishment by social program;
• A7—Informal dismantling
• A8—Informal recycling;
• A9—Adequately recycled regionally;
• A10—Dismantled by recycling company.

Step 4: The elicitation of the preferences of decision-makers, for the illustration of the model, will be made by using questionnaires composed of ordinal scales, ranging from 1 to 5, in which 1 is the lowest level and 5 the highest level. Therefore, the matrix to evaluate will be composed of 10 alternatives (A1–A10) and 24 criteria (C1–C24), as in Table 10.

To obtain the probability of an option i being the best ($M_{ij}$) or the worst ($m_{ij}$), according to each criterion j, statistical independence must be assumed to simplify the calculation of the joint probabilities by the product of the marginal functions of probability, according to Equations (2) and (3) [81].

$$M_{ij}=\underset{D_{X_i}}{{\displaystyle \int }}\left[{\displaystyle \prod }{F}_{X_{-i}}\left(x_{-i}\right)\right]f_{X_i}\left(x_i\right)d{x}_i$$

(2)

$$m_{ij}=\underset{D_{X_i}}{{\displaystyle \int }}\left[{\displaystyle \prod }(1-F_{X_{-i}}\left(x_{-i}\right))\right]f_{X_i}\left(x_i\right)d{x}_i$$

(3)

The parameters FX, fX, and DXi represent, respectively, the cumulative distribution function of the variable x−i, which represents the other options under the same criterion, except for the variable xi considered in calculation; the probability density function; and the support of the random variable xi, which indicates the lower and upper limits of the domain of the respective variable. In addition, it is important to state that in Equations (2) and (3), the parameters FX, fX, and DXi represent, respectively, the cumulative distribution function of the variable x−i, which represents the other options under the same criterion, except for the variable xi considered in calculation; the probability density function; and the support of the random variable xi, which indicates the lower and upper limits of the domain of the respective variable.

Then, depending on the approach considered, different joint probabilities can be used. In this sense, such approaches can be discriminated in terms of the choice between positions among two basic orientations, optimistic versus pessimistic and progressive versus conservative, as shown in

Table 11. Different approaches for combining probabilistic assessment.

Optimistic	Pessimistic
It considers satisfactory to meet at least one criterion.	It seeks optimization according to all criteria.
Progressive	Conservative
Consider the probabilities of maximizing preferences.	Consider the concern only in avoiding negative extremes; chances of not minimizing preferences.

Source: This table was adapted from [82].

[35].

The proposed methodology seeks to optimize according to all criteria and considers the probabilities of maximization (to obtain the highest risks), so it is necessary to use the pessimistic/progressive approach. Thus, to calculate the probabilities of preference in this study, it was only necessary to estimate the probabilities of maximum preference (Mij). Pessimistic/Progressive: (PPe) = ${\displaystyle \prod }_{j=1,\dots ,m}{M}_{ij}$;

For more details of this method, see [11].

Step 5: Application of the PCP model.

Table 12. Beta-PERT and triangular distribution.

Ranking
Beta-PERT			Triangular Distribution
1°	A10	9,23081e−92	1°	A10	6,3226e−28
2°	A9	2,28793e−60	2°	A9	1,7923e−25
3°	A5	5,669e−113	3°	A5	2,8322e−32
4°	A2	1,92132e−68	4°	A2	1,9254e−27
5°	A4	3,45752e−55	5°	A4	1,106e−24
6°	A1	6,6985e−106	6°	A1	4,3885e−31
7°	A6	3,1517e−194	7°	A6	3,6666e−39
8°	A8	4,3651e−217	8°	A8	2,7404e−40
9°	A7	3,55103e−51	9°	A7	9,2334e−24
10°	A8	6,27103e−37	10°	A8	9,3223e−21

also shows a noticeable concordance between the ranking obtained employing the beta and the triangular distribution. The beta distribution is a weighted average in which more weight is given to the most likely estimate, while the triangular distribution is defined by its minimum, maximum and mean values and it does not have to be symmetric. The beta-PERT (Program evaluation and review technique) is a tool for modeling expert data, based on the likelihood of components. In the case of beta distribution, the package ‘PCP’ was used, through the implementation in software R. In the case of triangular distribution, preferences were normalized in the interval (0.1), and these parameters were assumed as the modes of triangular distributions with amplitude (0.1).

It was possible to observe that the ranking was not modified in the two distributions (Beta-PERT and triangular), which demonstrates that the modeling is robust and stable.

Aside from the studies considered in the SLR, some other relevant papers discussed issues related to recycling and the reverse logistics of e-waste from several perspectives. The comparison of e-waste credits with the carbon credit market, in the context of Brazil, was made by [3]. The analysis of sectoral agreement for the reverse logistics of e-waste, also in the context of Brazil, was conducted by [4]. The authors used the SODA (strategic options and development analysis) approach to structure the decision-making context considering the opinion of several stakeholders involved. The decision support systems for the optimal location of e-waste treatment plants, in the context of Greece, was studied by [19].

The prioritization of barriers to be solved in the implementation of reverse logistics of e-waste, in the Brazilian context, considering the opinion of SMEs (small and medium companies), consumers, and government under an MCDA perspective was performed by [36]. The study of relevant factors in the implementation of end-of-life computer recycling in reverse supply chains was approached by [43]. The quality of recycling of e-waste from end-of-life computers was analyzed by [45]. The life-cycle perspective was approached by [83], which presented a quantitative framework to estimate life-cycle impacts and costs associated with e-waste management, with well-defined steps. Data from the State of Washington’s EPR program were used to test the framework, involving the activities of collection, transportation, processing, and disposal.

Thus, the theoretical model proposed in our paper differentiates from the previously mentioned because it considers several criteria related to economic, social, environmental, technical, and managerial issues to assess the alternatives to be used in the early stages of implementation of reverse logistics of e-waste. This context of decision is very characteristic in developing countries such as those from BRICS (Brazil, Russia, India, China and South Africa), which are in earlier stages of the reverse logistics of e-waste, differently from developed countries, in which the reverse logistics of e-waste is already institutionalized and consolidated. Our decision model provides well-defined steps to select from the literature the proper criteria and alternatives and to analyze them under a multicriteria approach, which does not consider trade-offs between criteria and does not require the assignment of weights. The trade-off of criteria and the assignment of weights are very often considered as contradictory in the literature, so avoiding these issues, the model can be considered more user-friendly and provide more balanced solutions.

Therefore, with the use of the model, it is possible to obtain inputs to make better decisions, considering a systemic analysis of the criteria and the alternatives of actions to implement the reverse logistics of e-waste in the earlier stages, which can be useful mainly for managers and policymakers acting in this field.

5. Conclusions

The purpose of this paper was to assess the implementation of the reverse logistics of e-waste in the earlier stages. The study was conducted in two steps. The first step was aimed to identify the main characteristics of publications related to reverse logistics considering multiple criteria, based on the literature from 2007 to 2017, in order to obtain inputs to insert in Step 2. Step 2 is related to multicriteria modeling; in this paper, we illustrated the model proposed with a numerical example containing 10 alternatives and 24 criteria.

In the first step, we identified the main characteristics related to the evolution of annual scientific production on this topic; the most cited papers; most productive countries; most productive universities, the collaboration between researchers around the world related to this topic; the most relevant sources of publications (journals); the analysis of co-occurrence and co-citation network/the authors’ coupling; the main criteria or variables considered in the decision context of reverse logistics of e-waste, main methods, or models used by the structure of these criteria/variables; and main technical procedure to collect and organize the data. Thus, besides the bibliometric analysis, we performed a content analysis considering the main contributions of the papers analyzed. This part of the paper can guide practitioners and researchers to identify the literature related to this topic of research.

Step 2 aimed to model the decision-making context, and for this purpose, we used the results from Step 1 to gather alternatives and criteria to apply the MCDA modeling, composed of seven steps: (i) identifying the decision-makers; (ii) validating the set of decision criteria; (iii) identifying the alternatives; (iv) eliciting the preferences of the decision-makers; (v) applying the PCP method; (vi) delivering results, and (vii) providing feedback.

Considering the results obtained in the first step, which was the SLR, we could verify that there is a large predominance of papers addressing decisions related to the optimization of reverse logistics networks as well as analyzing the barriers and challenges related to reverse logistics implementation. Many papers do not make clear which methods or models were used to analyze the criteria or variable, which contributes to the understanding that the authors qualitatively analyzed the factors related to the economic, social, and environmental impacts of reverse logistics. However, despite this, it is important to emphasize that a considerable part of the papers analyzed used structured methods/models from MCDA and linear programming approaches to analyze the dimensions involved in this context. Some papers have highlighted the characteristics of some regions/countries related to e-waste generation and management, others have described the new demands to the several actors involved in this context, and the behavior of consumers to dispose the e-waste. Considering the evolution of the number of studies published each year, it is clear that interest in the study of this topic is increasing, although 2009 did not present one paper published, considering the limitations of this paper related to the criteria of inclusion and exclusion of papers considered for this literature review.

Related to the second step, the MCDA modeling, it was possible to offer to decision-makers well-defined steps to guide the decision-making process in the context of the reverse logistics of e-waste. Considering the earlier stages in developing countries related to the implementation of the reverse logistics of e-waste, perhaps the alternatives as well as the criteria or performance indicators used to measure the implementation are not well-defined. Therefore, a framework aimed toward this purpose can be very useful, providing timing and accurate information in the form of feedback. Considering this feedback, it is possible to decide whether the current strategies can be maintained or changed, incorporating improvements in those alternatives with the worst performance in the criteria selected, covering the social, environmental, technical, and managerial dimensions. The consideration of these dimensions can provide a holistic view of the decision context.

As a suggestion for further studies, we recommend the use of other protocols of systematic literature review related to this topic to update the SLR. The proposition of structured methods dealing with the dimensions of economic, social, and environmental dimensions of the sustainability of reverse logistics of e-waste can be the object of future studies. Additionally, the integration of methods from problem structuring methods (PSM) with multicriteria decision aid (MCDA) or with derivations of linear programming methods can also be studied. The use of the concepts of Logistics 4.0 and Supply Chain Management 4.0, Internet of Things (IoT), and respective technologies and systems related are topics still not well exploited in the literature, and constitute a great opportunity for future studies. Additionally, research comparing the developed and developing countries’ best practices and challenges related to e-waste management and respective legal requirements can be the target of further studies.

The theoretical contributions of this paper are two-fold. First, we synthetized the existing knowledge on the reverse logistics of e-waste considering multiple criteria or variables of decision in the last years, which can be useful to researchers exploiting this topic. We also analyzed which topics were still underrepresented in the literature, identifying suggestions for future research, which can guide researchers and practitioners. Second, we propose a method with well-defined steps to guide decision-makers to provide more consistent and timing information. For business practice, this paper can be useful for managers to synthetize the main criteria and methods used in this context of decision, serving as a base for decision-making related to this topic.