Next Article in Journal
Water Use Behavior in a Multicultural Urban Area in Sweden
Previous Article in Journal
Frontiers in the Solicitation of Machine Learning Approaches in Vegetable Science Research
Previous Article in Special Issue
Building a Care Management and Guidance Security System for Assisting Patients with Cognitive Impairment
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A Sustainable Multicriteria Decision Framework for Obsolescence Resolution Strategy Selection

1
Mechanical Engineering Department, École de Technologie Supérieure, Montréal, QC H3C 1K3, Canada
2
Construction Department, École de Technologie Supérieure, Montréal, QC H3C 1K3, Canada
3
Systems Engineering Department, École de Technologie Supérieure, Montréal, QC H3C 1K3, Canada
*
Author to whom correspondence should be addressed.
Sustainability 2021, 13(15), 8601; https://0-doi-org.brum.beds.ac.uk/10.3390/su13158601
Submission received: 26 May 2021 / Revised: 24 July 2021 / Accepted: 26 July 2021 / Published: 2 August 2021

Abstract

:
Parts obsolescence has an important impact on the product life cycle, the manufacturing system and the environment leading to operational, logistical, reliability and cost implications. While current resolution models are cost-oriented, multiple studies have revealed that technological obsolescence is strongly involved in the electronic waste problem. In this study, based on academic literature and expert opinions, a sustainable decision framework for obsolescence resolution strategy (ORS) selection is proposed. It consists of economic, environmental, social and technological dimensions, integrating a total of fifteen criteria. Multicriteria decision-making (MCDM) methods are suggested to select the most sustainable solution. A case study was performed where the criteria weights and the alternatives performance were judged by five experts from the fields of environment, economy, human resources and obsolescence and operations management. Results from different MCDM methods were compared to the actual decision to evaluate their effectiveness. Using the suggested framework improved the decision process as integrating sustainability had a drastic impact on the selected strategy and consequently on the company’s performance. In addition to its managerial insights, this paper provides a new research perspective to sustainable and robust obsolescence management to effectively handle the increasing number and severity of obsolete components.

1. Introduction

Component obsolescence is a significant problem in our modern highly technological market. It is a result of the rapid evolution of technology by introducing new components with higher performance and additional features. It led to a growing pressure to upgrade components and systems. Obsolescence is defined as the loss of functions of a component or a system (hardware or software) so it cannot accomplish its essential purposes or is no longer useful because it is unsustainable, high-priced to repair, defective or the product is no longer available for purchase in its original form from the original manufacturer [1]. It specifically rises due to the mismatch between the life span of the product and the parts of the system. The conflict concerns sustainment-dominated systems, which take many years to design and manufacture and are typically maintained for decades such as aircraft and submarines, as well as nowadays technological products such as computers or mobile phones. They are usually composed of “commercial off-the-shelf (COTS)” components that are highly dependent on market tendencies and technological changes. COTS components have reduced life cycles and experience obsolescence rapidly. As of 2006, QTEC Solutions (an international leader in components obsolescence management) estimated that around 3% of the global pool of electronic parts becomes obsolete every month [2]. A more recent study evaluated that over 350,000 components became obsolete in 2013 [3]. It reflects the extent of the environmental and technological problem; the modern industry is fronting. The rapid pace of technology makes the problem more global and impacts even products with shorter life cycles. Every 24 months, computers’ processing speed doubles, known as Moore’s law, which leads to constant upgrading of components or systems such as military and technological products. Either at the production step or at the recycling, electronics have an important environmental impact as high-tech parts and equipment production require a broad range of materials as well as energy. Often, technical materials required in electrical and electronics are critical raw materials and rare earth elements. In addition, ultra-clean components and intermediate products require energy, water and chemicals [4]. Thus, electronics waste is one of the world’s fastest-growing waste streams with an annual growth rate of four percent [5] whose main driver is technological obsolescence.
Although its history is associated with the (maritime) defense industry, many non-defense systems face similar difficulties, such as avionics, oil well drilling and automotive industry [6]. Nowadays, decisions regarding obsolescence management across all these industries are often based on experience and intuition, leading to an increasing need to reduce the negative effects of obsolescence issues. However, there is a lack of knowledge of how to effectively choose obsolescence mitigation and resolution approaches to reach this goal. Selecting the most suitable obsolescence resolution strategy is then a crucial decision for the system, the manufacturer and the environment where multiple factors are considered [7,8] such as cost, technology, functionalities, etc. Recent studies revealed that the high rate of obsolescence in the electronics industry has led to one of the fastest-growing waste streams in the world [9]. As a simple example, computers are manufactured from over 1000 materials, many of which are toxic and contribute significantly to the e-waste stream, estimated at 53.6 million tonnes in 2019 [10]. Thus, considering sustainable factors becomes crucial as obsolescence is a major cause of component deterioration [11]. Our paper seeks to build a sustainable framework to support decision making in obsolescence management either at the reactive, proactive or strategic level from the design to the end of life of the system.
The paper is organized as follows: first, we present a literature review of obsolescence resolution strategy selection models and the considered criteria, alternatives and decision process. Second, the proposed ORS selection framework is presented. Then, the model is applied with an example from the avionics industry. Finally, our conclusions are summarized, and potential studies are discussed.

2. Literature Review

The study is mainly related to two streams of research in the literature: obsolescence management, in particular the resolution/mitigation strategy selection, and sustainability. The part’s obsolescence problem is also referred to as an “end-of-life (EOL) problem” or “DMSMS” (diminishing manufacturing sources and material shortages) in the literature and it can be divided into two branches. One focuses on forecasting the obsolescence risk of parts and the demand trend after obsolescence occurred. Statistical methods are used extensively in these works, and representative research for this branch can be found in [12,13]. Our paper is more related to the second branch which emphasizes the prescriptive perspective, investigating how to minimize part obsolescence impact and maintain satisfactory service levels with various strategies. Obsolescence is unavoidable, and the current methods of mitigating obsolescence are expensive and inefficient. For example, obsolescence is estimated to cost the U.S. Department of Defense (DoD) more than USD 750 million annually. This estimate means that more than USD 9 billion has been wasted over the past 12 years [7]. That is why multiple studies tried to solve the problem at three different management levels: reactive, proactive and strategic [1]. Reactive management addresses the problem after the component or part has already become obsolete or received a product change notification from the original component or part manufacturer. Proactive management addresses obsolescence before it occurs. This strategy is used for critical parts or systems that have a high risk of becoming obsolete or when the availability of the component or system will be low after it becomes obsolete. It employs forecasting methodologies to predict obsolescence dates for the various parts of a product, analyzes the risk of critical parts in a bill of material (BoM) and takes the necessary steps to manage obsolescence [12]. Strategic management uses the system status, forecasts the DMSMS risk and determines the status of expected needs of inventories and spares. It seeks for the optimal mix of reactive mitigation approaches and design refreshes (minor and major) that will minimize life-cycle costs (e.g., maximize cost avoidance) while continuing to meet all system requirements. Strategic management is also used for strategic planning, life-cycle optimization and long-term business case development to support the system, and it often combines reactive and proactive strategies [1,14,15]. Although obsolescence management is recommended over the three described management levels, it is still mainly reactive where decision making represents a crucial part of the process [1,15].
The second stream to which our research is related is sustainability as obsolescence risk is tightly related to environmental concerns such as e-waste and the usage of rare materials in addition to multiple other perspectives such as ecosystem services, economic and financial issues, social issues and operating licenses. Sustainability is not considered an easy task since it involves high levels of corporate management, production and consumption by society. In this sense, ecological, social and economic pressure is increasing in industrial organizations [16] as the pollution generated by them has increased to levels never reached before [17]. The measurement and monitoring of sustainability in industrial organizations based on indicators promote the simplification and quantification of information on triple bottom line aspects [18]. Sustainable production means producing less, with higher quality and durability, lower environmental impacts and higher profitability.
The first attempt to help decision makers in obsolescence management was an approach based on economic analysis [19]. The work provided a formulation of the net present value of last-time buy and design refresh planning. The model looked for a trade-off between the two possible strategies. It helps obsolescence engineers to develop a solution by providing a break-even year chart that can give the best year for redesign. This model allows a single design refresh at a time. The last-time buy (LTB) strategy is widely studied because of its common use in practice [20]. Reference [21] established a list of factors in cost calculation. It includes procurement cost, inventory cost, disposition cost and penalty cost. Reference [22] proposed a mixed-integer programming formulation to consider parts obsolescence in a two-level lot-sizing problem.
Design refresh has been well studied too. A model developed in reference [23] is based on cost analysis and provides the optimum design refresh plan in order to go through the life sustainment cost of the product. It optimizes the cost over multiple design refreshes. Product life evolution was simulated by [24] using stochastic dynamic programming to make decisions with a Markov decision process. Multiple other studies tried to solve the obsolescence resolution problem using different techniques such as the restless bandit model and Markov decision process [25], integer programming [26], graph theory and mixed linear programming [27]. The limitation of the previous research is that it is cost-oriented and considers only two resolution strategies (last-time buy and redesign) from all resolution profiles, ignoring then the sustainable dimension of the problem.
Other studies considered obsolescence as a criterion in the decision-making process of many problems such as the maintenance of the drinking water and sewer network [28], for legacy application software obsolescence assessment [29] and in the material purchasing problem where reference [30] considered technology obsolescence and developed a bi-objective model to support managers in their decisions in identifying a sustainable quantity purchase when a cap and trade mitigation policy are present. A summary and comparison of the most relevant decision-making papers in obsolescence resolution strategy selection are provided in Appendix A.
To the best knowledge of the authors, only references [3,7] studied the ORS selection more globally by considering multiple criteria and alternatives. While reference [3] applied MAUT (multi-attribute utility theory) to select the best replacement strategy, reference [7] used TOPSIS (technique for order of preference by similarity to ideal solution) and Monte Carlo simulations to rank alternatives during the product design phase. None of them considered sustainability in decision making despite the fact that the environment is directly impacted by electronics waste.
Finally, from a methodological perspective, this paper contributes to the application of a multicriteria decision-making model in obsolescence management. References [10,31] demonstrated the lead role of obsolescence in environmental problems and specifically in accumulation of electronic waste. Our goal in integrating sustainability in obsolescence management is to provide an efficient solution to both ORS selection and waste management problems.
To summarize, based on the extensive literature review, our paper studies the obsolescence resolution strategy selection problem considering sustainability. Obsolescence risk can be considered at any stage of the product life cycle. It will be managed efficiently providing sustainability and robustness. To the best of the authors’ knowledge, this research is among the first to consider such a model setting, particularly, including the environment axis in the ORS problem and deriving the most sustainable solutions.

3. Materials and Methods

Given the multiple elements that affect the ORS selection when a component becomes obsolete such as cost, technological and functional improvement, the stakeholder’s constraints and the solution sustainability, no single strategy focusing on a single element can provide a confident assessment. The best resolution strategy for component obsolescence would be based on several criteria rather than a single element. In addition to the complexity of the problem, the subjectivity of ORS selection plays a central role in the obsolescence management process. Reference [15] discussed a similar challenge of optimizing multiple objectives for all obsolescence management levels when looking for the best resolution/mitigation strategy. In this paper, we develop a multicriteria decision-making framework for ORS selection that supports the problem’s nature and complexity.
Obsolescence resolution/mitigation strategy goes through four steps: obsolescence notification, gathering information, resolution strategy selection and implementation and documentation. Few decision models were used within step 3, where multiple criteria and alternatives have to be considered in an uncertain and complex decision environment. In this paper, we consider the reactive management level to make it simple and easy to use since it is the most common management level within industry [6].
One of the best-suited tools to help in the decision making considering the uncertainty and the complexity of the decision are the multicriteria decision-making (MCDM) or the multicriteria decision aid (MCDA) methods. Following the guidelines proposed by reference [32] to choose an appropriate MCDM/MCDA method, an outranking non-compensatory approach dealing with a discrete set of alternatives with mixed information structure was adopted. In a non-compensatory strategy, the decision maker may state that the criteria are important enough to refuse any kind of compensation or trade-offs. The usage of non-compensatory strategy provides better satisfaction for the decision makers [33,34].
The adopted decision method is based on a highly regularized operating environment where compensating criteria is not preferred. Thus, compensatory single synthesizing approach such as MAUT (multi-attribute utility theory) or AHP (analytic hierarchy process) was rejected (although some techniques developed within AHP framework were used to determine the criteria weights). Reference [35] related compensation degree and the sustainability strength of the MCDM methods and proved that sustainability and compensation are opposite: the higher compensation is, the weaker sustainability will be. Based on the nature of the problem, reference [32] guidelines and reference [35] study, ELECTRE III (ELimination Et Choix Traduisant la REalité) and PROMETHEE (Preference Ranking Organization Method for Enrichment Evaluations) were selected to be used in this work for their support to the discrete, outranking and non-compensatory nature of the problem and for their substantiated effectiveness in similar contexts.
The major difficulty facing an MCDA methodology lies in the assessment and the modeling of the decision-maker (DM) preferences that affect both the decision process and the solution [32]. In this paper, decision-maker preferences are reflected in the criteria weights and the four binary relations established using the concepts of thresholds, concordance and discordance indexes. The four binary relations, for the pairwise comparison of alternatives, are indifference situation I, preference situation P, weak preference situation Q and incomparability R.
To develop the MCDM framework for the ORS selection problem, the process was divided into two stages: construction and exploitation, as recommended by [32]. First, at the construction stage, relevant criteria were determined based on literature then filtered by experts in the fields of environment, economy, human resources, technological obsolescence and operations management. The final criteria set C to be considered is described in Table 1. Unlike the criteria set which is valuable for all obsolescence cases, criteria weights need to be calculated at each obsolescence notification. It is completely natural that cheap parts’ cost will not be the most important criterion in the decision process compared to the ORS implementation time. For critical high-end parts, both requalification and technology readiness will be the most important. Determining weights is, then, performed case by case. Both swing method and pairwise comparison were applied. Sets W 1 and W 2 for weights generated from both techniques are normalized and validated. The exploitation stage involves the aggregation (calculation) and the results (recommendation) steps. The general simplified methodology consists of selecting or recommending the best ORS strategy that satisfies the three actors of the problem: the manufacturer, the customer and the stakeholders. A summary of the research methodology is presented in the Figure 1.

3.1. Alternatives

There are many possible resolution strategies to obsolescence problems. Their range in complexity is from a simple part substitution to a major redesign of the product. Reference [1] provided the most complete list of obsolescence resolution strategies. A total of eleven general solutions were identified. They form the alternatives set A in the proposed multicriteria decision framework. Reference [36] ranked these alternatives in approximate order of increasing cost. The negotiation with the manufacturer to keep producing the component and the usage of the existing stock are both cost-free solutions but rarely considered. The reclamation strategy is also a cheap solution that allows one to obtain a small number of obsolete parts by salvaging or cannibalizing used parts that still have a useful remaining life. For medium-cost solutions, alternate part, part substitution, uprating, aftermarket sources and emulation can be considered. While an alternate part is a part with equivalent or greater performance than the replaced part, part substitution is a complete process of selecting a part with fewer capabilities than the part it replaces. The selection process considers the form, fit and function criteria for a component designed as a replacement part. Uprating is a special range of part substitution. It is the process of assessing the ability of a part to meet the functionality and performance requirements of applications where it is necessary to use it outside the manufacturer’s specification range. Parts can be sourced from aftermarkets too (aftermarket refers to the period after the original manufacturer has moved a part out of production). This solution has high risks of counterfeit [37]. The choice has to be made carefully between approved sources that provide finished parts or those that remanufacture parts and unapproved sources or brokers. Electronic parts can be emulated. Emulation refers to the process of redesigning obsolete electronics from their slash sheets, datasheets, test vectors and other information. In some other studies, emulation is considered as part substitution of alternate parts.

3.2. Criteria

While most of the studies for obsolescence management considered mainly economic and technological factors, Reference [38] believed that considering global sustainability is essential for effective decision making since industrial, social and ecological systems are closely linked. In this paper, a sustainable set of criteria is considered to help make decisions in obsolescence management. In addition to the basic economic and technological criteria categories, environmental and social criteria are added. The strong relationship between technological obsolescence and the increasing stream of e-waste was demonstrated in multiple studies [9,10,39] which made the environmental factors a priority. Another example of a factor influencing the increase in the obsolescence of electronic components is the need to comply with increasingly strict and demanding legislation in the updated European Union regulation in this area of Restriction, Assessment and Authorization of Chemicals (REACH) and the Restriction of Hazardous Substances Directive (RoHS).
The criteria set in this study is based on sustainability indicators for industrial organizations developed by [40], the previous work on obsolescence decision making [7,41] and agreed upon by the case study decision makers. The sustainable framework is inspired by [40,42,43], and the criteria set was divided into 4 subsets: economic, environmental, social and technological as described in Table 1.

3.3. Criteria Weights

Criteria weights are used to run the selected outranking methods and should be calculated with the understanding that set W represents weights of relative importance of the different criteria. They are non-negative numbers and independent from the measurement units of the criteria.
The weights have an important role in measuring overall preferences of alternatives. Their assessment of the criteria is not straightforward. It involves the priorities and perceptions of the decision maker. In this study, two weighting methods are used and compared to help express the decision makers’ preferences. The first method is the value swinging method. It assigns to the most important criterion the highest weight in a selected weighting range (100 to 1–100 scale). First, voters assign 100 over the value measures based on their importance. They identify and discuss significant differences and revote until they agree on the ranking of value measures. A second tour of voting is performed to determine weights following the earlier agreement. The weights are then normalized, and the outliers are discussed. Voting continues until everybody agrees on the weights [3]. This method’s strength is that it considers the range of each criterion, and it is a relatively simple and straightforward method. However, it does not allow participants to directly compare criteria between each other. That is why the second method is the pairwise comparison method. It consists of calculating the relative importance coefficient α i j for all criteria pairs, where α i j = 1 α j i where i   and j are two different criteria. α i j value is equal to 1 if criterion i and criterion j are equally important, 3 if i   is slightly more important than j , 5 if i is more important than j , 7 if i is strongly more important than j and 9 if i is absolutely more important than j . A pairwise matrix is built and normalized by making equal to 1 the sum of the values on each column. Each normalized matrix entry is calculated by α i j ¯ = α i j l = 1 m α l j , where m is the number of considered criteria. Finally, the criteria weight vector w is calculated by calculating each row of the normalized pairwise matrix where w i = l = 1 m α i l m [44]. The pairwise comparison method was extended later to handle fuzzy criteria comparison for fuzzy hierarchical analysis [45].
Both weights from both methods are compared in order to get the most representative importance of the criteria. Weights are important and might have a large impact on the decision-making process. They will be used as an input to the selected outranking methods. The usage of pairwise comparison is limited to the criteria weights and is different the original technique of AHP.

3.4. Outranking

3.4.1. ELECTRE III

We relied upon outranking multicriteria decision-making process. In this context, ELECTRE III was selected among many other possible techniques for multiple reasons summarized in reference [46]. First, the context requires the use of non-compensatory aggregation procedures as the dropping on a given criterion cannot be compensated by the gain on another one. Second, the aggregation procedures usage is limited to decision problems with more than five criteria and less than twelve which can be a limitation on the use of our model. Under all of these constraints, ELECTRE III is considered the most adequate MCDM method that can support obsolescence resolution strategy selection problems.
ELECTRE III is used to rank alternatives. In this case, the alternatives are the resolution strategies. The multiple criteria decision-making method helps express the user preferences when a set of solutions must be ranked according to a set of contradictory criteria [46]. Outranking relations S are used to model preferences. S means “at least as good as”. Considering the two alternatives a and b :
  • a S b and not b S a , i.e., a P b ( a is strictly preferred to b );
  • b S a and not a S b , i.e., b P a ( b is strictly preferred to a );
  • a S b and b S a , i.e., a I b ( a is indifferent to b );
  • Not a S b and not b S a , i.e., a R b ( a is incomparable to b ) alternatives.
The decision maker contributes to the ranking process by defining the three preference parameters: the indifference thresholds q , preference thresholds p and veto thresholds v . Each objective or criterion has its own thresholds that will be used later to rank the alternatives when it comes to calculating concordance and discordance indexes.
The outranking in ELECTRE III is based on concordance and discordance principle. Alternatives are compared to each other and generating concordance index and discordance index. While the concordance c ( a S b ) (1) is considered when one alternative is as good as another, discordance d j ( a S b ) (2) calculates the possible rejection of one solution compared to another, basically when criteria are compared with consideration to thresholds:
c ( a S b ) = j J s w j + j J Q φ j w j s
d j ( a S b ) = { 1   i f   g j ( b ) > g j ( a ) + v j ( g j ( a ) ) 1   i f   g j ( b ) g j ( a ) + p j ( g j ( a ) ) g j ( b ) g j ( a ) p j ( g j ( b ) ) v j ( g j ( a ) p j ( g j ( a ) )   ,   o t h e r   w i s e
where the coefficient φ j decreases linearly from 1 to 0, and g j describes the range:
[ g j ( a ) + q j ( g j ( a ) ) ,   g j ( a ) + p j ( g j ( a ) ) ] J s = { j J   :   g j ( a ) + q j ( g j ( a ) ) g j ( b ) }     And , J Q = { j J   :   g j ( a ) + q j ( g j ( a ) ) < g j ( a ) g j ( b ) + p j ( g j ( b ) ) }
A credibility index ρ ( a S b ) (3) might later weaken the c ( a S b ) index of possible veto effects. A complete preorder is finally suggested considering preorders suggested by the distillation processes.
ρ ( a S b ) = c ( a S b )   j   J   : d j ( a S b ) > c ( a S b ) 1 d j ( a S b ) 1 c ( a S b ) ,

3.4.2. PROMETHEE

In general, multicriteria problems cannot be treated without additional information related to the preferences and the priorities of the decision makers. The information requested by PROMETHEE is particularly clear and easy to define for both decision makers and analysts. It consists of a preference function associated with each criterion as well as weights describing their relative importance.
Preference modeling regarding dominance relations is quite similar to ELECTRE III. Three dominance relations are considered: P , I and R , respectively, stand for preference, indifference and incomparability. They are defined below:
{   j :   g j ( a )   g j ( b ) k , g k ( a ) > g k ( b )       a P b ,   j :   g j ( a ) = g j ( b )     a I b {   s :   g s ( a ) > g s ( b ) r , g r ( a ) < g r ( b )       a R b ,
PROMETHEE II is selected to get a complete ranking. Its procedure is based on pairwise comparison of alternatives. After getting the weights set and the decision matrix, the amplitude deviation d j ( a , b ) and the deviation evaluation P j ( a , b ) are calculated. It denotes the difference between the evaluations of alternatives   a and b on criterion g j .
d j ( a , b ) = g j ( a ) g j ( b ) ,
P j ( a , b ) = F j [ d j ( a , b ) ]                 a , b   A ,   where   0 P j ( a , b ) 1
F j is the preference function. It could be usual, U-shape, V-shape, Level, V-shape with indifference or Gaussian models. The preference function, depending on its model, requires none or many from the following parameters to be defined: q , the threshold of indifference, p , the threshold of strict preference, and s , an intermediate value between q and p . While P j expresses the decision-maker preference for one criterion, the aggregated preference index π ( a , b ) (7) is calculated to express with which degree a is preferred to b over all the criteria.
π ( a , b ) = j = 1 k P j ( a , b ) w j
As soon as π ( a , b ) and π ( b , a ) are computed, positive outranking flow φ + (8) and negative outranking flow φ (9) are calculated to help complete ranking alternatives. When PROMETHEE II is considered, all the alternatives are comparable, and either P or I   dominances (10) result. No incomparabilities remain, and alternatives are ranked.
φ + ( a ) =   1 n 1 x     A π ( a , x )
φ ( a ) =   1 n 1 x     A π ( x , a )      
φ ( a ) = φ + ( a ) φ ( a ) a P b   i f f   φ ( a ) > φ ( b ) a I b   i f f   φ ( a ) = φ ( b )

4. Case Study

This case study was conducted at one of the largest international companies of vertical flights. The production facility includes various departments that handle numerous operations including the engineering department where change management is planned and executed. One of the most recurrent changes to make are those related to technological obsolescence. It happens unpredictably to any system at any stage of its life cycle. Technological obsolescence is mainly related to the obsolescence of technology over time which means that it happens necessarily at the end of life of the product [1] or at least after an important period of usage. With the fast technological evolution, technological obsolescence can be faced in the early stages of the product life cycle and even before the product’s useful life starts, at the production stage, for example [37]. The problem is more significant for sustainment-dominated systems [15], where the gap between these systems’ expected useful life and the interval of technological change continues to expand. The inherent impact of the obsolescence risk is hard to manage. The task of selecting the optimal resolution strategy is difficult. In this case study, the multicriteria decision-making method, described earlier, was applied to select the most suitable obsolescence resolution strategy considering sustainability criteria for one of the most sensitive parts of the system. First, fifteen selection criteria and nine alternatives were identified. The criteria weights were calculated using two different weighting methods involving multiple stakeholders and department managers. Following this, two ranking methods were applied to rank different resolution alternatives to solve the obsolescence case. The performance table is illustrated in Table 2. It includes nine alternatives or different resolution strategies and their performance on the fifteen criteria.
The first step after identifying the available alternatives and their performance is to determine the criteria weights. The results after normalization were close. The weights from both methods and the deviation (Table 3) were validated and approved by the decision makers. A total of eight scenarios are considered and compared (Table 4 and Table 6).

Implementing MCDM Ranking Methods

The ranking methods require multiple input parameters for the criteria. On one hand, for ELECTRE III, the criteria, their optimization, the indifference, preference and veto thresholds are in Table 5.
On the other hand, PROMETHEE requires a different format for criteria thresholds as each criterion has its own preference function which requires specific preference parameters. Appendix B describes the criteria data input for the case study.

5. Results and Discussion

5.1. MCDM Results

To evaluate the impact of the suggested framework, the results from the eight scenarios are described in Table 6.
While considering sustainability criteria, it is important to note that the first three alternatives were not sensitive to the weighting method as scenarios 1 and 3 results were almost the same in the first place. The same was observed for scenarios 5 and 7. Decision making using only cost, procurement, time, functionality, technology readiness level, open architecture and software compatibility was sensitive to the weighting method. It is explained by the fact that criteria weights change their importance when the mindset of the decision maker changes. When sustainability criteria were deleted, the cost and functionality criteria became the most important with large weights. It is not the case when considering sustainable criteria such hazardous material or pollution. In that case, there is no compromise and respecting legislation is a binary decision—one either respects it or not. That is why cost and functionality lose some of their weights.
It is clear that A3 dominates the first place in the sustainable ranking compared to A7 if sustainability is not considered. Comparing these two alternatives shows that the algorithm converged to a cheaper solution with acceptable technological characteristics compared to the best alternatives when the sustainable framework was used. In that case, alternatives respecting the environmental constraints with good technological performance were ranked first as the economical aspect is no longer a priority.

5.2. Sensitivity Analysis

Sensitivity analysis is an indispensable concept in the effective use of the quantitative decision model, and the purpose of this analysis is to assess the stability of an optimal solution under changes in parameters, the impact of unmanageability of some parameters and the need for accurate estimation of values of parameters. However, research on sensitivity analysis in multicriteria decision support is limited. The decision support process does not stop when one has an answer. A good decision support process must, generally, study the sensitivity of the parameters that led us to this response and the robustness of the response itself. After obtaining the final ranking of both methods, a sensitivity analysis is essential to measure the solution’s robustness. The solution sensitivity to both weights and thresholds can be studied. In our case, sensitivity to thresholds is not studied as these parameters are mostly determined by legal documents and standards. Where ELECTRE III sensitivity analysis was done manually by increasing and decreasing each weight by 10% and checking if the solution changed, PROMETHEE sensitivity analysis was performed using Visual PROMETHEE software. For all scenarios, solutions were stable for the first three alternatives. As an example, Table 7 describes the stability intervals for scenario 7.
The solution has multiple industrial contributions such as respecting international standards and legal obligations regarding environmental restrictions. It also defines a clear process ensuring sustainability and competitiveness with an easy preference modeling in a highly subjective decision-making environment. It allows the company to decrease pollution levels and hazardous materials usage while increasing customer satisfaction and technological performance.

6. Conclusions and Future Work

This paper describes a sustainable multicriteria decision framework to manage obsolescence. Once alternatives were determined, MCDM tools were employed to select the best obsolescence resolution strategy according to a set of sustainable criteria. This helps companies obtain the best trade-off between economic, social, environmental and technological constraints.
Since MCDM tools were employed, it was necessary to determine a set of criteria, their weights and the available alternatives. Prior to this work, quantitative MCDM tools were used but limited in most cases as they support neither qualitative data nor uncertainty. MCDM tools’ effectiveness and accuracy depend strongly on the problem and the decision maker. They require strong preference modeling. The case study demonstrated that in all scenarios, a ranking of the solutions was obtained, but it widely depended on the input parameters. It is acceptable to select the least sustainable option, but it may impact the company’s engagement and ethics and break legal rules related to the environmental aspect. Incorporating sustainability in the decision making is then important in a complex and constrained decision environment.
The comparison between the weighting methods demonstrated that pairwise is much more appreciated by the decision makers due to its simplicity and its comparison principle. Between ELECTRE III and PROMETHEE II, preference modeling was easier with the second method using preference functions. To support MCDM uncertainty, it is recommended to use both methods as they deal differently with preferences and there is no guarantee that one will always work better than the other. In addition, the sustainability of the decision is also important and is represented by its sensitivity and robustness. The sensitivity to the criteria weights was studied, but sensitivity thresholds could be done for further analysis. As noted previously, the success of the suggested framework is based on data quality and the company engagement in reducing its environmental impact to achieve sustainable global policy.
The results of this study indicate that MCDM ranking methods are effective in ORS selection via the proposed sustainable framework. This was validated with the comparison of the decisions before and after applying the sustainable criteria set. The research findings suggest that integrating sustainability to an unsustainable problem that combines economic, environmental, social and technological factors would provide useful insights to support decision making and improve obsolescence resolution strategies for a sustainable and green system.
Although the capability of the suggested framework and the MCDM ranking methods to handle sustainable decisions in the ORS selection process was approved in the current study, the application was limited to the reactive management level. The framework would be used for future research at the proactive management level to examine the impact of the strategies in the product design, for example. Alternatively, the effectiveness of the proposed MCDM tools could be compared to other decision models such as multi-objective optimization. In the case of the proactive management, dynamic decision making would be more advantageous considering the volatile and uncertain evolution of both technological and environmental parameters. However, more studies should be conducted to assess the environmental impact caused by obsolescence and to work on it proactively, reactively and strategically.

Author Contributions

Conceptualization and methodology design: I.Z., R.A.-M. and Y.B. Formal analysis: I.Z. and Y.B. Data curation and validation: I.Z., R.A.-M., Y.B. and M.P. Writing—original draft preparation: I.Z. Writing—review and editing: R.A.-M. and Y.B. Supervision: Y.B. and M.P. All authors have read and agreed to the published version of the manuscript.

Funding

The authors would like to acknowledge CRSNG and CRIAQ for funding the research through grant CRDPJ 459185-13.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table A1. A Summary of Most Relevant Decision-Making Papers in Obsolescence Resolution Strategy Selection.
Table A1. A Summary of Most Relevant Decision-Making Papers in Obsolescence Resolution Strategy Selection.
MethodologyCriteriaResolution Strategy
ReferencesMono-ObjectiveMulti-ObjectiveCostOther CriteriaLTBDesign RefreshOther Strategies
[19]x x xx
[47]x x x
[23]x x x
[21]x x x
[24]x x xx
[25]x x xx
[48]x x x
[27]x x x
[49]x xxxxx
[26]x x x
[50]x x x
[7] xxx x
[51]x x x
[52]x x x
[22]x x x
[20] xx x

Appendix B

Table A2. PROMETHEE GAIA Input.
Table A2. PROMETHEE GAIA Input.
Criteria’s CategoriesCriterionFunctionThreshold
EconomicCostLinear p = USD   2000
q = USD   6000
ProcurementU shape p = n / a
q = 1
Vendor reliabilityLevel p = 2
q = 1
Implementation and training timeGaussian s = 3
Customer satisfactionLevel p = 40
q = 30
EnvironmentalPollutionU Shape p = n / a
q = 20
Hazardous materialV Shape p = 02
q = n / a
Resource usageGaussian s = 4
SocialEmployment opportunitiesV Shape p = 5
q = n / a
Security and work environmentLinear p = 4
q = 2
TechnologicalFunctionalityLinear p = 3
q = 1
Technology readiness levelLevel p = 4
q = 2  
Life time duration/extensionV Shape p = 6
q = n / a  
Open architecture and standardsUsual p = n / a
q = n / a
Software compatibilityLinear p = 25
q = 10  

References

  1. Bartels, B.; Ermel, U.; Pecht, M.; Sandborn, P. Strategies to the Prediction, Mitigation and Management of Product Obsolescence; John Wiley & Sons: Hoboken, NJ, USA, 2012. [Google Scholar]
  2. Q Star. Approximately 3% of the Global Pool of Electronic Components Goes Obsolete Every Month; Technical Report; Q Star Technologies: Navarre, FL, USA, 2006. [Google Scholar]
  3. Pingle, P. Selection of Obsolescence Resolution Strategy Based on a Multi Criteria Decision Model. Ph.D. Thesis, Iowa State University, Ames, IA, USA, 2015. [Google Scholar]
  4. Eionet. Electronic Products and Obsolescence in a Circular Economy; ETC/WMGE 2020/3; Eionet: Copenhagen, Denmark, 2020. [Google Scholar]
  5. Baldé, C.P.; Forti, V.; Gray, V.; Kuehr, R.; Stegmann, P. The Global E-Waste Monitor 2017: Quantities Flows and Resources; United Nations University: Tokyo, Japan; International Telecommunication Union: Geneva, Switzerland; International Solid Waste Association: Rotterdam, The Netherlands, 2017. [Google Scholar]
  6. Seuren, T.; Tan, T.; Vermeulen, B.; Rosierse, R.; Marine, R.; Draijer, J.W.; Navy, R.N. From Reactive to Proactive Obsolescence Management. Master’s Thesis, Eindhoven University, Eindhoven, The Netherlands, 2018. [Google Scholar]
  7. Adetunji, O.; Bischoff, J.; Willy, C.J. Managing system obsolescence via multicriteria decision making. Syst. Eng. 2018, 21, 307–321. [Google Scholar] [CrossRef]
  8. Reimert, W. Obsolescence Management Strategies at Thales Hengelo; University of Twente: Enschede, The Netherlands, 2020. [Google Scholar]
  9. Echegaray, F. Consumers’ reactions to product obsolescence in emerging markets: The case of Brazil. J. Clean. Prod. 2016, 134, 191–203. [Google Scholar] [CrossRef]
  10. Ewastemonitor. The Global E-Waste Monitor. 2020. Available online: http://ewastemonitor.info/ (accessed on 1 March 2021).
  11. Sabbaghi, M.; Behdad, S. Environmental Evaluation of Product Design Alternatives: The Role of Consumer’s Repair Behavior and Deterioration of Critical Components. J. Mech. Des. Trans. ASME 2017, 139. [Google Scholar] [CrossRef]
  12. Jennings, C.; Wu, D.; Terpenny, J. Forecasting obsolescence risk and product life cycle with machine learning. IEEE Trans. Compon. Packag. Manuf. Technol. 2016, 6, 1428–1439. [Google Scholar] [CrossRef]
  13. Sandborn, P. Forecasting technology and part obsolescence. J. Eng. Manuf. 2017, 231, 2251–2260. [Google Scholar] [CrossRef]
  14. Adams, C. Getting a handle on COTS obsolescence. Avion. Mag. 2005, 1. Available online: https://www.aviationtoday.com/2005/05/01/getting-a-handle-on-cots-obsolescence/ (accessed on 1 August 2021).
  15. Wilkinson, C. Obsolescence and Life Cycle Management for Avionics. 2015. Available online: http://www.tc.faa.gov/its/worldpac/techrpt/tc15-33.pdf (accessed on 1 August 2021).
  16. Büyüközkan, G.; Karabulut, Y. Sustainability performance evaluation: Literature review and future directions. J. Environ. Manag. 2018, 217, 253–267. [Google Scholar] [CrossRef] [PubMed]
  17. Li, Y.; Mathiyazhagan, K. Application of DEMATEL approach to identify the influential indicators towards sustainable supply chain adoption in the auto components manufacturing sector. J. Clean. Prod. 2018, 172, 2931–2941. [Google Scholar] [CrossRef]
  18. Singh, R.K.; Murty, H.R.; Gupta, S.K.; Dikshit, A.K. An overview of sustainability assessment methodologies. Ecol. Indic. 2009, 9, 189–212. [Google Scholar] [CrossRef]
  19. Porter, G.Z. An Economic Method for Evaluating Electronic Component Obsolescence Solutions; Boeing: Chicago, IL, USA, 1998. [Google Scholar]
  20. Shi, Z.; Liu, S. Optimal inventory control and design refresh selection in managing part obsolescence. Eur. J. Oper. Res. 2020, 287, 133–144. [Google Scholar] [CrossRef]
  21. Feng, D.; Singh, P.; Sandborn, P. Lifetime buy optimization to minimize lifecycle cost. In Proceedings of the 2007 Aging Aircraft Conference, Palm Springs, CA, USA, 16–19 April 2007. [Google Scholar]
  22. Acevedo-Ojeda, A.; Contreras, I.; Chen, M. Two-level lot-sizing with raw-material perishability and deterioration. J. Oper. Res. Soc. 2020, 71, 417–432. [Google Scholar] [CrossRef]
  23. Singh, P.; Sandborn, P. Obsolescence driven design refresh planning for sustainment-dominated systems. Eng. Econ. 2006, 51, 115–139. [Google Scholar] [CrossRef] [Green Version]
  24. Hu, G.; Bidanda, B. Modeling sustainable product lifecycle decision support systems. Int. J. Prod. Econ. 2009, 122, 366–375. [Google Scholar] [CrossRef]
  25. Kumar, U.D.; Saranga, H. Optimal selection of obsolescence mitigation strategies using a restless bandit model. Eur. J. Oper. Res. 2010, 200, 170–180. [Google Scholar] [CrossRef]
  26. Zheng, L.; Terpenny, J.; Sandborn, P. Design refresh planning models for managing obsolescence. IIE Trans. 2015, 47, 1407–1423. [Google Scholar] [CrossRef]
  27. Meng, X.; Thornberg, B.; Olsson, L. Strategic proactive obsolescence management model. IEEE Trans. Compon. Packag. Manuf. Technol. 2014, 4, 1099–1108. [Google Scholar] [CrossRef]
  28. Werey, C.; Le Gat, Y.; Curt, C.; Tourment, R.; Tacnet, J.M. Asset management for water and sewer networks. levees and mountain protection works: Sectorial developments and crossed perspectives. Tech. Sci. Methodes 2018, 113, 161–185. [Google Scholar]
  29. Bowlds, T.F.; Fossaceca, J.M.; Iammartino, R. Software obsolescence risk assessment approach using multicriteria decision-making. Syst. Eng. 2018, 21, 455–465. [Google Scholar] [CrossRef]
  30. Battini, D.; Calzavara, M.; Isolan, I.; Sgarbossa, F.; Zangaro, F. Sustainability in material purchasing: A multi-objective economic order quantity model under carbon trading. Sustainability 2018, 10, 4438. [Google Scholar] [CrossRef] [Green Version]
  31. Herat, S. Sustainable management of electronic waste (e-waste). Clean Soil Air Water 2007, 35, 305–310. [Google Scholar] [CrossRef]
  32. Guitouni, A.; Martel, J.-M. Some Guidelines for Choosing an MCDA Method Appropriate to a Decision Making Context; Faculté des Sciences de l’Administration de l’Université Laval: Quebec, QC, Canada, 1997. [Google Scholar]
  33. Kottemann, J.E.; Davis, F.D. Decisional conflict and user acceptance of multicriteria decision-making aids. Decis. Sci. 1991, 22, 918–926. [Google Scholar] [CrossRef]
  34. Adelbratt, T.; Montgomery, H. Attractiveness of decision rules. Acta Psychol. 1980, 45, 177–185. [Google Scholar] [CrossRef]
  35. Ziemba, P. Towards strong sustainability management—A generalized PROSA method. Sustainability 2019, 11, 1555. [Google Scholar] [CrossRef] [Green Version]
  36. Shaw, W.; Speyerer, F.; Sandborn, P. Diminishing Manufacturing Sources and Material Shortages (DMSMS) Non-Recurring Engineering (NRE) Cost Metric Update; Defense Microelectronics Activity: McClellan, CA, USA, 2010. [Google Scholar]
  37. Sandborn, P. Design for obsolescence risk management. Procedia CIRP 2013, 11, 15–22. [Google Scholar] [CrossRef] [Green Version]
  38. Fiksel, J. Sustainability and resilience: Toward a systems approach. Sustain. Sci. Pract. Policy 2006, 2, 14–21. [Google Scholar] [CrossRef]
  39. Efendigil, T.; Ozkir, V.; Demirel, N.C.; Demirel, T.; Ondemir, O. A framework proposal upon the estimation of electronic-waste products in Istanbul. In Proceedings of the 10th International Fuzzy Logic and Intelligent Technologies in Nuclear Science Conference, FLINS 2012, Istanbul, Turkey, 26–29 August 2012; pp. 1184–1189. [Google Scholar]
  40. Feil, A.A.; Schreiber, D.; Haetinger, C.; Strasburg, V.J.; Barkert, C.L. Sustainability indicators for industrial organizations: Systematic review of literature. Sustainability 2019, 11, 854. [Google Scholar] [CrossRef] [Green Version]
  41. Herald, T.; Genaw, D. Technology and obsolescence sustainment for integrated systems. In Proceedings of the 16th Annual International Symposium of the International Council on Systems Engineering, INCOSE 2006, Orlando, FL, USA, 9–13 July 2006; pp. 456–471. [Google Scholar]
  42. Hemdi, A.R.; Saman, M.Z.M.; Sharif, S. Sustainability evaluation for decision making. In Proceedings of the 11 Asia Pacific Industrial Engineering and Management Systems Conference, APIEMS2010, Melaka, Malaysia, 7–10 December 2010; pp. 1–6. [Google Scholar]
  43. Shokrian, M.; High, K.A.; Sheffert, Z. Screening of process alternatives based on sustainability metrics: Comparison of two decision-making approaches. Int. J. Sustain. Eng. 2015, 8, 26–39. [Google Scholar] [CrossRef]
  44. Saaty, T. The Analytic Hierarchy Process Mcgraw Hill, New York. Agric. Econ. Rev. 1980, 70. Available online: https://www.scirp.org/(S(lz5mqp453edsnp55rrgjct55))/reference/ReferencesPapers.aspx?ReferenceID=1943982 (accessed on 1 August 2021).
  45. Buckley, J.J. Fuzzy hierarchical analysis. Fuzzy Sets Syst. 1985, 17, 233–247. [Google Scholar] [CrossRef]
  46. Figueira, J.; Mousseau, V.; Roy, B. ELECTRE methods. In Multiple Criteria Decision Analysis: State of the Art Surveys; Springer: Berlin, Germany, 2005; pp. 133–153. [Google Scholar]
  47. Teunter, R.H.; Fortuin, L. End-of-life service. Int. J. Prod. Econ. 1999, 59, 487–497. [Google Scholar] [CrossRef]
  48. Zheng, L.; Terpenny, J.; Sandborn, P.; Nelson, R. Design refresh planning models for managing obsolescence. In Proceedings of the ASME 2012 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC/CIE 2012, Chicago, IL, USA, 12 August 2012; pp. 579–590. [Google Scholar]
  49. Yuelin, S.; Willems, S.P. Modeling sourcing strategies to mitigate part obsolescence. Eur. J. Oper. Res. 2014, 236, 522–533. [Google Scholar] [CrossRef]
  50. Caetani, A.P.; Ferreira, L.; Borenstein, D. Development of an integrated decision-making method for an oil refinery restructuring in Brazil. Energy 2016, 111, 197–210. [Google Scholar] [CrossRef] [Green Version]
  51. Shi, Z. Optimal remanufacturing and acquisition decisions in warranty service considering part obsolescence. Comput. Ind. Eng. 2019, 135, 766–779. [Google Scholar] [CrossRef]
  52. Nguyen, P.H.; Wang, K.-J. Strategic capacity portfolio planning under demand uncertainty and technological change. Serv. Manuf. J. 2019, 31, 926–944. [Google Scholar] [CrossRef]
Figure 1. Research methodology chart.
Figure 1. Research methodology chart.
Sustainability 13 08601 g001
Table 1. Criteria table.
Table 1. Criteria table.
Criteria’s CategoriesCriterionAssessment
EconomicCostmin
Procurementmax
Vendor reliabilitymax
Implementation and training timemin
Customer satisfactionmax
EnvironmentalPollutionmin
Hazardous materialmin
Resource usagemin
SocialEmployment opportunitiesmax
Security and work environmentmax
TechnologicalFunctionalitymax
Technology readiness levelmax
Lifetime duration/extensionmax
Open architecture and standardsmax
Software compatibilitymax
Table 2. Performance matrix.
Table 2. Performance matrix.
CriteriaUnitA1A2A3A4A5A6A7A8A9
C (Cost)USD766523,500913625,00016,50021,000220019,00022,000
PR (Procurement)5-point scale343435243
VR (Vendor Reliability)5-point scale445534545
T (Implementation and Training Time)Days10141228101441220
CS (Customer Satisfaction)%658060636868558780
PL (Pollution)5-point scale434353533
HM (Hazardous Material)SI units161514161514211116
RU (Resource Usage)9-point scale636444655
EM (Employment)5-point scale343444243
SWE (Security and Work Environment)9-point scale765666466
F (Functionality)5-point scale344534335
TRL (Technology Readiness Level)1–9 TRL levels676777578
LTE (Lifetime Duration/Extension)Months244036403638124040
OAS (Open Architecture and Standards)0/1101011011
SC (Software Compatibility)%806575507560908270
Table 3. Normalized criteria weights obtained using the swing and pairwise methods.
Table 3. Normalized criteria weights obtained using the swing and pairwise methods.
EconomicEnvironmentalSocialTechnological
CPRVRTCSPLHMRUEMSWEFTRLLTEOASSC
Swing Weights16.5%8.6%6.3%5.0%3.2%5.7%7.3%5.1%1.2%1.8%14.0%9.8%13.8%1.6%3.0%
Pairwise Comparison Weights 17.6%7.4%6.2%5.3%2.1%5.5%8.6%5.2%1.0%2.0%13.9%10.4%11.2%1.1%2.5%
Table 4. Description of different scenarios’ parameters.
Table 4. Description of different scenarios’ parameters.
ScenarioWeightingRankingSustainable Criteria
1SwingELECTRE IIIYes
2SwingELECTRE IIINo
3SwingPROMETHEEYes
4SwingPROMETHEENo
5Pairwise ComparisonELECTRE IIIYes
6Pairwise ComparisonELECTRE IIINo
7Pairwise ComparisonPROMETHEEYes
8Pairwise ComparisonPROMETHEENo
Table 5. Criteria thresholds for ELECTRE III.
Table 5. Criteria thresholds for ELECTRE III.
EconomicEnvironmentalSocialTechnological
CPRVRTCSPLHMRUEMSWEFTRLLTEOASSC
OptimizationMinMaxMaxMinMaxMinMinMinMaxMaxMaxMaxMaxMaxMax
Indifference threshold q2000112100121201305
Preference threshold p35002263012324136120
Veto threshold v10,0004414603107453524230
Table 6. Ranking results for all scenarios.
Table 6. Ranking results for all scenarios.
RankingScenario 1Scenario 2Scenario 3Scenario 4Scenario 5Scenario 6Scenario 7Scenario 8
1A3A7A3A7A3A7A3A7
2A9A2A9A6A9A4A9A3
3A1A9A1A1A1A9A1A1
4A2, A8A6A8A5A2A6A6A5
5---A1A2A8A8A1A8A8
6A6A3A6A3A6A2A2A6
7A7A4A4A9A7A3A4A9
8A5A5, A8A7A2A5A5A7A2
9A4 A5A4A4A8A5A4
Table 7. PROMETHEE stability intervals for scenario 3.
Table 7. PROMETHEE stability intervals for scenario 3.
CriteriaWeightMinMax
C17.6%12.26%23.26%
PR7.4%0%15.33%
VR6.2%0.4%31.48%
T5%0%12.59%
CS2.1%0%19.76%
PL5.5%0.43%18.27%
HM8.6%0%25.78%
RU5.2%0.0%39.9%
EM1%0.0%15.65%
SWE2%0.0%30.38%
F13.9%0.3%23.33%
TRL10.4%9.4%48.44%
LTE11.2%6.65%16.77%
OAS1.1%0%100%
SC2.5%0%20.29%
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Zaabar, I.; Arango-Miranda, R.; Beauregard, Y.; Paquet, M. A Sustainable Multicriteria Decision Framework for Obsolescence Resolution Strategy Selection. Sustainability 2021, 13, 8601. https://0-doi-org.brum.beds.ac.uk/10.3390/su13158601

AMA Style

Zaabar I, Arango-Miranda R, Beauregard Y, Paquet M. A Sustainable Multicriteria Decision Framework for Obsolescence Resolution Strategy Selection. Sustainability. 2021; 13(15):8601. https://0-doi-org.brum.beds.ac.uk/10.3390/su13158601

Chicago/Turabian Style

Zaabar, Imen, Raul Arango-Miranda, Yvan Beauregard, and Marc Paquet. 2021. "A Sustainable Multicriteria Decision Framework for Obsolescence Resolution Strategy Selection" Sustainability 13, no. 15: 8601. https://0-doi-org.brum.beds.ac.uk/10.3390/su13158601

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop