R&D Project Selection Incorporating Customer-Perceived Value and Technology Potential: The Case of the Automobile Industry

Abstract

As user-centric innovation has recently emerged as a successful way of developing new products, services, and concepts, it is worth considering the perspectives of potential technology users during R&D project selection processes. Nevertheless, little effort has been made to reflect customer-perceived value in establishing selection criteria, with the focus mainly on technological potential instead. Therefore, this study aims to develop an R&D project selection model incorporating not only technological potential but also customer-perceived value. For this purpose, a new R&D project evaluation model and process is proposed, and its feasibility is tested by potential users in a real scenario. The automobile industry is suitable for our evaluation model because it is a B2C and system-based industry where customer needs are critical to market success and a number of R&D projects are proposed every year. Finally, a supporting tool is developed to help interact with various evaluators and visualize the evaluation results, as customer involvement is recommended for accurate project evaluation from the perspective of technology users. This study is one of the earliest attempts to reflect customer-perceived value in R&D project selection, and practically, the research outputs are expected to be useful to automobile manufacturers in creating value from R&D projects.

1. Introduction

In an environment of rapidly changing technologies and mega-competition on a global scale, many industries are undergoing radical transformations, and firms are under pressure to act quickly to respond to these threats. Central to their survival and growth strategy is the need to innovate, as new technology may offer companies a leading edge over competitors in the current market and open new markets [1]. That is, R&D, as a core of innovation, plays a major role in developing competitive technologies to support core business growth.

The ability to consistently select the best R&D projects to invest is thus essential to increasing the probability of successful R&D investments, which has long been a major concern for both practices and academics. According to Cooper et al. (2000) [2], companies that use formal project selection approaches show greater project launch success as well as better sales and profit performance than others. As a result, extensive academic research has been conducted to help organizations make better decisions in R&D project selection, and a number of decision models and methods, such as analytic hierarchical process (AHP)-based modes [3], fuzzy evaluation processes [4,5], and portfolio-based methods [6,7], have been developed in the past four decades. However, current research findings indicate that many such complex models and methods are not being used, having only limited effects on decision-making for real-world project selection [8,9]. In a similar vein, Cooper et al. (2001) [10] found that, among the many project selection models, financial methods have been the most frequently used at large firms, but companies using a scoring model, which is quite simple to use, together with financial analysis, generally outperform companies relying merely on the financial aspects of projects.

When a company employs a scoring model, it must adopt a set of R&D project selection criteria, which may vary across companies or studies. For example, Henriksen et al. [11] suggested four criteria, relevance, risk, reasonableness, and return, while Lawson et al. [1] proposed six criteria incorporating technical, corporate, and strategic regulatory, market, financial, and application aspects. Oliveira [12] further developed a method to analyze such criteria-based decision-making process at the fuzzy front end of innovation. Despite the variety of criteria, most studies have commonly incorporated the technological and market potential of R&D projects in their project selection models, where market potential has been measured mainly by financial value from the perspectives of technology providers.

Though financial value is one of the most important measures, as indicated in previous studies, it is also worth considering the perspectives of potential technology users during the R&D project selection processes. In measuring the market potential of R&D projects, it is significant to analyze what potential utilities a new technology will bring to users, whether the utilities are regarded as valuable to users or not, and whether the potential users are likely to pay for the value or not. Customer-perceived value is defined as “the consumer’s overall assessment of the utility of a product based on perceptions of what is received and what is given” [13] (p. 14). In the marketing literature, it is considered an underlying source of competitive advantage [14], being one of the most influential forces in a marketplace [15]. Hence, R&D projects must be evaluated from the perspectives of technology users as well as technology providers to increase the accuracy of evaluation. Nevertheless, little effort has been made to reflect the customer-perceived value of new technology as outputs of R&D in establishing criteria for R&D project selection. Particularly in the automobile industry, which is the focus of this study, several R&D projects are suggested simultaneously. Among them, some new technologies will be incorporated in a new model as a basic function and others as an optional function, and some options will be preferred by customers, generating extra revenue for automakers, while others will not. Hence, the value of new technologies as perceived by potential customers is vital for companies to decide which R&D projects to invest in.

To meet the aforementioned needs, this study aims to develop a new R&D project selection model incorporating both technological potential and customer-perceived value and apply them to an automobile manufacturer in Korea, trying to answer the following research question: how can we identify potentially valuable R&D projects from the customers’ perspective? For this purpose, first, a new R&D project evaluation model and process are proposed, and second, its feasibility is tested by potential users in a real scenario. Finally, a supporting tool is developed to help interact with various participants and visualize the evaluation results, as customer involvement is recommended for accurate R&D project evaluation from the perspective of technology users. Theoretically, this study is one of the earliest attempts to reflect customer-perceived value in R&D project selection, and practically, the research outputs are expected to be used in automobile manufacturing.

The remainder of this paper is organized as follows. Section 2 briefly reviews the previous studies, and Section 3 describes the overall processes. In Section 4, a new R&D project selection model is proposed in the context of automobile manufacturing, and a prototype system to implement the model is developed. Finally, Section 5 discusses the contributions, limitations, and future directions of this study.

2. Background

2.1. R&D Project Evaluation

R&D project selection is an organizational decision-making process commonly employed in such organizations as technology-intensive companies, public and private funding agencies, universities, and research institutes [16]. It is complicated and challenging work, firstly because the future success or impacts of candidate projects are hard to predict and secondly because a multi-stage multi-person decision-making process is required to obtain a reliable outcome [9,11].

Consequently, over the past 35 years, extensive research has been carried out and produced a wide range of methods to support R&D project selection [1]. These methods can be grouped into two categories—financial and non-financial—according to their approaches. First, in a financial approach, the potential value of R&D projects are is mainly based on financial terms, that is, expected return on investment (ROI) or net present value (NPV). However, this approach has several limitations. It fails to consider the strategic importance of R&D projects, as some projects that are low in ROI may be essential for a company to achieve its long-term strategic goals. Furthermore, this approach is not applicable to all R&D projects, particularly those at the early stage of development, since forecasting accurate cash flows associated with the projects is likely to be infeasible for such projects. Indeed, when large firms in the USA with proven success in R&D activities were surveyed, it was found that those using a scoring model and risk analysis as well as financial analysis generally had better performance than those using only financial analysis [2].

In a non-financial approach, experts are used to evaluate potential values of R&D projects [17,18]. What is important here is the selection of projects using appropriate criteria [19]. Recently, with the growing complexity of technologies, strategic technology management must consider a number of factors, such as technology paradigm shifts, socioeconomic pressures for change, the management of diverse knowledge bases, and global standards and intellectual property [20]. Quite naturally, these factors should also be reflected in developing the criteria for R&D project selection to deal with the high degree of uncertainty that results from non-technical attributes of R&D projects. Accordingly, various decision-supporting models with different sets of criteria are developed and applied to R&D project selection. For example, Pretorius and de Wet [21] suggested a framework to investigate the impact of manufacturing technology on corporate productivity and competitiveness based on three dimensions—hierarchy, fundamental functions, and business cycle. Assefa et al. [22] developed a technology valuation method considering technological and economic aspects. Kalbar et al. [23] measured the value of technology in terms of the technology life cycle, robustness, sustainability, and so on. Then, in aggregating the values for several criteria, previous studies have adopted a multi-criteria decision-making (MCDM) model, which helps a decision-maker to choose the most desirable options (R&D projects in our case) based on conflicting criteria. The MCDM models frequently applied include AHP [3], analytic network process (ANP) [24], the technique for order of preference by similarity to ideal solution (TOPSIS) [22], data envelopment analysis (DEA) [25], fuzzy-set theory [4,5], and decision-making trial and evaluation laboratory (DEMATEL) [26]. Sometimes, these techniques are integrated to better support decision-making (e.g., [27,28]).

This study is in line with MCDM-based research but takes a simple approach: a scoring model, incorporating both financial and risk aspects as criteria for project selection, is adopted as a basic model, while the AHP technique is applied to assign weights to each criterion.

2.2. Characteristics of R&D Project Evaluation in the Automobile Industry

Since the core technologies underlying the automobile may be considered mature, leading automobile manufacturers have been investing heavily on R&D not only to enhance their products’ potential to satisfy customer needs but also to achieve the next technological breakthrough in the industry [20]. Selecting the best R&D projects among a number of candidates is a primary concern for manufacturers; it has a significant influence on new product development and manufacturing practices, and thus some cautious decision-making to allocate resources for new technology developments is required based on a comparison of a number of available technological options.

Moreover, the automobile industry has been undergoing radical changes recently. As environmental awareness is growing, the energy paradigm is shifting to replace conventional fuel with other energy sources, such as hydrogen or electronics [29], and environmental regulations are expected to influence diffusion patterns of new technology in the industry [30]. Customers are becoming more demanding about safety and performance improvements in automobiles [20]. In addition, new technologies such as IT are emerging and embodied in automobiles as core components, exerting a profound impact on both R&D and manufacturing processes; this is expected to cause a global restructuring of the industry.

Automobile manufacturing is a “system industry” and “B2C industry” sector, so several issues must be considered in developing an R&D project selection model. First, it is a “system industry” with growing complexity. Automobiles have numerous parts that are interrelated to each other, which may be hard to understand fully, and require an understanding of the interaction between the objects and even with a human being [20,31]. Changes in one part as a result of new technology development will affect the other parts, and these possible changes, whether they are positive or negative, necessitate earnest study and examination. In a similar vein, automobile technologies by nature are multi-use technologies that benefit multiple segments of an industry. That is, a new technology developed for a particular product line can be used for other lines manufactured by a company. Accordingly, these spillover effects at the enterprise level must be considered in evaluating the technological potential of R&D projects.

Second, automobile manufacturing is a “B2C industry” sector, so incorporating customer requirements is critical for technological innovation. There are growing expectations for safety and comfort in vehicles and growing awareness of environmental issues, driving automobile makers to strive to develop environmentally friendly and energy-efficient vehicles. In addition to these major concerns, customer requirements are becoming more sophisticated, so new technology offerings should be carefully evaluated to ensure that they are in line with the requirements. In addition, it should be noted that new technologies are embodied through two-stage R&D investment processes—the first stage for product technologies and the second for production technologies. Different criteria are needed for different types of R&D projects, so the characteristics of the R&D stages should be considered in developing an R&D project selection model. In our scenario, we focus on the first-stage R&D, which involves more complicated decision-making criteria.

2.3. Customer-Perceived Value

Value is determined based on a trade-off between costs and benefits, representing an exchange between what is obtained and what is given [32]. Thus, customer value is created when the degree to which someone judges that what he has gained, that is, the benefits of a product or service perceived by the customer, is superior to what he has lost in the exchange, that is, all the related costs, such as price, search, operation, and disposal [33,34]. These benefits and costs can be interpreted in both monetary and non-monetary terms. Non-monetary benefits include competence, market position, and social rewards [35], whereas non-monetary costs are related to the time, effort, and energy consumed to obtain the product or service [36].

The evaluation of value is subjective in nature [37,38], and the concept of customer-perceived value has been examined extensively by marketing researchers. Perceived value is the global evaluation of the utility of a product or service based on the customer perception of what is given and what is received [13]. Because of this subjectivity, perceived value is hard to measure, particularly for products and services with intangible elements or complexity, for benefits or costs that are not immediate, and for products or services that are new to the customer.

With all these difficulties, recent studies have tried to measure the degree of perceived value and conceptualized it as a multidimensional construct [39,40]. For instance, Floh et al. [41] adopted a multi-dimensional conceptualization of value that includes functional, economic, emotional, and social value. Traditionally, valuations have included product or service attributes, pricing, and elements of the delivery process [42], emphasizing the functional and economic aspects of value, while the multi-dimensional approach overcomes traditional approaches’ over-concentration on economic value [43]. Nevertheless, this study is limited in focus to functional and economic value because the other constructs such as emotional and social values are difficult to measure before a product or service is widely diffused.

3. Overall Research Framework

3.1. Research Process

The overall research process consists of five steps, as shown in Figure 1. In the first step, the needs and conditions of R&D project selection environments are examined. Based on the analysis results in the previous step and a rich literature review, the most appropriate evaluation approach is selected in the next step. In the third and fourth steps, an evaluation model for R&D project selection is developed, its feasibility is validated, and an evaluation process is designed. Finally, a prototype system to support the evaluation process is designed in the fifth step. The detailed procedures are explained in the following section.

3.2. Detailed Procedures

3.2.1. Step 1. Analyzing the Context of R&D Project Evaluation

An essential prerequisite for developing an R&D project selection model is the analysis of the context in which the R&D project evaluation is made. The characteristics of technologies targeted by candidate R&D projects must be understood. The resources, including the financial as well as human resources available for the evaluation, must be estimated. In addition, the data required and their availability for the evaluation must be analyzed. In this study, we developed a clear understanding of automobile technologies and the characteristics of R&D projects to be evaluated, targeting product technologies rather than production technologies. Moreover, time and budgets allowed for evaluation were analyzed; these factors were very limited because the organization encouraged engineers to suggest as many creative R&D projects as possible, so the minimum evaluation was the most desirable.

3.2.2. Step 2. Selecting an Appropriate Evaluation Approach

Once the context of R&D project evaluation is analyzed, the evaluation approach that is the most appropriate for the context is selected. To do this, the research trends regarding technology evaluation are reviewed, and various alternatives for key elements in the approaches are identified as follows:

• Evaluation model: financial models, scoring models, and so on.
• Participants: internal expert panels, external expert panels, customers, and so on.
• Data type for evaluation: quantitative data, qualitative data, and so on.
• Methods for evaluation: absolute value, relative value, checklist, guided scoring, and so on.

Then, the advantages and disadvantages of alternatives for each key element are compared to suggest the best approach for the context. In our case, a weighted scoring model was adopted as a practical model considering the limited time and budgets but a number of R&D projects for evaluation. Moreover, most of our R&D projects were targeting technologies in the development stage, and elaborated financial models were difficult to use. Furthermore, it seemed to be easier to reflect customer-perceived value in a scoring model than in a financial model. Participants as evaluators included expert panels and customers to reflect customer-perceived value. As for data type and methods for evaluation, we concluded that different alternatives needed to be adopted for different evaluation criteria.

3.2.3. Step 3. Developing and Validating an Evaluation Model

When developing and validating an evaluation model, the evaluation perspectives and criteria should be defined first. To consider both technological potential and customer-perceived value, we chose technology and market/business aspects as one dimension to define evaluation perspectives. Furthermore, because good R&D projects are those corresponding to external drivers and based on internal competencies, we took into account attractiveness and corporate fit as another dimension, referencing the BMO (Bruce Merrifield-Ohe) approach [39]. By combining the two dimensions, four perspectives are identified, and two criteria for each perspective are defined: technology attractiveness—“innovativeness” and “appropriability”; market attractiveness—“customer-perceived functional value” and “customer-perceived economic values”; technology fit—“R&D capabilities” and “manufacturing capabilities”; and business fit—“suitability for new products” and “suitability for R&D investment” (see Figure 2).

Then, the evaluation indexes for criteria are determined and AHP analysis is conducted to assign weight to the criteria. The evaluation indexes are proxy measures of evaluation criteria, and more than one index can be used for one criterion. The AHP, introduced by Saaty (1994) [44], is a multi-criteria decision-making approach that is broadly applied in calculating the weight of different criteria to change the complex qualitative analysis into simple and transparent quantitative analysis. In the case study, two or three indexes for each evaluation criteria were adopted from those verified in the previous studies and the AHP was applied to assess priority weights for different evaluation indexes. Finally, the feasibility of the suggested R&D project selection model must be tested. In the case study, an expert panel was employed to verify the appropriateness of evaluation perspectives, criteria, and indexes. In addition to the use of an expert panel, a pilot evaluation was carried out to check the degree of consistency in evaluation of the same R&D projects by different evaluators. If the consistency is high, the validity is also high.

3.2.4. Step 4. Developing an Evaluation Process

After an evaluation model has been established, it is possible to devise an evaluation process. In this step, the most suitable evaluators for criteria (or indexes) are suggested, and specific processes are defined to guide their effective and efficient participation in the evaluation.

3.2.5. Step 5. Designing a Supporting System for Evaluation

A supporting system for evaluation can help increase the efficiency of an evaluation by collecting data from a number of different evaluators, synthesizing them, and visualizing the evaluation results. Therefore, in the final step, a supporting system for evaluating R&D projects is designed, and its Web-based prototype system is developed. The system will be particularly useful when external experts or customers in different regional areas are involved in the evaluation.

4. R&D Project Evaluation Method

4.1. Developing and Validating an Evaluation Model

4.1.1. Evaluation Perspectives, Criteria, and Indexes

The evaluation model suggested in this study comprises four perspectives, eight criteria, and 20 indexes, as shown in

Table 1. Evaluation perspectives, criteria, and indexes for R&D project selection.

Perspectives	Criteria	Indexes	Description	References
Technology attractiveness	Innovativeness	Novelty	The novelty of the proposed technology—whether similar technologies exist or not	Patentability requirements [45]
		Inventive steps	The superiority of the proposed technology to the existing competitive technology
		Utility	The utility, identifiable benefits, that the proposed technology provides and the capability of its use in the industry
	Appropriability	Exclusivity	The possibilities to obtain intellectual property rights for the proposed technology	Technology valuation [46,47]
		Life cycle	The technology life cycle remained—the possibilities of obsolescence
Technology fit	R&D capabilities	R&D difficulties	The difficulties in developing the proposed technology	BMO—technology capabilities [48]
		R&D risk	Technical risk to R&D project completion (e.g., patent infringement risk, technology complexity, resource limits)
		Application domain	The range of products or services to which the proposed technology can be applied—product range, growth potential, synergy with other products/processes
	Manufacturing capabilities	Additional cost for manufacturing	Additional cost required to commercialize the proposed technology in new products or services—ability to implement production or service offering process (e.g., the availability of existing facilities, the size and complexity of new facilities)	BMO—manufacturing capabilities [48]
		Additional time for manufacturing	Additional time required to commercialize the proposed technology in new products or services
Market attractiveness	Perceived functional value	Functional awareness	The degree of recognition for the function offered by the proposed technology—the ease of awareness by users	Perceived value—functional [42]
		Functional importance	The degree of perceived importance for the function offered by the proposed technology
	Perceived economic value	Brand awareness	The possibilities of improving brand awareness through the proposed technology	Perceived value—economic [42]
		Intention to pay	The possibilities of directly pricing the function offered by the proposed technology
		Price to cost	The ratio of price to cost for the function offered by the proposed technology
Business fit	Fit with product plan	Applicability	The applicability of the proposed technology to new products or services that are planning to be introduced	Impacts of R&D [49,50]
		Impacts on performance	The negative or positive impacts of inserting the proposed technology to products or services on their core performance
	Fit with R&D plan	Investment cost	Availability of investment costs	Validity of R&D [24]
		ROI	The level of expected ROI
		Time and human resources	Suitability of project schedule and human resources planning

. From the perspective of technology attractiveness, the “innovativeness” and “appropriability” of R&D projects were selected as key criteria. Technologies will be attractive, as they are innovative, and an innovator’s ability to capture profits generated by the technologies is superior to the other technologies: innovativeness is measured by three indexes concerning patentability requirements, while appropriability is measured by two indexes commonly used for technology valuation.

From the perspective of technology fit, we adopted two criteria, corporate “R&D capabilities” and “manufacturing capabilities” by referencing technology fit-related criteria of BMO analysis devised for evaluating a business item. We claim that the essential conditions for a new technology to be promising for innovation in a company are two-fold: it should be feasible to develop the technology for new products or services in the company—R&D capabilities can be measured with respect to the difficulty, risk, and application domains, the technology could then be developed into (mass) production to create value for the company, and manufacturing capabilities can be evaluated by additional time and cost for manufacturing the new products or offering the new services.

Regarding market attractiveness, we focused on two types of customer-perceived value—“customer-perceived functional value” and “customer-perceived economic value”—and accordingly designed five indexes to measure them. Two indexes for the functional value measured whether a new function offered by new technology is recognized by users and whether it is regarded as important to users. For example, increased airbag functionality through R&D may affect the safety of automobiles. If a new function is easily recognized by users and, moreover, regarded as significant, the perceived functional value will be high. Most users may need the function, but its utility may not be directly recognized by them until they experience an accident. In this case, the value for awareness becomes high, but that for importance is low. On the other hand, the other three indexes for the economic value concerned whether a firm can create profit from the new function by directly pricing it where the user’s intention to pay and the price compared to cost are measured or by indirectly improving brand image. If potential users are willing to pay a high price for the new function, the economic value will be high. For example, a firm may improve its brand image by investing in environmentally friendly technology, but users are unwilling to pay for it. Then, the relevant R&D will have high value for brand awareness but low value for intention to pay and price to cost indexes.

Finally, with respect to business fit, “fit with product plan” and “fit with R&D plan” were used as evaluation criteria, as good R&D projects are aligned with product plans as well as R&D plans. To evaluate the fit with the product plan, we designed to indexes to assess the impacts of new technology on new product or service offerings: one to measure the applicability of the new technology to new products or services that are scheduled to be introduced and the other to measure the impacts of the new technology on them. The new technology will be of value only if it can be used for new products or services in a firm without having any negative impacts—in other words, having only positive impacts on the overall performance of the products or services. To assess the fit with the R&D plan, we adopted three indexes that are commonly used to test the validity of R&D projects—total investment cost required, time and human resources required, and expected return on investment (ROI). A good R&D project uses appropriate cost and human resources with a feasible schedule and a high expected ROI, other things being equal. The evaluation perspectives, criteria, and indexes for R&D project selection suggested in this study are summarized in Table 1. Of course, the indexes must be customized to the context of evaluation.

A five-point Likert scale was used for measurements, with detailed guidelines for assigning the values provided for each index to ensure the convenience and objectivity of evaluation.

Table 2. Evaluation guidelines—an example of a “novelty” index.

Value	Characteristics	Detailed description
5	New-to-the-world product technology	The proposed technology has never been developed anywhere in the world; no relevant patents are found for the proposed technology.
4	New-to-the-world production technology	The proposed technology has been presented in a conference or an exhibition but has not yet been applied to production; there are a few relevant patents but still room for further patent applications on production technologies.
3	New-to-the-country production technology	The proposed technology has been applied to production in other countries but has not yet been applied in the native country; there are a few relevant patents and there is room for further patent applications on applied production technologies.
2	New-to-the-company production technology	The proposed technology has been adopted by competitors in the native country as well as other countries but has not been applied to production in the company.
1	Minor change to existing technology	The proposed technology is simply a minor upgrade to the technology the company already has.

exemplifies the guidelines for the “novelty” index.

4.1.2. Verification of Evaluation Perspectives, Criteria, and Indexes

Then, to ensure the validity of the suggested indexes, we used a panel with four experts—an automobile engineer, an expert in customer-perceived value, an expert in technology valuation, and an expert in R&D planning. They assessed the validity of evaluation perspectives, criteria, and indexes, especially focusing on the significance of indexes, the ease of measurement, and the clarity of guidelines. The validity test was conducted between 11 and 15 December 2013. The test results indicate that most indexes showed a value greater than or equal to 2.0, where the maximum value is 3.0, in the ‘significance’ and ‘ease of measurement’, whereas two indexes, life cycle and functional importance showed the value less than 2.0 in the ‘ease of measurement’. Therefore, we firstly suggested a proxy measure to get a value for the two indexes; patents data are available to estimate a technological life cycle, while customer data are needed to evaluate functional importance of the proposed technology. We also designed a system module to help get a value for the two indexes.

4.1.3. Weights of Evaluation Perspectives, Criteria, and Indexes

The weight values were assigned to the 20 indexes using AHP analysis. The analysis was carried out between 11 and 15 December 2013, and seven experts were involved—four were the same as those who participated in the validity test, and the other three were potential users of this evaluation system, those who are in charge of R&D planning and execution in an automobile industry. The analysis results are summarized in

Table 3. AHP results to determine weights of indexes, criteria, and perspectives.

Perspectives (Weighs)	Criteria (Weights)	Indexes	Weights	Rankings
Technology attractiveness (0.274)	Innovativeness (0.098)	Novelty	0.028	14
		Inventive steps	0.031	13
		Utility	0.039	12
	Appropriability (0.176)	Exclusivity	0.079	5
		Life cycle	0.097	3
Technology fit (0.100)	R&D capabilities (0.029)	R&D difficulties	0.007	19
		R&D risk	0.004	20
		Application domain	0.018	17
	Manufacturing capabilities (0.071)	Additional cost for manufacturing	0.026	15
		Additional time for manufacturing	0.045	10
Market attractiveness (0.400)	Perceived functional value (0.207)	Functional awareness	0.064	6
		Functional importance	0.143	1
	Perceived economic value (0.193)	Brand awareness	0.048	8
		Intention to pay	0.063	7
		Price to cost	0.082	4
Business fit (0.227)	Fit with product plan (0.149)	Applicability	0.046	9
		Impacts on performance	0.103	2
	Fit with R&D plan (0.078)	Investment cost	0.014	18
		ROI	0.022	16
		Time and human resources	0.042	11

. The overall inconsistency value is 0.00, and the responses are consistent and valid.

The analysis results revealed that the most significant three indexes include. “functional importance,” “impacts on performance,” and “life cycle”. As expected, the customer-perceived functional value is regarded as quite important and thus should be evaluated with great concern. At the criteria level, all the top-nine indexes are included in four criteria—“technology appropriability,” “perceived functional value,” “perceived economic value,” and “fit with product plan.” Customer-perceived value is worth considering. Nevertheless, it should be noted that these weights are flexible. Different weight values are needed for different technologies, evaluation purposes, and evaluation contexts. Hence, it is essential to involve potential users of the evaluation system when the weights are determined. Once the weights of indexes are established, the weights of criteria can be calculated by adding the weights of relevant indexes. Similarly, the weights of perspectives can be calculated by adding the weights of relevant criteria.

4.1.4. Pilot Test Results

Finally, a pilot test was conducted to verify the robustness of the evaluation system on 20 December 2013. Three potential users of the evaluation system were involved in the test and assessed two candidate R&D projects, following the guidelines we provided to them. The evaluation results are summarized in

Table 4. The pilot evaluation results for two R&D projects.

Perspectives	Project A					Project B
	Evaluator 1	Evaluator 2	Evaluator 3	Mean	Max-Min	Evaluator 1	Evaluator 2	Evaluator 3	Mean	Max-Min
Technology attractiveness	17	17	18	17.34	1	17	16	16	16.33	1
Technology fit	15	14	12	13.66	3	18	17	17	17.33	1
Market attractiveness	14	10	14	12.68	4	11	14	13	12.67	3
Business fit	17	18	20	18.34	3	22	22	22	22.00	0
Total scores	63	59	64	62.02	5	68	69	68	68.33	1

. The values are generally consistent across different appraisals and thus are reliable. However, compared to the other evaluation perspectives, the evaluation results for market attractiveness that measure the customer-perceived value of new technology are relatively unstable.

To increase the reliability of evaluation, it is recommended that different evaluators value different indexes. For example, external experts rather than internal experts may be more appropriate for assessing technology attractiveness indexes to ensure the objectivity of evaluation results. On the contrary, internal experts are suitable for assessing technology fit and business fit because an accurate evaluation requires concrete information about the firm. In assessing market attractiveness indexes, customers can provide meaningful feedback about R&D projects, particularly the value they expect from the functional areas improved or provided by the projects.

Table 5. Types of different evaluators for different criteria.

Perspectives	Criteria	Indexes	Internal Evaluators		External Evaluators
			Engineering Departments	Other Departments	Technological Experts	Potential Customers
Technology attractiveness	Innovativeness	Novelty	■		□
		Inventive steps	■		□
		Utility	■		□
	Appropriability	Exclusivity			■
		Life cycle			■
Technology fit	R&D capabilities	R&D difficulties	■
		R&D risk	■	□
		Application domain	■	□
	Manufacturing capabilities	Additional cost for manufacturing	■	□
		Additional time for manufacturing	■	□
Market attractiveness	Perceived functional value	Functional awareness				■
		Functional importance				■
	Perceived economic value	Brand awareness		□		■
		Intention to pay				■
		Price to cost		□		■
Business fit	Fit with product plan	Applicability	■	□
		Impacts on performance	■	□
	Fit with R&D plan	Investment cost	■
		ROI	■	□
		Time and human resources	■

■ Main evaluators □ Supporters (e.g., providing references for evaluation).

suggests types of evaluators suitable for each index, categorized by four groups—internal experts from engineering departments, internal experts from other departments, external technological experts, and potential customers.

4.2. Designing a Supporting System for Evaluation

When different types of evaluators are involved in R&D project selection to increase the validity of evaluation, it is advantageous to have a system to support the evaluation. In this study, we designed a prototype R&D project selection system incorporating both customer-perceived value and technology potential. The system will help adjust the weights of evaluation indexes, involve various participants in the evaluation process, and visualize the evaluation results, showing the strong and weak points of individual projects. Therefore, the suggested system consists of five models, as shown in Figure 3.

Module 1 is used to determine the weights of evaluation indexes based on AHP analysis. Thus, it is possible to adjust the weights according to the users and context. By setting the weight value at 0, we can remove the relevant index from the initial evaluation model.

Modules 2 and 3 are used to input data about evaluation results into the system. Module 2 is for data about customer-perceived value (i.e., market attractiveness) indexes. The system enables one to conduct a customer survey to measure “functional awareness,” “functional importance,” “brand awareness,” “intention to pay,” and “price to cost.” Potential customers can access this Web-based system and answer the survey questions that are used to calculate the index values. Figure 4 presents an example of evaluating the functional importance index value for an R&D project.

Module 3 is for data about technological potential (i.e., technological attractiveness, technological fit, and business fit) related indexes. After evaluators provide the value for 20 indexes according to the evaluation guidelines provided by the system, the weighted index values and the total values are calculated as shown in Figure 5.

Modules 4 and 5 are output-related modules, synthesizing the input data for evaluation and showing the evaluation results. Module 4 focuses on the evaluation results for individual project (see Figure 6). The figure on the left side visualizes the average evaluation results for 20 indexes, categorized by four perspectives, in radar charts. The table on the right side summarizes the strong and weak points of the projects and the overall ranking of the R&D project among all the candidate projects.

Module 5 shows the evaluation results for multiple R&D projects. Sometimes, it is necessary to evaluate a portfolio of R&D projects as well as a single project. For example, if all high-ranking projects require huge time and costs in common, it may not be feasible to select projects on the basis of total value; a balance between long-term and short-term projects and high-risk (cost) and low-risk (cost) projects is needed. Using the system, it is possible to develop various portfolio maps by combining two indexes, criteria, or perspectives. Figure 7 presents three portfolios—time and cost, technology attractiveness and technology fit, and market attractiveness and business fit.

4.3. Discussions

To make the evaluation model more valid and reliable, there are several issues to be discussed. First, the evaluation indexes must be updated continuously. This study employs only an initial set of indexes for evaluation, and the final goal should be to develop a customized set by collecting and analyzing actual evaluation data. For example, an ad hoc analysis to compare the evaluation results and actual R&D performance will help identify the indexes that are highly related with R&D performance. Assigning greater weight to these indexes will improve the evaluation model. In addition, it is necessary to reconsider the indexes that show only minor differences in values across different R&D projects. Using these indexes will increase the time and cost of the evaluation without having discrimination power.

Second, the evaluation model needs improvement by developing a method to consider “a set of R&D projects,” that is, an R&D portfolio in the selection process. The current model purposes to evaluate an individual project, but R&D investment decisions are usually made not on a single project but on a set of projects. Balancing and optimizing the project portfolio are significant for R&D project selection. Though the system proposed in this system has a module to analyze an R&D project portfolio, the ultimate goal of the module is to visualize the characteristics of the portfolio from various aspects and thus is not used directly for R&D project selection. The system should be elaborated to help evaluate and further manage a portfolio consisting of various R&D projects.

Finally, the evaluation model can be evolved to include more indexes, enabling to consider other various factors that may affect the value of R&D projects. For example, sustainability and regulations can be one of the most significant drivers of technology development in the automotive industries, which may be conflicting requirements to business efficiency. Recognizing the existence of such conflicting requirements, previous studies have proposed a framework for automotive sustainability assessment [51]. According to Sachs, the following four criteria—ecological, economic, social and political sustainability—need to be met to make a system sustainable [52]. Though the proposed system focused mainly on the two criteria—economic and social sustainability, the other criteria are worth considering in the evaluation. Other evaluation models particularly developed to address the sustainability issues can be used together with the proposed model.

4.4. Implications

Practically, the research findings from the AHP analysis indicate that the “functional importance” is the most significant factor to be taken into account when evaluating technology from the customer’s perspective, followed by “price to cost”, “functional awareness”, “intention to pay”, and “brand awareness”. The managers need to capture the importance of functions that a target technology is expected to offer, the level of intention-to-pay for the functions, the level of recognition of the functions, and the contribution of those functions to improve brand awareness during the R&D evaluation, if they want to choose R&D projects that are attractive to their potential customers.

Theoretically, this study is one of the earliest attempts to conceptualize customer-perceived value in the context of automobile industry. The AHP analysis results indicate that the two elements that constitute the customer-perceived value explain 40% of the total importance of criteria that need to be considered in evaluating technology. Interestingly, the level of importance is almost the same for the two elements (criteria)—functional value and economic value. Unlike the previous models focusing mostly on technological potential, we explicitly incorporated the indexes that can measure customer-perceived value offered by a target technology in the evaluation model. Furthermore, in our model, we proposed the use of different evaluators to obtain values for diverse indexes, as the contribution from a diverse group of people enables to take the combined perspectives of different people, which ensures the credibility and robustness of the process.

5. Conclusions

This study aims to develop a new R&D project selection model incorporating both technological potential and customer-perceived value and apply it to an automobile manufacturer in Korea. While the model was developed in cooperation with an automobile manufacturer in Korea and customized to the automobile industry, it is still a generalizable model that can be applied to other industries that have similar characteristics, “system industry” and “B2C industry.” This is one of the earliest attempts to reflect customer-perceived values in the early stage of technology development, which will help in the pursuit of customer-oriented technology development. As user-driven innovation, which engages users as active participants in innovation activity, is emerging as a successful way of developing new products, services, and concepts, the proposed model will be greatly useful for reflecting potential user needs. Accordingly, theoretically, this study provides a basis for user-centric R&D project selection, unlike most existing approaches of technology-centric R&D project selection. Because user-centric R&D project selection enables one to consider market-related factors quite early in the innovation process, it becomes more feasible to create value in the market through innovation. The model suggested in this study is simple and thus practical; it is possible for any type of firm to adopt it to support decision-making on R&D project selection, which will increase R&D efficiency, although customization is needed. In addition, the system developed in this study will help various evaluators—internal and external experts and customers—participate in the R&D selection process more easily.

Despite these values, this study is subject to a few limitations. First, the evaluation model was developed and tested by potential users in practice but has not yet been applied to a real case. It is essential to introduce the model and system in an automobile company for their R&D project selection process to have external validity. Second, the evaluation model in this study is appropriate for R&D projects in the automobile industry. The identification of sectoral differences in the evaluation models is worth investigating. Third, we adopted a simple survey method to measure customer-perceived value, but more elaborated methods to collect data from customers are needed. Furthermore, while it is significant to describe the function expected to be offered by new technology as an outcome of an R&D project to customers to collect data from them accurately, this is difficult in the early stage of innovation. Moreover, customers may have difficulty understanding the non-existent function, which may cause bias in the evaluation results. Effective description of an R&D project to customers without violating confidentiality is an important issue to be addressed in the future. Finally, only a prototype system was proposed in this study, so a more elaborated system must be developed.