Next Article in Journal
Economic Values and Resource Use
Previous Article in Journal
Synthesis of Household Yard Area Dynamics in the City of San Juan Using Multi-Scalar Social-Ecological Perspectives
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

A Non-Parametric Delphi Approach to Foster Innovation Policy Debate in Spain

1
Department of Development Economics, Universidad Autónoma de Madrid, Madrid 28049, Spain
2
Departamento Economía Aplicada I e Historia e Instituciones Económicas y Filosofía Moral, Facultad de Ciencias Jurídicas y Sociales, Universidad Rey Juan Carlos URJC, Madrid 28933, Spain
*
Author to whom correspondence should be addressed.
Sustainability 2016, 8(5), 487; https://0-doi-org.brum.beds.ac.uk/10.3390/su8050487
Submission received: 28 February 2016 / Revised: 10 May 2016 / Accepted: 11 May 2016 / Published: 18 May 2016
(This article belongs to the Section Economic and Business Aspects of Sustainability)

Abstract

:
The aim of this paper is to identify some changes needed in Spain’s innovation policy to fill the gap between its innovation results and those of other European countries in lieu of sustainable leadership. To do this we apply the Delphi methodology to experts from academia, business, and government. To overcome the shortcomings of traditional descriptive methods, we develop an inferential analysis by following a non-parametric bootstrap method which enables us to identify important changes that should be implemented. Particularly interesting is the support found for improving the interconnections among the relevant agents of the innovation system (instead of focusing exclusively in the provision of knowledge and technological inputs through R and D activities), or the support found for “soft” policy instruments aimed at providing a homogeneous framework to assess the innovation capabilities of firms (e.g., for funding purposes). Attention to potential innovators among small and medium enterprises (SMEs) and traditional industries is particularly encouraged by experts.

1. Conceptual Framework

Innovation is a complex process where agents produce knowledge based on other pieces of disperse and fragmented knowledge. As a consequence, we can reasonably expect that agents will almost never go through this process in isolation, but interact with other agents to acquire and exchange knowledge, information, and other resources [1]. Social and technological changes in past decades induced more complexity on the innovation process, increasing the potential connections among actors [2]. Indeed, it has been argued that innovation increasingly depends on external knowledge sources [3].
The theoretical approaches to assess innovation have reacted to such changes, shifting from a “linear” and autonomous perspective on the production of knowledge [4] to a comprehensive approach where the innovation process is the result of the co-evolution of society and technology, where agents do not innovate in isolation but within a “system”, and where interactions and feedback processes are key to explaining innovation paths [5,6,7,8]. This holistic vision of the innovation process gave rise to new research agendas, such as “open innovation”, in which the discussion is centered at acknowledging the value of using external knowledge [9,10,11]. There has been an increasing interest in the research of empirical evidence relating open innovation and innovation performance [12], but its implications for public policy-making have not yet been analyzed in detail (with few exceptions [13]).
These changes in innovation practice and theory also induced changes in innovation policy [14,15]. For example, they showed the increasing need to manage interfaces between users and producers of innovations beyond the pure transfer of knowledge and technologies (e.g., stimulating demand articulation and bridging gaps between actors). These changes also urged policy-makers to embed innovation policies in a broader socio-economic context (i.e., the need to broaden the policy domain), to be more consistent with the management of a network. Finally, these new approaches allowed for the possibility of legitimizing innovation policy not only as a solution to market failures, but also to tackle system imperfections [16].
Even within this complex environment, innovation is considered a necessary path for growth and increasing competitiveness [17] and, thus, careful design of innovation policy has been a matter of concern for governments and managers that aim for sustainable development through technology changes. On that regard, most innovation-related indicators show that Spain is below the average of OECD countries [18], which suggests that it is not using its scientific and technological potential to adequately generate innovation. This might be seriously damaging the country’s possibilities of overcoming the economic crisis and modifying its production model. In this paper we focus on the analysis of problems and policy instruments of the Spanish innovation system. According to the Innovation Union Scoreboard of the European Commission [19], Spain is below the European Union (EU) average for most indicators on innovation performance, and is considered a “moderate innovator” (among other countries such as Croatia, Cyprus, Czech Republic, Estonia, Greece, Hungary, Italy, Lithuania, Malta, Poland, Portugal, and Slovakia).
The primary objective of this paper is, therefore, to identify the changes needed in Spanish innovation policy, in the belief that innovation should be a strategic component of a new growth model in Spain; a model that should be adopted by all the stakeholders involved in its sustainable development. More specifically, the efficiency of the innovation policy is tightly linked to its four main activities [20]:
(a)
provision of knowledge and technological inputs to the innovation process, which include the provision of R and D and competence-building (e.g., education and training);
(b)
demand-side activities, directed to the formation of new product markets;
(c)
provision of constituents for systems of innovation, such as the creation and change of organizations, innovation, networking, and the creation and change of innovation-related institutions (e.g., regulatory instruments); and
(d)
support services for innovating firms, like financing of innovation, incubation activities (start-ups, entrepreneurship, and small firms), provision of consultancy, etc.
These four types of activities can be used to identify specific problems within the innovation system that might be (or are being) tackled through policy. In this sense, this classification can be used for both diagnosis and evaluation purposes of innovation policy, and sets the theoretical framework through the following general research question:
R1. Taking into account the past and current Spanish situation, which are the main lessons to be learned in order to improve innovation policy efficiency?
The activities of the innovation system might also be affected by policy instruments. Even though they can be very diverse, the most accepted classification of such instruments in the literature [21] group them in three:
(a)
regulatory instruments, like the regulation of intellectual property rights, the regulation of research and higher education organizations; competition (anti-trust) policy regulations concerning R and D, etc.;
(b)
economic and financial instruments, providing pecuniary incentives to innovation activities (primarily through public support to research organizations); and
(c)
soft instruments, which are non-coercive and can be used to complement financial and regulatory instrument (like public–private partnerships, voluntary standardization of information, campaigns and public communication instruments, etc.). This last type of instrument is changing the role of the policy-maker, to become not only a provider and regulator, but also a coordinator and facilitator [22].
There is a recent body of literature that can be linked to the discussion of emerging soft instruments for innovation policy; that is the case of literature on knowledge management and intellectual capital, which deals with the importance of intangible capital of the firm [23,24], and the need for companies to manage and report on this capital [25]. With a few exceptions [26,27] this literature is not sufficiently linked to innovation studies and this is damaging innovation measurements [28] which, in turn, hampers the design of an appropriate innovation policy.
What this line of research tells us is that the most important assets of companies are not bricks and equipment, but human capital, know-how, organizational capabilities, networking capacities, or technological knowledge [29,30]. These are of an intangible nature and are not well reflected, if at all, in firms’ annual accounts [31]. This produces an important asymmetry in capital markets [32] because, despite being valuable firms with strong potential, their accounts reflect less than their real value, so they are then forced to pay more for financing in capital markets than others of less value. Up to now there has been no standard procedure to report on the above-mentioned intangibles and although financial analysts are taking into account the real situation of the companies, particularly the public ones, banks are still reluctant to accept information on intellectual capital which is not audited. This is especially damaging for SMEs, the largest group of companies within the Spanish economy. Therefore, one of our objectives in this study is to fill the gap between the two bodies of literature (innovation policy studies and intellectual capital analysis) by investigating whether encouraging standard and reliable reports on intangibles would be a good innovation “soft” policy instrument. This is stated in our second research question:
R2. Would encouraging a firm’s management and reporting of intellectual capital be an effective innovation policy instrument?
The importance of activities and instruments should be discussed by all the agents involved and taking into consideration the full complexity of the innovation system. One popular classification of relevant agents within the innovation system comes from the Triple Helix approach [33,34,35], where the results of the innovation process are seen as a consequence of the co-evolution of three main sub-sets of social systems: Industry, University, and Government. This approach emphasizes the role of linkages between the industrial, academic, and governmental agents. Of course, the goals of each type of agent might be different (e.g., academic excellence, or popular support), and the evolution of innovation policy might be deemed as a result of policy discussion among the relevant actors of the system (i.e., academics, businessmen, and policy-makers). In this sense, our analysis on this topic will be guided by the following research question:
R3. Are there significant differences among the main players in the Spanish national innovation system with respect to the variables to be tackled and the direction that innovation policy should take?
To address these three research questions, we use a structured method to quantify the discussion of expert groups called the Delphi technique. Information is gathered iteratively through questionnaires where experts have to position themselves towards certain problems related to the activities of the Spanish innovation system and the feasible policy instruments to solve them. Their statistical opinion, and the degree of consensus of the group, will throw light regarding our first research question. The analysis on the opinion of experts towards certain “soft” instruments of innovation policy, related with intellectual capital reporting, will provide information on our second research question. Finally, the analysis of opinion by groups of agents (academics, businessmen and policy-makers) will answer our third question.
The rest of the paper is structured as follows. In Section 2 the non-parametric Delphi method introduced in this study is presented. Section 3 is devoted to presenting and discussing the research results. Section 4 includes the conclusions of the paper and the contribution to addressing the research questions. Section 5 relates to final remarks and future research. Appendix A describes the bootstrap Delphi method in algorithmic form while Appendix B details the results of its application to the case in hand. Appendix C summarizes the analysis by highlighting the most significant identified trends.

2. The Delphi Method

2.1. Design

Delphi is very useful whenever traditional statistical models of forecasting are not appropriate due to the complexity of the problem [36] or in the absence of adequate sets of data (historic, economic, or technical), which is when expert judgment is necessary [37].
As is well known, the Delphi method is used to help in the decision-making process whenever an expert opinion is gathered, mostly through the use of questionnaires with propositions [38]. Its key characteristics are:
(1)
Iteration, by having the experts answer the questionnaire in at least two rounds.
(2)
Controlled feedback, by providing the experts with information about the answers in-between rounds (for example, averages, modes, medians, etc.).
(3)
Anonymity, by not including information that might bias the results.
(4)
Group opinion, by aggregating the answers of the individual experts, which might be categorized according to a critical variable.
The selection of experts [38,39] and the development of the questionnaire are, therefore, key issues in Delphi analysis. In this case we followed a snowball process to select the experts and develop the questionnaire. In the first place four experts from the two sponsoring institutions were called in to define the scope of the analysis and the precise topics the propositions should explore. An initial set of topics was defined and discussed through in-depth interviews with five additional experts from different backgrounds and positions. As a result of the discussions and having in mind the above-mentioned conceptual framework, a preliminary questionnaire addressing the topics was tested with all these experts and a further 10 who joined them (names and positions available at [40]). These 19 experts provided the names of those who would be participating in the Delphi and then went on to review and refine the whole list, which resulted in a total of 315 people (40% in business, 39% in academia, and 21% in public administration). From these, a total of 82 experts were involved in the two rounds (38% from firms (29 experts), 41% from academia (37 experts), and 21% high officials in public administration (16 experts)).
The questionnaire addresses the following final list of issues, which are all relevant within the conceptual framework, with the number of propositions included within brackets:
(1)
Investments in physical capital and in education (13)
(2)
Information society (14)
(3)
R and D investments (9)
(4)
Disruptive innovation/Entrepreneurship (9)
(5)
R and D and innovation funding (9)
(6)
R and D and innovation fiscal incentives (4)
(7)
Social value of innovation (8)
(8)
Cooperation for R and D and innovation between academia and industry (10)
Even though these proposed topics were obtained from the discussion with the experts, each problem or policy identified in the propositions of the questionnaire can be related to the theoretical framework based on activities of the innovation system [20]. Taking a broad perspective, Investments in physical capital and in education, and R and D investments can be related with the activities for the provision of knowledge and technological inputs to the innovation process, since education and R and D are the main sources of competence building and new knowledge creation. Demand-side activities are studied through topics such as promotion (demand-side) of Information society and Social value of innovation. Although Information society might be also viewed as a promoter of networking for innovation, our focus in the questionnaire was on policies directed to increase the diffusion rate of innovation and communication technologies. That is why it is related here with demand-side activities (and policies). Cooperation for innovation and Disruptive innovation/Entrepreneurship are related with the provision of constituents for systems of innovation capturing, respectively, the links between agents (collaboration agreements, industry-university research, etc.) and the generation of variety within the spectrum of innovation plans. Finally, R and D Funding and R and D incentives for innovation activities can be clearly identified as financial support services for innovating firms. Of course, the list of topics included in the questionnaire do not pretend to be an exhaustive list of problems and policies related with these activities, but specific cases that can contribute to discussion with a systemic view of innovation process.
In addition to the theoretical frameworks, three criteria were taken into consideration when finally including propositions dealing with specific issues. First, the existence of data for Spain, comparable with those of other OECD countries (for example, Education investments); second, the use of the variable in the scientific literature (for example, R and D firm expenditures), and third, the interaction of at least two of the three helices (for example, Fiscal incentives are designed by the government and affect firms; R and D cooperation needs both industrial and academic involvement).
Taking into account the path dependent nature of technology and innovation [41], the experts were provided with a set of graphs and figures, mostly based on OECD data, which reflected the Spanish situation in comparison with other countries.
The result of the design is a questionnaire of 70 propositions that had to be scored on a five-point Likert scale, with score 1 representing strong disagreement with a given proposition and score 5, strong agreement:
(1)
Strong disagreement
(2)
Weak disagreement
(3)
Controversy
(4)
Weak agreement
(5)
Strong agreement
Each expert gives a score on a Likert scale for each proposition in the questionnaire in each round. Experts are also asked to provide additional comments to all propositions. This is particularly important when, in the second (and possibly subsequent) round, the expert disagrees with the group opinion. Very often the explanation of discrepancies is an interesting source of knowledge [36,42].
We have tested the validity of the questionnaire, demonstrating that the results of the survey are jointly reliable (Cronbach’s alpha coefficient = 0.8818). We also have assessed how individual items are related to the overall scale, and all individual coefficients of Cronbach’s alpha are above 0.80.
As for the traditional analysis of the results of the Delphi experiment, the following descriptive statistics are readily calculated:
(1)
Average score per round, group and proposition, to assess opinion.
(2)
Variance of the score per round, group and proposition, to determine consensus in opinion.
(3)
Difference in the variance of the score in-between rounds or groups per proposition, to evaluate persistence in consensus.
(4)
Difference in average score in-between rounds or groups per proposition, to quantify the differences of opinion.
Simply graphing the results or comparing them against thresholds is usually enough to derive useful information. In the case at hand, and due to the importance and novelty of the topics, a more robust inference analysis is sought.

2.2. Non-Parametric Bootstrap Analysis

The main reasons for being unable to carry out a sound and robust inferential analysis of a Delphi experiment are usually the small size of the sample and the Likert scale in which the propositions are quantified. In these circumstances, the use of traditional parametric tools is questionable and other techniques have proved more reliable. Resampling methods are a good alternative [43]. Among them, the non-parametric bootstrap [44,45] should be highlighted, not only due to its widespread use but also because it is used to analyze the results of Delphi experiments, although as mentioned before, only in medical studies [46,47,48]. However, it was only a matter of time before the bootstrap would be used in other areas, like technology and innovation, as we do in this paper. To the best of our knowledge, the non-parametric bootstrap Delphi is methodologically applied for the first time in the field of technology and innovation policy forecasting, providing a sound basis for a thorough discussion on innovation policy issues.
The idea behind bootstrap is both simple and statistically robust. The aim of bootstrap is to generate a lot of data by resampling from the original sample. This data intensive strategy has proven reliable in estimating statistics like averages and their confidence intervals [49], quantiles [50], or even p-values [51]. Non-parametric bootstrap is also appropriate when analyzing the results based on a Likert scale and different sample sizes (for example, smaller than 100 [52], larger than 1000 [53]).
Methodologically, bootstrap works as follows. Out of the original sample of I values, a new sample, called the bootstrap sample is obtained. Each original value is assigned a probability of 1/I, and I samples are obtained by sampling with replacements. The procedure might be repeated r = 1, …, R times to obtain R bootstrap samples. Each time, sample statistics might be calculated, arriving at their sampling distributions.
Then, to compare the sample statistic with the underlying sampling distribution, resampling methods use the frequency distribution of the statistics obtained from the newly generated samples instead of using a theoretical parametric distribution. The p-values might then be calculated by just using relative frequencies corresponding to the null hypothesis or the predefined thresholds. As usual, p-values lower than the significance level call for the rejection of the null hypothesis that is set to evaluate a predefined research question. The non-parametric bootstrap method used in this paper is mathematically specified in Appendix A.
Therefore, by resampling the experts’ scores which were given to the statements proposed in the questionnaire, a discussion on policy issues can be fostered based on the following assumptions, as in usual Delphi analysis:
(1)
Propositions with extreme scores (strong agreement or disagreement) and higher levels of consensus provide stronger messages on policy directions which might either encourage innovation or hamper it, than propositions with lower levels of agreement or consensus.
(2)
Propositions with higher levels of agreement among the three groups (academics, entrepreneurs, and policy makers) indicate policy directions that are more well-defined than those where there are greater differences among groups.
The initial descriptive results of the two-round Delphi experiment showed the huge potential for discussion of innovation policy topics but were not considered robust enough to define policy directions and it was then that the resampling method was developed.
As mentioned before, 82 experts from three sectors participated: 37 from academia, 29 from the business sector, and 16 from public administration. They gave scores for each of the 70 propositions.

2.3. Aggregate Analysis of Opinions

The descriptive results of the Delphi can be seen in [40]. We will concentrate here on the more robust results drawn from the inferential analysis. The latter do not contradict the former, but allow us to see, for example, whether a strong agreement is statistically significant.
The paragraphs below show the results obtained when individually testing each proposition looking for statistical significance in which to base the discussion on innovation in Spain:
  • Test 1. For a proposition to be considered as policy-prone the expected score is 4 or higher, or 2 or lower.
  • Test 2. For a proposition to have demonstrated solid opinions, the variance of the scores between rounds decreases.
  • Test 3. For a proposition where expert groups differ, the pair-wise difference in the expected score is other than 0.
The results of the bootstrap analysis by proposition are detailed in Appendix B. Appendix C summarizes them and helps highlight the significant propositions in terms of extreme scores, higher consensus, and differences among expert groups. The tables included in this section summarize the results by topic.

2.3.1. Test 1: Looking for Extreme Average Scores

Initially we focus on those propositions with high or low average scores. In the case of strong agreements, the null hypothesis is that the true expected score for the proposition is 4 (Ho: μj = 4), and is tested against the alternative hypothesis H1: μj > 4. Accordingly, in the case of strong disagreement, the null hypothesis is Ho: μj = 2, and the alternative is H1: μj < 2.
More than half of the 70 propositions show statistical significance for strong agreement or disagreement at 95% confidence (see Table 1 for a summary of the detailed results provided in Appendix C). In addition, we have carried out an additional test for controversial results, to see the number of propositions for which the average score is 3 (i.e., H0: μj = μ0j = 3, H1: μj ≠ 3), which turns out to be 6 (8.6% of the 70 propositions). Thus, the number of propositions with weak agreement or disagreement is 26 (37.1%).
Some controversial topics appear for the following issues: Information society, Disruptive innovation, and Social value of innovation. Strong agreement (or disagreement) is the typical stance in most of the topics, except for Information society and R and D and innovation funding, where most of the propositions receive weak agreement (or disagreement). It should provide a concise and precise description of the experimental results, their interpretation as well as the experimental conclusions that can be drawn.

2.3.2. Test 2: Looking for Solid Opinions

An expected outcome of Delphi is an increase in consensus through rounds. Such an increase could be interpreted as a more solid opinion and would therefore provide a more solid basis for identifying policy directions. The test we have used in this case is based on the differences in the variances of the scores through rounds:
(H0: σ2j2 − σ2j1 = 0; H1: σ2j2 − σ2j1 ≠ 0)
The results show that, with 95% confidence, consensus increases in 60% of the propositions and only decreases in 11.4% of them (Table 2). Increasing consensus dominates in all topics, and decreasing consensus is present only in three of them. The presence of stability in consensus is particularly important in Information society and Disruptive innovation. It is interesting to note that in the case of Disruptive innovation, the dominance of a non-increasing consensus is in line with the low proportion of strong agreements or disagreements (see Table 1).

2.3.3. Test 3: Looking for Inter-Group Differences

It is important to study differences in opinion among institutional agents (i.e., researchers (r), businessmen (b) and policy-makers (p)). First, the results of the second round for all of the groups are simultaneously compared:
(H0: μ = μr2 = μb2 = μp2)
Then, the average score from each group of experts is individually compared against the average score of the whole sample:
(H0: μg2 = μ2, for g = r,b,p)
Table 3 summarizes these results by topic (since we are comparing the opinion of groups only in the second round, subscripts indicating the round are omitted to simplify notation).
The fact that the proportions of significant differences are “low” shows homogeneity of opinion. One interesting aspect is that businessmen appear to be more favorable to the propositions than the other two groups, since in 18.6% of propositions they were significantly above the average score of the whole sample. On the other hand, researchers look more cautious than the other groups, since it is the group that scores below average in more propositions (7.1%).
There are no significant differences in opinion among groups in the topics R and D investment and R and D and innovation fiscal incentives, but there seems to be more discrepancies in topics like Investment in physical capital and education and Information society.

3. Results of the Experiment

Particularly interesting are the findings detailed below. They have been grouped first according to the list of issues given to the experts. A final subsection is included to summarize the results.

3.1. Investments in Physical Capital and Education

The propositions of this topic can be related to the activities of provision of knowledge and technological inputs to the innovation process. Although experts identify a problem of the innovation system regarding these activities, there is less consensus surrounding the specific instrument to improve performance.
We raised two propositions related to investment in equipment. As could be expected, the experts as a whole are not happy either with the amount of current investment or the choice of industries which have received investment in the past. Although agreement is weak, the consensus increased during the process and no discrepancies appear among the three groups. We must say, however, that policy-makers (partly responsible for such investments) are more comfortable with the current status of physical capital than the other two groups.
Notwithstanding the importance of physical capital, we were particularly interested in the part of the questionnaire which referred to the main changes needed in education. The agreement in favor of increasing investment in education is strong and stable. Consensus increases between rounds when agreeing that there is room for improvement.
The three groups of experts agree on the need to develop competences for innovation in both secondary and tertiary education, while only businessmen think that this is also a must in primary education.
To develop a culture of effort and raise social awareness of teachers’ value gains strong agreement from all groups with an increase of consensus throughout the process. Already in the first round all the experts agreed there was a clear need to pay more attention to vocational training.
Some controversy arises, however, on how suitable it is to foster institutions of excellence at the different education levels. Agreement is strong when talking about the tertiary education level but consensus decreases in the second round and there is only weak agreement on secondary schooling.

3.2. Information Society

The propositions in this set were looking for areas where public incentive would be most useful in fostering the spread of the information society as a framework condition for innovation. In general, these propositions intended to capture demand-side policy to promote the use of information technologies.
Large Spanish companies compare very well with others at European level, while the opposite is true of Spanish SMEs and households. Weak agreement is found in all the groups for supporting investment in the information society with subsidies. As could be expected the businessmen are more in favor of supporting companies, whatever size, than the other experts.
There is no clear support either in favor of the other incentives suggested (supporting equipment acquisitions, reducing taxes for on-line buying, etc.). In some of these direct support incentives there are clear discrepancies among the three expert groups.
In contrast, there is a very wide support for improving the teaching skills needed to introduce the information society at primary level in schools. Again, education is considered a key issue.
With respect to e-government, one of the main indicators for measuring the scope of the information society in a particular economy [54], the Spanish tax agency has been a benchmark in using on-line procedures to communicate with taxpayers and allowing on-line presentations and tax payments. Strong support, already shown in the first round, is given to extending this example into other areas of Government (health-care, education, etc.).

3.3. R and D Efforts

This topic is one of the most relevant issues for innovation policy, as it is clearly directed to increasing the provision of knowledge inputs to the innovation process.
There is weak agreement in favor of increasing public R and D. However, it becomes stronger when referring to an increase in the private sector. Reflected here are the experts’ concerns about R and D expenses and their sectorial distribution in Spain. Not only is the ratio R and D/GDP smaller in Spain than the OECD average (1.35% against 2.34% in 2008) but also most of the R and D spending rests on the Government’s shoulders. Industry spends 0.74% in relation to GDP, while in the OECD this figure is 1.63% [55].
There is also strong agreement about the need to evaluate and follow up public R and D policy, although, as could be expected, policy-makers are less inclined to support this opinion.
The particular comments added to this set of propositions highlight the need to take into account SMEs when designing new R and D policies. A significant increase in the number of Spanish companies undertaking R and D efforts should necessarily come from increasing SMEs’ share.

3.4. Disruptive Innovation/Entrepreneurship

With regard to constituents, once again, experts show their preference for improving framework conditions (such as reducing red tape) to encourage disruptive innovation rather than using direct subsidies.
Their opinions, however, differ on whether there should be a size threshold to be a “disruptive innovator”. Businessmen clearly reject this suggestion and there is agreement, albeit weak, on the possibilities of innovation in traditional industries and on the existence of “gazelle” entrepreneurs in these areas. Businessmen are particularly convinced of it.
The importance of traditional industries and the ways of improving their innovation capacity through the use of high technology is mentioned by many experts in their additional comments. A provocative proposition (number 41) suggesting that business success derived from innovation occurs mainly in high-tech industries raises an interesting degree of controversy. Businessmen particularly disagree with this assumption, even those whose business is actually high-tech.
On the whole, the experts draw attention to SMEs and traditional industries, highlighting how important they may be in improving innovation performance in Spain. It must be noted that these two groups have been very much ignored by the Spanish innovation policy up to now.

3.5. R and D and Innovation Funding

The lack of financial support services is a clear barrier for R&D and innovation in Spain [56]. All the experts agree, with nuances, on the idea that the increase in funding would be more efficient if made by companies rather than by government.
As mentioned before, one of our objectives was to explore the possible benefits of reporting intellectual capital in order to increase innovation funding and thus to have it encouraged through policy mechanisms. Accordingly, this part of the questionnaire was a long and important one where the experts were particularly asked to give opinions on the propositions related to this issue.
Six propositions confronted them with the fact that annual company accounts do not clearly show their intangible investments and assets (like R and D and other innovation efforts) and that this creates a serious barrier to obtaining external funds, especially for knowledge-intensive small firms [57]. The experts were asked their opinions in relation to taking into account the potential of those investments by having reliable (e.g., verified by external experts) additional information on them.
A current characteristic of the Spanish innovation policy makes these questions particularly interesting. Not only are banks reluctant to provide loans when they are not backed up by physical assets, but public sector institutions, the loan providers of R and D and innovation activity funding, require bank guarantees whenever the main assets are intangibles. Potentially innovative companies, intensive in intangibles, are thus trapped in a vicious circle when needing funds.
There is no controversy around this issue. Regardless of any small discrepancy which may stem from the novelty of connecting intangibles reporting with innovation funding, all the experts agree on the need to change the current status, in order to accurately measure and value the intangibles. By achieving this, companies will be in a better position when demanding innovation funding. This is an interesting result since, to the best of our knowledge, this topic -the relationship between reporting on intangibles and better performance in capital markets- has not been properly linked to innovation policy analysis in the literature.

3.6. R and D and Innovation Fiscal Incentives

Spain is the OECD country with the largest tax credits for R and D [58]. With this in mind, the fact that it lags behind the OECD average in R and D and innovation investments may be considered a proxy for the inefficiency of government fiscal sacrifice.
Looking for explanations to such poor results from the fiscal incentives and support services, especially in SMEs, experts agreed clearly on: (a) tax incentives are distributed badly and are not well-known to company managers; and (b) fiscal norms should be drafted more clearly and have continuity to be more effective. The consensus increases around these ideas in the three groups of experts.

3.7. Social Value of Innovation

Through a yearly survey exploring public interest (demand-side) in science and technology produced by the Spanish Foundation for Science and Technology [59], it can be seen that even among the well-educated population, scientific, and technological issues attract little interest. As could be expected, this is considered very damaging to innovation by the experts.
They coincide on the need for companies to report more on the beneficial consequences of their innovation efforts, the importance of scientists’ publicizing their work, the need for reporting on scientific and technological developments and the value of innovative managers for society. These are all necessary conditions for growth and welfare.

3.8. University-Industry Cooperation for R and D and Innovation

The experts strongly agree on the need for companies to play a more important role in university governance and on the need for universities to be more receptive to ideas from business (constituents). They are also very clearly in favor of rewarding researchers fairly for improving business success as a result of collaborative projects.

3.9. Summary for Spain

It is interesting to emphasize two specific recommendation that emerged from the analysis of the topics and that summarize the results, as specified by the experts concerning the situation of Spain:
(a)
Framework conditions might be more effective than subsidies or direct support for encouraging innovation; and
(b)
SMEs and traditional industries must be drawn into research and innovation policy.
(a) Framework conditions might be more effective than subsidies or direct support for encouraging innovation. This argument, clearly backed up in some literature [60], is particularly clear in several issues such as education policy, e-government, and private firms’ involvement in innovation.
The strong support for continuing to increase investment in education is in line with OECD recommendations, but this is not all that matters. Spain is showing bad results according to the Pisa report [61] which calls for developing a culture of effort and a greater social awareness of teachers’ worth [62]. Developing competences for the introduction of innovation in both secondary and tertiary education is to be encouraged, however, among the three groups, only businessmen think that this should also apply to primary education. In this regard, they are closer to recent postulates that suggest that cognitive capabilities and stimulating intellectual curiosity, which serve as a lifelong motivator for continued learning, take place at primary, and even at pre-primary, school level [62]. The importance of fostering institutions of excellence at the different education levels is controversial among the experts, particularly when talking about the secondary level. It is an issue that also causes controversy in the Pisa report [61], and deserves closer future attention. On the whole, experts are considering the education institutions in a holistic way, which is how any system of innovation should be analyzed [63].
Framework conditions are also highly relevant when spreading the Information Society. Actions where the government leads by example (using on-line procedures in its dealings with the public) would have a stronger effect than direct subsidies for specific information technology use. E-government is considered one of the main indicators for measuring the scope of the information society in any economy [54].
Moreover, framework conditions should be changed so as to allow companies to really participate in the knowledge economy. Reducing red tape would facilitate their engagement in innovation activities, while direct measures, such as tax credits to R and D and innovation expenses, should be well-diffused and stable over time. Finally, companies should have a real say in teaching and research in higher education institutions bringing the governance model closer to the entrepreneurial and managerial system [64,65].
Framework conditions are especially important in critical periods when deficits must be reduced because, very often, they do not require additional or significant public investment [56]. This is the case of many of the actions supported by our experts; for example, the general use of e-government will reduce the cost of public services in the medium term.
However, direct subsidies continue to be in use and are particularly encouraged for SMEs in some cases. This can be seen, for example, in some Baltic countries [66,67] or in the OECD recommendations to Serbia [68,69]. Nevertheless, when looking at these recommendations in detail, the SMEs implicitly or explicitly considered are those which are already conscious of R and D relevance and are involved in these activities to some extent.
Regarding the framework conditions, our results are also consistent with some recent work on the implications of open innovation practices for innovation policy. It has been argued [13] that policies for open innovation should not target only firms’ behaviors, but also need to secure the key external conditions of open innovation. These conditions can be classified in three topics: (1) strong public knowledge base; (2) mobile and educated labor force; and (3) good access to finance. The strong support of our experts to policies affecting education and public expenditure on R and D activities, points also to the need for a stronger public knowledge base and a more educated labor force. The same can be said about the consensus towards improving the access of SMEs to finance. We suggested before that these consensus is aligned with the need to improve Spanish innovation system, but we can also state that this support might be also connected with the need for a more “open” system [70].
(b) SMEs and traditional industries must be drawn into research and innovation policy. The experts identify niches in those two groups (e.g., efficient use of water in the energy industries; tourist industry and health; the development of logistics centers in transport and communication) where support may be particularly effective.
SMEs and traditional industries have not been targeted in past or present innovation policy “soft” instruments and this has been an oversight recognized by even current policy makers. When SMEs are targeted as happens in the case of tax credits, the actual impact of the subsidy is small. This may be due to the fact that government institutions are not sufficiently aware of how these firms work [71], and that most Spanish SMEs are potential innovators or non-innovators and do not respond to these types of instruments. The European Commission is very clear about this: When dealing with “SMEs (that) do not practice any R and D activities, it is in general not useful to have an R and D policy for them that consists purely of financial support” [72] but the policy should call for a proactive attitude to address their specificities.
The lack of conclusive results in the literature with respect to the relationship between company size and innovation capacity is also reflected in the experts’ opinions. Their replies denote disagreement when referring to the need for a size threshold for developing disruptive innovation. The businessmen clearly reject any such need which is in line with the idea that SMEs are actually more agile and responsive to change [73]. Markides [74] develops Christensen’s disruptive innovation concept [75] and suggests that the early pioneers that create “new-to-the-world” products are rarely the ones that scale them from little to big markets. They are usually small, even start-up firms, which nurture the network large firms need to extend their business.
Examples showing how business-model disruptive innovation may happen in traditional industries are found in the literature. This is the case of Amazon or Swatch [76].
Thus, the suggestion is that when targeting SMEs to increase innovation, their size is not the most important consideration but their role within the Clarysse and Duchêne classification [77] and that particular attention should be paid to the large group of potential innovators.

4. Conclusions

R1. Taking into account the past and current Spanish situation, which are the main lessons to be learned in order to improve innovation policy efficiency?
Regarding our first question, we follow the framework based on activities [20] to discuss our policy recommendations in the light of their relation with one of the four types of activities within an innovation system.
Regarding the provision of knowledge and technological inputs to the innovation process, the main lessons or policy recommendations are:
  • increasing R and D activities, but primarily in private sector;
  • increasing investment in education;
  • developing innovative competences in both secondary and tertiary education, but experts have a weaker agreement towards extending this promotion to primary education;
  • fostering institutions of excellence at the different education levels is controversial among the experts, particularly when talking of the secondary level; and
  • increasing investment in technological equipment is more controversial.
The main focus of the questionnaire regarding demand-side activities fostering the use of information technologies in firms and households. The main results are:
  • weak agreement towards establishing incentives for SMEs and households to invest more in the information society, but no agreement on extending this policy to large firms; and
  • no agreement towards fiscal incentives (such as VAT or income tax)
From the point of view of the provision of constituents for systems of innovation, there are interesting lessons regarding entrepreneurship and networking. The main results are:
  • elimination of bureaucratic obstacles to business creation can have deeper effects on innovation than subsidies;
  • encourage processes that foster “open innovation” (i.e., collaboration with external partners to develop innovation);
  • increase the effective participation of firms in the governing bodies of universities; and
  • research results from industry-university collaboration should be recognized in the evaluation of researchers
The main topics in our study regarding the support services for innovating firms are the financing of innovation, and “incubation” activities. Our main lessons are:
  • public sector should not be the most important source of funding for R and D activities;
  • there is no clear diffusion of existing tax incentives for innovation among firms;
  • entrepreneurship is primarily financed with own funds, given the restricted access to external funding;
  • weak agreement towards specific incentives for spin-offs;
  • risk analysis of financial institutions should go beyond the financial statements of firms, to consider variables that capture the capacity to generate profits derived from the innovative potential of the applicants credit.
R2. Would encouraging a firm’s management and reporting of intellectual capital be an effective innovation policy instrument?
Regarding our second research question, we find support for the idea that R and D and innovation funding may be increased if intangible assets and competencies are duly measured and reported. This would require standardized and reliable intangibles valuation procedures.
The need for adequate funding to improve innovation capabilities [4] is clearly acknowledged by the experts. They also state that funding for innovation is hampered by the characteristics of the current accounting model which fails to show the company’s intangibles and intellectual capital in the annual accounts [29].
Some recent international initiatives [77,78] are insisting on the need to have standards for valuation and reporting on intangibles and the need to follow the guidelines suggested in the literature [22]. The strongest recommendation is to produce a statement on intellectual capital which supplements the financial information provided by the annual accounts [38]. The experts support both standardization of reports and external validation.
Should companies produce this type of report, the financial markets would have access to most relevant information which would, in turn, decrease information asymmetries and reduce capital costs [32]. Many of these incentives do not need to be costly for the public sector [53].
Being able to rely on information about R and D and innovation directly provided by companies instead of capturing it through surveys (for example, the Community Innovation Survey [79]) as is currently done, would improve the knowledge available to public authorities and allow a much more accurate policy design [28]. However, the precise mechanisms to be designed and the way to incorporate them into actual innovation policy are yet to be tackled in future research projects.
R3. Are there significant differences among the main players in the Spanish National Innovation system with respect to the variables to be tackled and the direction that innovation policy should take?
Regarding our third research question, it is interesting to note that, with few exceptions, there is big consensus in the recommendations made by the three groups (researchers, businessmen, and policy-makers), which means that they are talking “the same language” and that the research and innovation policy agenda can be developed in an interactive way. This is considered an advantageous situation by Leydesdorff and Etzkowitz who drew attention to it when they first wrote about the Triple Helix framework [33].

5. Final Remarks

Our intention has been to identify the main trends and changes needed in Spanish innovation policies to catch up with other better performing economies in this respect. Given the relevant role that innovation plays in growth and competitiveness, a shift in policies in the right direction should help to overcome the current economic crisis.
Having in mind the complexity of the system being dealt with, the multiplicity of agents involved and the path-dependency that characterizes the system, a Delphi methodology was found to be the most appropriate to deal with our research questions. In order to have inferential analysis and arrive at more robust conclusions we have applied for the first time to our knowledge a non-parametric bootstrap method to analyze the results of a Delphi exercise concerning innovation.
Our research has contributed to the literature in several ways. We have identified necessary changes in innovation policy to be addressed in Spain (R1), such as focusing attention on SMEs’ potential innovators and traditional industries and on framework conditions through open innovation. High levels of consensus have been reached among the three main players of the innovation system (R3). Particularly interesting is the need for companies to manage and report their intangibles as a way to improve the likelihood of innovation funding (R2). This issue has not received enough attention in the literature and deserves further research. A promising line would be to test the hypothesis that firms that manage their intellectual capital are more productive than those that do not, making them more attractive to potential investors.
The main limitations are related to the scope of variables and issues which have an influence on innovation and may be affected by policy action. Choices had to be made to arrive at questionnaires of reasonable length and because of this, important elements of the national innovation system have not been completely addressed, maybe hampering its full validity, but clearly establishing a promising line of research. This is the case for instance of interface institutions which favor knowledge exchange between academia and industry or the research and innovation priority setting mechanisms in the public sector. We also have to put a word of caution on the applicability of the results since they are related to Spain, which also has forced for the sample to be somewhat small. Finally, the paper shows that it is possible to improve the Delphi methodology and, thus, allow inferential analysis to increase the reliability of the results.

Acknowledgments

This research project was developed in the Autonomous University of Madrid (UAM)-Accenture Chair in Economics and Management of Innovation. We want to acknowledge the sponsorship of both Accenture and AETIC (Asociación de Empresas de Electrónica, Tecnologías de la Información y Telecomunicaciones de España) for its development.

Author Contributions

The authors contributed equally to the work, although each author was responsible for one part of the research. Juan Carlos Salazar-Elena carried the questionnaires and developed the bootstrap routine. M. Paloma Sánchez designed the conceptual framework and the questionnaire. F. Javier Otamendi was responsible for the statistical analysis and structuring the paper.

Conflicts of Interest

The authors declare no conflict of interest. The founding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, and in the decision to publish the results.

Abbreviations

The following abbreviations are used in this manuscript:
OECDOrganisation for Economic Co-operation and Development
SMESmall and Medium Enterprises

Appendix A: The Non-Parametric Delphi Algorithm

Step 1: Setting the hypothesis to select propositions

  • Hypothesis Test 1: Looking for extreme average scores
For strong agreement, the null hypothesis is that the expected score for the true proposition, µ, is 4, with an alternative expected score greater than 4 for each proposition j:
H0
µj = µ0j = 4
H1
µj > 4
Accordingly for the case of strong disagreement:
H0
µj = µ0j = 2
H1
µj < 2
  • Hypothesis Test 2: Looking for solid opinions
The test is based on the comparison of the variances in the scores between rounds k, s2jk, to estimate the difference in population variances, σ2jk. Differences greater than 0 are considered significant.
The hypothesis and statistics in this case are:
H0
σ2j2 − σ2j1 = 0
H1
σ2j2 − σ2j1 ≠ 0
  • Hypothesis Test 3: Looking for inter-group differences
The average scores from the three groups (µg2, for g = r, b, p) are compared, simultaneously, with the total expected score (µ2):
H0
µ2 = µr2 = µb2 = µp2
H1
Not all the same
For individual comparisons with the total average score, the hypotheses are:
H0
µg2 = µ2
H1
µg2 ≠ µ2 for g = r, b, p.

Step 2: The expert sample

For each proposition j = 1,…, J of the questionnaire and in each round k = 1, …, K, the expert I = 1, …, I provides an answer with a score on a Likert scale in a five-point scale: xijk = {1, 2, 3, 4, 5}.
The average score per group of experts is computed as the main output indicator [63]:
x ¯ j k = 1 I i = 1 I x i j k j , k
The variance of the score is also calculated to understand the degree of consensus among the experts:
s j k 2 = 1 I i = 1 I ( x i j k x ¯ j k ) 2 j , k
The average score, as well as the variance, is also calculated for each group g = 1 ,…, G, or category of experts:
g x ¯ j k , g s 2 j k    g , j , k

Step 3: Bootstrap sample

Out of the original sample of I values, a new sample, called bootstrap sample is obtained. Each original value is assigned a probability of 1/I, and I samples are obtained by sampling with a replacement/replacements? The procedure might be repeated r = 1…R, times to obtain R bootstrap samples. In our case, given the low variability of our statistics (average and variance), R = 999 samples are enough to obtain stable results.
The average and the variance are calculated for each bootstrap sample r:
x ¯ r j k = 1 I r = 1 R x r j k    j , k
s j k 2 r = 1 I r = 1 R ( x r j k x r ¯ j k ) 2    j , k
The average score, as well as the variance, is also calculated for each group g = 1 ,…, G, or category of experts:
g x ¯ r j k , g s 2 j k r    g , j , k

Step 4: Standardized Statistics and Significance Rule

The standardized statistic of the sample using the sampling distribution is represented by tj, for each proposition j. The standardized statistic using the sampling distribution is represented by Tjr, for each proposition j. Both will be specified according to the null hypothesis. If the bootstrap statistic is more extreme than the sample statistic in at least 95% of the bootstrap samples, then the null hypothesis is rejected.
  • Hypothesis Test 1: Looking for extreme average scores
    t j = x j ¯ μ 0 j s j I 1
    T j r = x j r ¯ x j ¯ s j I 1
  • Hypothesis Test 2: Looking for solid opinions
    t j = ln ( s j 2 2 ) ln ( s j 1 2 ) V a r ( ln ( s j 1 2 ) ) + V a r ( ln ( s j 2 2 ) )
    T j r = [ ln ( s j 2 2 r ) ln ( s j 1 2 r ) ] [ ln ( s j 2 2 ) ln ( s j 1 2 ) ] V a r ( ln ( s j 1 2 ) ) + V a r ( ln ( s j 2 2 ) )
  • Hypothesis Test 3: Looking for inter-group differences
    g t j = g x ¯ j x j ¯ g s j I g 1 ,  for   g = r , b , p
    g T j r = g x j r ¯ g x j ¯ g s j I g 1 ,   for   g = r , b , p
    F j = I g n 1 g = 1 3 ( g x ¯ j x j ¯ ) 2 I g 1 n g = 1 3 g s j
    F j r = I g n 1 g = 1 3 ( g x ¯ j r g x j ¯ ) 2 I g 1 n g = 1 3 g s j

Step 5: The algorithm

The algorithm for the non-parametric bootstrap is further specified in pseudocode format:
  • Define the null H0 and alternative hypothesis H1.
  • Calculate the statistic tj out of the observed sample and H0.
    • For each bootstrap sample
      • Sample with replacement after assigning a probability of 1/I to each value of the observed sample
      • Calculate the statistic Tjr (or, alternatively Fjr) out of the bootstrap sample, assuming that the null hypothesis is true (e.g., ensuring it is centered on zero).
      • Calculate the indicator by comparing the bootstrap statistic and the observed threshold: Ir(•). The indicator will be equal to one if the condition within the parenthesis is verified and 1 otherwise [10]: I r ( )={ 1 ,if { T j r t j ,if   t j > 0 T j r t j ,if   t j <0 0 otherwise
    • Repeat the process R times.
    • Calculate p-value: p= 1 R r=1 R I r ( )
  • Assign significance to the p-value (The p-value is defined as the probability of obtaining a statistic at least as extreme as the observed value, given that the null hypothesis is true. In quantitative terms, the p-value is the probability of being inside the interval [tj,∞) if tj > 0, or the probability of being in the interval (−∞,tj] if tj < 0) if p-value < α; for, α = 0.01, 0.05, 0.1.
The procedure has to be adapted therefore according to the null hypothesis that is being tested. The specific calculation of the statistics tj and Tjr follows accordingly.

Appendix B: Detailed Bootstrap Results

Table B1. Bootstrap results.
Table B1. Bootstrap results.
TopicPropositionAverage Score (X)Variance s²(2)Level of Agreement/DisagreementConsensusOpinion of Groups
Ho:μ = 2Ho:μ = 3Ho:μ = 4Ho:σ²(1) = σ²(2)Ho:μ = μr = μb = μpHo:μr = μHo:μb = μHo:μp = μ
tptpTptpFptptptp
Investments in physical capital and in education12.240.582.900.00−8.980.00−20.850.002.030.010.790.47−0.010.47−0.720.200.950.13
22.220.542.700.00−9.580.00−21.860.000.210.400.860.45−0.920.150.400.350.870.18
34.630.5631.990.0019.850.007.700.000.860.142.000.152.710.00−0.640.24−0.970.14
43.800.6819.860.008.850.00−2.150.024.120.000.120.90−0.160.460.340.38−0.280.42
54.150.6923.320.0012.460.001.590.041.890.034.430.020.600.281.310.09−2.180.02
64.270.5029.190.0016.320.003.450.002.320.014.840.010.640.211.240.08−2.430.01
74.440.4035.030.0020.670.006.310.002.260.004.870.01−0.580.262.490.00−2.030.01
84.590.3241.400.0025.390.009.370.003.370.001.470.24−0.990.171.420.04−0.180.39
94.760.1957.760.0036.800.0015.850.004.620.000.640.540.840.15−0.760.28−0.050.37
104.800.2155.640.0035.800.0015.970.001.290.060.900.42−0.590.301.590.04−0.380.38
113.711.5212.550.005.200.00−2.150.02−5.180.000.820.45−0.390.32−0.370.351.530.05
124.451.0421.760.0012.880.004.010.00−2.510.000.970.41−0.110.45−0.650.222.670.00
134.260.8821.730.0012.100.002.470.00−3.240.001.080.36−0.520.29−0.320.361.680.04
Information society142.730.697.960.00−2.920.00−13.800.002.030.013.870.03−2.060.021.390.091.430.06
153.950.5424.030.0011.710.00−0.600.241.300.080.300.75−0.550.250.500.31−0.100.41
163.670.7717.280.006.940.00−3.410.00−0.700.230.270.78−0.380.38−0.080.450.870.17
172.880.958.170.00−1.130.12−10.440.00−0.070.451.390.27−0.970.171.410.08−0.020.44
182.460.894.440.00−5.140.00−14.720.000.810.218.400.00−2.940.002.620.000.540.29
192.760.907.210.00−2.320.01−11.860.003.520.004.500.02−2.160.021.940.02−0.030.43
202.540.675.930.00−5.120.00−16.170.005.450.000.210.820.420.32−0.130.46−0.490.32
213.050.7411.050.000.510.28−10.020.003.380.000.560.58−0.840.210.640.260.070.43
223.820.7419.080.008.580.00−1.920.032.640.002.190.13−0.770.201.540.05−0.930.14
234.260.3932.700.0018.200.003.710.003.680.000.990.40−0.420.330.980.17−0.850.15
244.790.1762.000.0039.800.0017.600.003.980.000.030.98−0.130.410.010.430.200.40
254.780.3244.400.0028.430.0012.460.001.250.130.050.960.050.43−0.150.400.320.39
264.830.1762.500.0040.410.0018.320.002.480.000.140.89−0.280.400.400.47−0.170.35
273.100.6612.260.001.090.12−10.080.002.910.000.240.79−0.130.450.420.31−0.620.26
R and D investments284.130.6124.720.0013.130.001.550.071.740.000.360.710.450.29−0.650.290.260.46
294.770.2352.320.0033.420.0014.520.003.640.000.040.970.230.44−0.100.40−0.130.38
304.560.7227.360.0016.680.005.990.001.310.011.980.160.750.210.760.25−1.170.14
Disruptive innovation314.610.6629.080.0017.930.006.790.000.440.180.440.660.310.380.290.41−0.770.23
324.200.6325.080.0013.660.002.230.010.690.230.520.61−0.570.260.870.18−0.060.41
334.110.5725.350.0013.330.001.320.111.840.030.530.600.240.450.360.40−1.010.17
342.941.447.080.00−0.460.30−8.000.00−2.650.011.340.291.030.14−1.160.110.240.45
353.670.9615.410.006.190.00−3.040.002.870.003.140.06−1.590.062.100.020.070.45
363.001.118.590.000.000.49−8.590.00−2.800.005.220.011.390.07−2.640.001.430.04
373.790.9616.600.007.340.00−1.920.021.260.094.850.01−2.000.022.590.00−0.160.45
384.370.4631.690.0018.290.004.900.00−0.280.390.020.99−0.120.450.120.450.060.43
394.290.2838.980.0021.980.004.980.005.920.001.190.32−0.870.221.300.10−0.300.41
R and D and innovation funding404.130.4628.390.0015.090.001.780.040.570.302.220.13−0.950.201.690.03−0.850.15
412.570.646.470.00−4.820.00−16.120.001.790.020.880.43−0.520.27−0.330.381.150.14
424.660.3043.830.0027.340.0010.860.001.200.021.600.211.370.08−1.110.110.240.41
433.790.4424.530.0010.850.00−2.840.001.210.051.630.22−0.070.451.410.05−1.460.08
443.760.4822.880.009.850.00−3.180.001.230.072.090.15−0.220.451.590.04−1.400.06
453.830.4125.720.0011.660.00−2.400.013.370.000.600.57−0.650.260.780.25−0.170.36
464.150.4529.060.0015.520.001.980.020.380.330.020.980.130.47−0.070.45−0.170.40
474.040.4826.620.0013.550.000.480.343.140.000.010.99−0.090.45−0.010.450.180.41
483.990.4626.640.0013.240.00−0.160.403.290.001.140.341.190.11−0.810.21−0.450.37
R and D and innovation fiscal incentives494.200.5327.320.0014.880.002.430.002.180.000.260.77−0.260.37−0.160.440.780.26
504.740.2946.010.0029.240.0012.470.001.190.010.510.630.880.18−0.440.31−0.470.39
514.040.5824.240.0012.340.000.440.361.980.010.150.87−0.270.38−0.010.450.570.26
524.760.2451.370.0032.730.0014.090.003.520.000.220.820.400.290.020.47−0.570.21
Social value of innovation531.380.68−6.820.00−17.780.00−28.740.002.220.020.180.85−0.420.390.430.28−0.010.48
544.800.1859.260.0038.140.0017.010.002.280.010.380.690.090.43−0.490.280.820.12
553.841.0716.100.007.350.00−1.390.08−0.320.370.670.540.890.20−0.210.41−0.900.18
564.020.6922.060.0011.160.000.270.381.030.060.130.88−0.320.390.320.350.270.41
573.100.7611.430.001.020.14−9.400.002.360.003.090.06−0.560.30−0.900.202.560.01
584.240.3832.780.0018.170.003.560.001.970.020.650.530.570.34−0.770.190.570.21
594.290.4829.930.0016.880.003.820.001.670.031.240.31−1.090.150.640.251.130.10
604.780.2748.260.0030.900.0013.550.001.240.071.910.17−0.720.263.140.00−0.780.21
Cooperation for R and D and innovation between academia and industry614.130.5625.780.0013.700.001.620.061.730.043.790.03−1.810.042.210.01−0.050.41
624.740.4437.470.0023.810.0010.160.00−0.590.290.850.46−0.340.35−0.410.333.100.00
634.570.7227.520.0016.830.006.130.00−2.210.011.940.15−1.140.120.950.161.760.00
644.221.1119.060.0010.470.001.890.02−1.560.032.810.07−1.360.082.610.000.110.48
654.370.2840.180.0023.200.006.210.007.700.001.300.29−1.040.151.240.090.070.40
664.540.3837.500.0022.720.007.930.005.160.000.150.87−0.220.430.470.32−0.230.40
673.890.5423.220.0010.940.00−1.350.08−1.520.080.130.880.010.460.290.36−0.410.29
684.230.6525.080.0013.840.002.600.00−2.510.011.500.251.080.13−1.560.060.080.47
694.400.2940.200.0023.470.006.730.007.980.001.040.37−0.640.311.220.09−0.510.28
704.610.3639.150.0024.150.009.150.005.360.000.520.60−0.150.440.790.19−0.600.29

Appendix C: Summary of the Significance Analysis

Table C1. Significance.
Table C1. Significance.
TopicPropositionAverage Score (X)Level of Agreement/DisagreementConsensusOpinion of Groups
μ < 22 ≤ μ < 33 < μ ≤ 3μ > 4σ²(1) > σ²(2)σ²(1) < σ²(2)μ ≠ μr,μb,μpμr > μμr < μμb > μμb < μμp > μμp < μ
Investments in physical capital and in education 12.24 (***) (**)
22.22 (***)
34.63 (***) (***)
43.80 (***) (***)
54.15 (**)(**) (**) (*) (**)
64.27 (***)(**) (**) (*) (***)
74.44 (***)(***) (**) (***) (**)
84.59 (***)(***) (**)
94.76 (***)(***)
104.80 (***)(*) (**)
113.71 (***) (***) (*)
124.45 (***) (***) (***)
134.26 (***) (***) (**)
Information society142.73 (***) (***) (**) (**)(*) (*)
153.95 (***) (*)
163.67 (***)
172.88 (*)
182.46 (***) (***) (***)(***)
192.76 (***) (***) (**) (**)(**)
202.54 (***) (***)
213.05 (***)
223.82 (***) (***) (**)
234.26 (***)(***)
244.79 (***)(***)
254.78 (***)
264.83 (***)(***)
273.10 (***)
R and D investments284.13 (*)(***)
294.77 (***)(***)
304.56 (***)(**)
Disruptive innovation314.61 (***)
324.20 (***)
334.11 (***) (**)
342.94 (***)
353.67 (***) (***) (*) (*)(**)
363.00 (***)(***)(*) (***)(**)
373.79 (***) (*) (**) (**)(***)
384.37 (***)
394.29 (***)(***) (*)
R and D and innovation funding404.13 (**) (**)
412.57 (***) (**)
424.66 (***)(**) (*)
433.79 (***) (**) (*) (*)
443.76 (***) (*) (**) (*)
453.83 (***) (***)
464.15 (**)
474.04 (***) (***)
483.99 (***) (***)
R and D and innovation fiscal incentives494.20 (***)(***)
504.74 (***)(***)
514.04 (***) (**)
524.76 (***)(***)
Social value of innovation531.38(***) (**)
544.80 (***)(***)
553.84 (***)
564.02 (***) (*)
573.10 (***) (*) (***)
584.24 (***)(**)
594.29 (***)(**)
604.78 (***)(*) (***)
Cooperation for R and D and innovation between academia and industry614.13 (*)(**) (**) (**)(***)
624.74 (***) (***)
634.57 (***) (**) (***)
644.22 (**) (**)(*) (*)(***)
654.37 (***)(***) (*)
664.54 (***)(***)
673.89 (***) (*)
684.23 (***) (***) (*)
694.40 (***)(***) (*)
704.61 (***)(***)
Note: The symbols (*), (**) and (***) stand for 90%, 95%, and 99% confidence, respectively.

References

  1. Edquist, C.; Zabala-Iturriagagoitia, J.M. Public Procurement for Innovation as mission-oriented innovation policy. Res. Policy 2012, 41, 1757–1769. [Google Scholar] [CrossRef]
  2. David, P.A.; Foray, D. An introduction to the economy of the knowledge society. Int. Soc. Sci. J. 2002, 54, 9–23. [Google Scholar] [CrossRef]
  3. Grillitsch, M.; Todtling, F.; Hoglinger, C. Variety in knowledge sourcing, geography and innovation: Evidence from the ICT sector in Austria. Papers Reg. Sci. 2015, 94, 25–43. [Google Scholar]
  4. Kline, J.; Rosenberg, N. An overview of innovation. In The Positive Sum Strategy: Harnessing Technology for Economic Growth; Landau, R., Rosenberg, N., Eds.; National Academy Press: Washington, DC, USA, 1986; pp. 275–305. [Google Scholar]
  5. Freeman, C. The National System of Innovation in historical perspective. Cambr. J. Econ. 1995, 19, 5–24. [Google Scholar]
  6. Freeman, C. The diversity of national research systems. In Science in Tomorrows Europe; Barré, R., Gibbons, M., Maddox, J., Martin, B., Papon, P., Eds.; Economica International Editions: Paris, France, 1997. [Google Scholar]
  7. Lundvall, B.A. National Systems of Innovation: Towards a Theory of Innovation and Interactive Learning; Pinter: London, UK, 1992. [Google Scholar]
  8. Rip, A.; Kemp, R. Technical Change, in Human Choice and Climate Change; Battelle Press: Columbus, OH, USA, 1998. [Google Scholar]
  9. Chesbrough, H.W. The era of open innovation. MIT Sloan Manag. Rev. 2003, 44, 35–41. [Google Scholar]
  10. Dahlander, L.; Gann, D.M. How open is innovation? Res. Policy 2010, 39, 699–709. [Google Scholar] [CrossRef]
  11. Yoon, B.; Shin, J.; Lee, S. Open Innovation Projects in SMEs as an Engine for Sustainable Growth. Sustainability 2016, 8, 146. [Google Scholar] [CrossRef]
  12. Greco, M.; Grimaldi, M.; Cricelli, L. Open innovation actions and innovation performance. Eur. J. Innov. Manag. 2015, 18, 150–171. [Google Scholar] [CrossRef]
  13. De Jong, J.P.J.; Kalvetc, T.; Vanhaverbeke, W. Exploring a theoretical framework to structure the public policy implications of open innovation. Technol. Anal. Strat. Manag. 2010, 22, 877–896. [Google Scholar] [CrossRef]
  14. Smits, R.; Kuhlmann, S. The rise of systemic instruments in innovation policy. Int. J. Foresight Innov. Policy 2004, 1, 4–32. [Google Scholar] [CrossRef]
  15. Magro, E.; Wilson, J.R. Complex innovation policy systems: Towards an evaluation mix. Res. Policy 2013, 42, 1647–1656. [Google Scholar] [CrossRef]
  16. Jacobsson, S.; Johnson, A. The diffusion of renewable energy technology: An analytical framework and key issues for research. Energy Policy 2000, 28, 625–640. [Google Scholar] [CrossRef]
  17. Fagerberg, J. Innovation: A Guide to literature. In Handbook on Innovation; Fagerberg, J., Mowery, D.C., Nelson, R.R., Eds.; Oxford University Press: Oxford, UK, 2005; pp. 1–27. [Google Scholar]
  18. OECD. Science, Technology and Industry Outlook 2010; OECD: Paris, France, 2010. [Google Scholar]
  19. European Commission, Innovation Union Scoreboard 2015. Available online: http://ec.europa.eu/growth/industry/innovation/facts-figures/scoreboards/files/ius-2015_en.pdf (accessed on 16 April 2016).
  20. Borrás, S.; Edquist, C. The choice of innovation policy instruments. Technol. Forecast. Soc. Chang. 2013, 80, 1513–1522. [Google Scholar] [CrossRef]
  21. Salamon, L.M. The Tools of Government, a Guide to the New Governance; Oxford University Press: Oxford, UK, 2002. [Google Scholar]
  22. Jordana, J.; Levi-Faur, D. The politics of regulation in the age of governance. In The Politics of Regulation: Institutions and Regulatory Reforms for the Age of Governance; Jordana, J., Levi-Faur, D., Eds.; Routledge: London, UK, 2004. [Google Scholar]
  23. Hand, J.; Lev, B. Intangible Assets: Values, Measures and Risks; Oxford University Press: Oxford, UK, 2003. [Google Scholar]
  24. Lev, B. Intangible Assets: Concepts and Measurements. In Encyclopedia of Social Measurement; Kempf-Leonard, K., Ed.; Elsevier: Amsterdam, The Netherlands, 2005; Volume 2, pp. 299–305. [Google Scholar]
  25. Lev, B.; Cañibano, L.; Marr, B. An accounting perspective of intellectual capital. In Perspectives on Intellectual Capital. Multidisciplinary Insights into Management, Measurement, and Reporting; Marr, B., Ed.; Elsevier: Amsterdam, The Netherlands, 2005; pp. 42–55. [Google Scholar]
  26. Foray, D. The Economics of Knowledge; MIT Press: Cambridge, MA, USA, 2004. [Google Scholar]
  27. Teece, D.J. Dynamic Capabilities and Strategic Management: Organizing for Innovation and Growth; Oxford University Press: Oxford, UK, 2009. [Google Scholar]
  28. Cañibano, L.; García-Ayuso, M.; Sánchez, M.P. Shortcomings in the measurement of innovation: Implications for accounting standards setting. J. Manag. Gov. 2000, 4, 105–116. [Google Scholar] [CrossRef]
  29. Lev, B.; Zambon, S. Intangibles and intellectual capital: An introduction to a special issue. Eur. Account. Rev. 2003, 12, 597–603. [Google Scholar] [CrossRef]
  30. Powell, S. Accounting for Intangible Assets: Current requirements, key players and future directions. Eur. Account. Rev. 2003, 12, 797–811. [Google Scholar] [CrossRef]
  31. Cañibano, L.; García-Ayuso, M.; Sánchez, M.P. Accounting for intangibles: A literature review. J. Account. Lit. 2000, 19, 102–130. [Google Scholar]
  32. García-Ayuso, M. Factors explaining the inefficient valuation of intangibles. Account. Audit. Account. J. 2003, 16, 57–69. [Google Scholar] [CrossRef]
  33. Leydesdorff, L.; Etzkowitz, H. The Triple Helix as a model for innovation studies. Sci. Public Policy 1998, 25, 195–203. [Google Scholar]
  34. Leydesdorff, L. The triple helix: An evolutionary model of innovations. Res. Policy 2000, 29, 243–255. [Google Scholar] [CrossRef]
  35. Ranga, M.; Etzkowitz, H. Triple Helix systems: An analytical framework for innovation policy and practice in the Knowledge Society. Ind. Higher Educ. 2013, 27, 237–262. [Google Scholar] [CrossRef]
  36. Turoff, M.; Linstone, H. The Delphi Method; Addison Wesley: New York, NY, USA, 1975. [Google Scholar]
  37. Wright, G.; Lawrence, M.J.; Collopy, F. The role and validity of judgment in forecasting. Int. J. Forecast. 1996, 12, 1–8. [Google Scholar] [CrossRef]
  38. Landeta, J. Current validity of the Delphi method in social sciences. Technol. Forecast. Soc. Chang. 2006, 73, 467–482. [Google Scholar] [CrossRef]
  39. Helmer, O. Looking Forward; Sage Publications: Beverly Hills, CA, USA, 1983. [Google Scholar]
  40. Sánchez, M.P.; Salazar, J.C. El Papel de la Innovación en un Nuevo Modelo Económico Español; Accenture: Madrid, Spain, 2010. (In Spainish) [Google Scholar]
  41. Dosi, G. Technological paradigms and technological trajectories: A suggested interpretation of the determinants and directions of technical change. Res. Policy 1982, 11, 147–162. [Google Scholar] [CrossRef]
  42. Steinert, M. A dissensus based online Delphi approach: An explorative research tool. Technol. Forecast. Soc. Chang. 2009, 76, 291–300. [Google Scholar] [CrossRef]
  43. Hoyle, R.H. Statistical Strategies for Small Sample Research; SAGE Publications: Thousand Oaks, CA, USA, 1999. [Google Scholar]
  44. Efron, B. Bootstrap Methods: Another Look at the Jackknife. Ann. Stat. 1979, 7, 1–26. [Google Scholar] [CrossRef]
  45. Davidson, A.C. Bootstrap Methods and Their Application; Cambridge University Press: Cambridge, UK, 1997. [Google Scholar]
  46. Glassel, A.; Kirchberger, I.; Kollerits, B.; Amann, E.; Cieza, A. Content Validity of the Extended ICF Core Set for Stroke: An International Delphi Survey of Physical Therapists. Phys. Ther. 2011, 91, 1211–1222. [Google Scholar] [CrossRef] [PubMed]
  47. Callahan, J.L.; Aubuchon-Endsley, N.; Borja, S.E.; Swift, J.K. Pretreatment Expectancies and Premature Termination in a Training Clinic Environment. Trai. Educ. Prof. Psycholog. 2009, 3, 111–119. [Google Scholar] [CrossRef]
  48. Akins, R.B.; Tolson, H.; Cole, B. Stability of response characteristics of a Delphi panel: Application of bootstrap data expansion. BMC Med. Res. Methodol. 2005, 5, 1–12. [Google Scholar] [CrossRef] [PubMed]
  49. Wakai, A.; O’Sullivan, R.; Staunton, P.; Walsh, C.; Hickey, F.; Plunkett, P. Development of key performance indicators for emergency departments in Ireland using an electronic modified-Delphi consensus approach. Eur. J. Emerg. Med. 2013, 20, 109–114. [Google Scholar] [CrossRef] [PubMed]
  50. Ramu, P.; Arul, S. Estimating probabilistic fatigue of Nitinol with scarce samples. Int. J. Fatigue 2016, 85, 31–39. [Google Scholar] [CrossRef]
  51. Doncel, L.M.; Grau, P.; Otamendi, J.; Sainz, J. The truth about mutual funds across Europe. Appl. Econ. Lett. 2011, 18, 687–692. [Google Scholar] [CrossRef]
  52. Alqasim, K.M.; Ali, E.N.; Evers, S.M.; Hiligsmann, M. Physicians’ views on pay-for-performance as a reimbursement model: A quantitative study among Dutch surgical physicians. J. Med. Econ. 2016, 19, 148–157. [Google Scholar] [CrossRef] [PubMed]
  53. Kao, H.Y.; Wu, W.H.; Liang, T.Y.; Lee, K.T.; Hou, M.F.; Shi, H.Y. Cloud-Based Service Information System for Evaluating Quality of Life after Breast Cancer Surgery. PLoS ONE 2015, 10. [Google Scholar] [CrossRef] [PubMed]
  54. OECD. OECD Guide to Measuring the Information Society; OECD: Paris, France, 2011. [Google Scholar]
  55. OECD. Main Science and Technology Indicators 2010/2; OECD: Paris, France, 2011. [Google Scholar]
  56. Madrid-Guijarro, A.; Garcia, D.; Van Auken, H. Barriers to Innovation among Spanish Manufacturing SMEs. J. Small Bus. Manag. 2009, 47, 465–488. [Google Scholar] [CrossRef]
  57. European Commission. RICARDIS: Reporting Intellectual Capital to Augment Research, Development and Innovation in SMEs; European Commission: Brussels, Belgium, 2006. [Google Scholar]
  58. OECD. Tax Incentives for Research and Development: Trends and Issues; OECD: Paris, France, 2003. [Google Scholar]
  59. FECYT. Percepción Social de la Ciencia y la Tecnología 2008; FCYT (Fundación Española para la Ciencia y la Tecnología): Madrid, Spain, 2009. (In Spainish) [Google Scholar]
  60. Lundvall, B.A.; Borras, S. Science, Technology and Innovation Policy. In Innovation Handbook; Fagerberg, J., Mowery, D.C., Nelson, R., Eds.; Oxford University Press: Oxford, UK, 2005; pp. 599–631. [Google Scholar]
  61. OECD. Pisa 2009 Results: What Makes a School Successful? Resources, Policies and Practices IV; OECD: Paris, France, 2010. [Google Scholar]
  62. OECD. The OECD Innovation Strategy: Getting a Head Start on Tomorrow; OECD: Paris, France, 2010. [Google Scholar]
  63. Edquist, C.H. Innovation Policy. A Systemic Approach. In The Globalising Learning Economy_ Major Socio-Economic Trends and European Innovation Policy; Archibugi, D., Lundvall, B.A., Eds.; Oxford University Press: Oxford, UK, 2002; pp. 219–238. [Google Scholar]
  64. Allen, D.K. Organisational climate and strategic change in higher education: Organisational insecurity. High. Educ. 2003, 46, 61–92. [Google Scholar] [CrossRef]
  65. Clark, B. Creating Entrepreneurial Universities: Organizational Pathways of Transformation; International Association of Universities Press and Pergamon: Oxford, UK, 1998. [Google Scholar]
  66. European Investment Fund. Available online: http://www.eif.org/what_we_do/equity/news/2012/baltic_innovation_fund.htm (accessed on 30 October 2012).
  67. Baltic Innovation Fund. Available online: http://www.arcticstartup.com/2012/09/26/baltic-innovation-fund-launches-to-pump-e200-million-into-baltic-companies (accessed on 30 October 2012).
  68. OECD. Edif Reform Prioritization Serbia. Belgrade, 1 October 2012. Available online: http://www.inovacionifond.rs/dokumenta/OECD%20SRB%20reform%20prioritisation.pdf (accessed on 30 October 2012).
  69. Serbian Innovation Fund. Available online: http://www.innovationfund.rs/?page_id=523 (accessed on 30 October 2012).
  70. Ranga, L.M.; Miedema, J.; Jorna, R. Enhancing the innovative capacity of small firms through triple helix interactions: Challenges and opportunities. Technol. Anal. Strat. Manag. 2008, 20, 697–716. [Google Scholar] [CrossRef]
  71. Ioppolo, G.; Cucurachi, S.; Salomone, R.; Saija, G.; Shi, L. Sustainable Local Development and Environmental Governance: A Strategic Planning Experience. Sustainability 2016, 8, 180. [Google Scholar] [CrossRef]
  72. European Commission. Third European Report on Science and Technology Indicators. Towards a Knowledge-Based Economy; European Commission: Brussels, Belgium, 2003. [Google Scholar]
  73. Von Tunzelmann, G.N.; Ranga, L.M.; Martin, B.R. The Effects of Size on Research Performance. Concentration of Research and Quality of Output: A Critical Review; Research Report to the Office for Science and Technology—DTI SPRU; Sussex University: Brighton, UK, 2003. [Google Scholar]
  74. Markides, C. Disruptive Innovation: In need of better theory. J. Prod. Innov. Manag. 2006, 23, 19–25. [Google Scholar] [CrossRef]
  75. Christensen, C.M. The Innovator’s Dilemma; Harper Collins Publisher: New York, NY, USA, 1997. [Google Scholar]
  76. Clarysse, B.; Duchêne, V. Participation of SMEs in Government R&D Programmes: Towards a Segmented. Approach to SME Innovation Policy. In Proceedings of the OECD-Working Group on Innovation and Technology Policy, París, France, 20–21 June 2000.
  77. World Intellectual Capital Initiative. Available online: http://www.wici-global.com/ (accessed on 19 March 2012).
  78. The International Integrated Reporting Council. Available online: http://www.theiirc.org/ (accessed on 19 March 2012).
  79. European Commission. Community Innovation Survey (CIS). Available online: http://ec.europa.eu/eurostat/web/microdata/community-innovation-survey (accessed on 16 February 2016).
Table 1. Distribution of level of agreement/disagreement (% of propositions).
Table 1. Distribution of level of agreement/disagreement (% of propositions).
TopicNumber of PropositionsStrongWeakControversy
Investments in physical capital and in education1369.2%30.8%0.0%
Information society1428.6%50.0%21.4%
R and D investments366.7%33.3%0.0%
Disruptive innovation/Entrepreneurship944.4%33.3%22.2%
R and D and innovation funding933.3%66.7%0.0%
R and D and innovation fiscal incentives475.0%25.0%0.0%
Social value of innovation862.5%25.0%12.5%
Cooperation for R&D and innovation between academia and industry1080.0%20.0%0.0%
Total7054.3%37.1%8.6%
Table 2. Changes of consensus over Delphi rounds (% of propositions).
Table 2. Changes of consensus over Delphi rounds (% of propositions).
TopicNumber of PropositionsHigherLowerStable
Investments in physical capital and in education1353.8%23.1%23.1%
Information society1464.3%0.0%35.7%
R and D investments3100.0%0.0%0.0%
Disruptive innovation/Entrepreneurship933.3%22.2%44.4%
R and D and innovation funding966.7%0.0%33.3%
R and D and innovation fiscal incentives4100.0%0.0%0.0%
Social value of innovation862.5%0.0%37.5%
Cooperation for R&D and innovation between academia and industry1050.0%30.0%20.0%
Total7060.0%11.4%28.6%
Table 3. Differences of opinion among groups of experts.
Table 3. Differences of opinion among groups of experts.
TopicNumber of Propositionsμ ≠ μr,μbpμr > μμr < μμb > μμb < μμp > μμp < μ
Investments in physical capital and in education1323.1%7.7%0.0%23.1%0.0%15.4%23.1%
Information society1421.4%0.0%21.4%21.4%0.0%0.0%0.0%
R and D investments30.0%0.0%0.0%0.0%0.0%0.0%0.0%
Disruptive innovation/Entrepreneurship922.2%0.0%11.1%22.2%11.1%11.1%0.0%
R and D and innovation funding90.0%0.0%0.0%22.2%0.0%0.0%0.0%
R and D and innovation fiscal incentives40.0%0.0%0.0%0.0%0.0%0.0%0.0%
Social value of innovation80.0%0.0%0.0%12.5%0.0%12.5%0.0%
Cooperation for R and D and innovation between academia and industry1010.0%0.0%10.0%20.0%0.0%20.0%0.0%
Total7012.9%1.4%7.1%18.6%1.4%8.6%4.3%

Share and Cite

MDPI and ACS Style

Salazar-Elena, J.C.; Sánchez, M.P.; Otamendi, F.J. A Non-Parametric Delphi Approach to Foster Innovation Policy Debate in Spain. Sustainability 2016, 8, 487. https://0-doi-org.brum.beds.ac.uk/10.3390/su8050487

AMA Style

Salazar-Elena JC, Sánchez MP, Otamendi FJ. A Non-Parametric Delphi Approach to Foster Innovation Policy Debate in Spain. Sustainability. 2016; 8(5):487. https://0-doi-org.brum.beds.ac.uk/10.3390/su8050487

Chicago/Turabian Style

Salazar-Elena, Juan Carlos, M. Paloma Sánchez, and F. Javier Otamendi. 2016. "A Non-Parametric Delphi Approach to Foster Innovation Policy Debate in Spain" Sustainability 8, no. 5: 487. https://0-doi-org.brum.beds.ac.uk/10.3390/su8050487

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop