How the value of experience affects learning for Innovation

(1)

How the value of experience affects learning for Innovation

Ewout Masereeuw S2206137

Faculty of Economics and Business MSc BA Strategy & Innovation Management

Supervisor: H.J.Kok

Co-assessor: dr. K.R.E. Huizingh

Date: 20-06-2017

Word count: 13.364 Abstract

Recognizing the importance of experience to the organizational learning process, scholars have predominantly studied experience as a count, arguing that organizations are most likely to learn from experience if the activity is repeated. We argue that learning opportunities do not only result from a high quantity of experience but can also originate from the quality of an experience. We offer a new perspective on the heterogeneity of experience. Two causal mechanisms of salience and reusability are used to explain why quality of experience is likely to provide a learning opportunity to the organization.

We collected data on fuel cell patents funded by the U.S. Department of Energy (DOE) and matched these patents to their respective R&D projects. We then conducted a regression analysis. The results indicated that the quality of experience has a non- linear U- shaped effect on innovation performance, confirming our expectation that firms learn more from failure and success due to high salience and reusability characteristics. Organizations perform comparatively better when learning from high and low quality experiences than when learning from average quality experiences. Results further indicate that a high quantity of average quality experience further constrains an organization‟s ability to learn from experience. This indicates that organizational members face difficulties in interpreting an average quality experience when time is constrained.

Our findings clearly show that the content of experience will affect organizational learning. The

process of learning from experience can therefore not be approximated by a repetition of activities

without accounting for heterogeneity. We urge scholars of Organizational Learning theory to include

experience heterogeneity into future empirical research on experience.

(2)

1 1. Introduction

In today‟s fast moving world, organizations are becoming increasingly dependent on their ability to innovate consistently (Roberts, 1999). Organizations can increase their innovation performance by learning from past experience (Nerkar & Roberts, 2004). Central to a firm‟s ability to innovate is the accumulation of knowledge through organizational learning (Argote & Hora, 2017).

The process of organizational learning is described as “a change in an organizations knowledge as a function of its experience” (Argote & Spektor, 2011, p. 1124 ). Organizational knowledge is retained in the minds of all organizational members and is reflected in the processes, culture and functions of that organization (Argote & Spektor, 2011).Whilst performing organizational tasks, employees are learning by doing; they gain experience and specialize and improve future performance (Boh, Slaughter, Espinosa, 2007). What is learned from experience is then diffused through social networks, work routines and tools (Argote & Fahrenkopf, 2016). As a result the organization as a whole adapts and learns from experience.

Recognizing the relevance of experience for organizational learning, scholars have taken to further studying experience as a count, arguing that organizations are most likely to learn from experience if the activity has been repeated a number of times (Cyert and March, 1963; Nerkar & Roberts, 2004;

Zollo & Reuer, 2010; Ghosh, Martin, Pennings & Wezel, 2014). However, although extant research on experience provides valuable insights into how the frequency of experience can improve performance (Nerkar & Roberts, 2004; Hoang & Rothaermel, 2005; Hoang & Ener, 2015), not enough consideration has been given to the heterogeneity of experience. An organization can conduct two high quality innovation projects and is likely to learn different lessons from the experience than when it would conduct two „regular‟ innovation projects. As such we argue that the content of an experience will influence its value as a learning opportunity.

To our knowledge there has been little research that examines the actual value of separate experiences and its impact on organizational learning. Scholarly interest has increasingly been focused on organizational learning from both success and failure (March, Sproull & Tamuz, 1991; Baum & Dahlin, 2007; Kim, Kim & Miner, 2009; Madsen & Desai, 2010; Deichmann & Van den Ende, 2014). This research typically views success and failure as the only two available outcomes. However, we argue that it is beneficial to place the quality of an experience on a continuum, where its position will determine what lessons the experience will offer. We intend to fill this research gap by researching the effect of the project quality of past experiences on a firm‟s innovation performance.

The value of past experience for future innovation is dependent on the salience and reusability

characteristics of that experience. First, it depends on the salience of the experience: is the experience

meaningful enough to be remembered? Second, it depends on the usefulness of the experience: can the

experience be reused in a different situation? We argue that the project quality of an experience will

(3)

2 determine its salience and reusability characteristics. A firm is better able to utilize an experience for innovation when that experience is both salient and useful, making that experience more valuable. Using insights from Organizational Learning theory our research will evaluate how the project quality of past innovation experience influences the value of that experience.

We hypothesize that an experience‟s salience and reusability is highest when past innovation project experience is of high or low quality. Projects that are of high or low quality will therefore contribute the most to future innovation performance, whilst projects of average quality will contribute the least. Additionally we hypothesize, that due to resource constraints, this effect is negatively moderated by the quantity of experience. We argue that for a firm to learn from the quality of experience, organizational members need to take time to interpret and reflect upon the experience (Keegan & Turner, 2001; Beck & Plowman, 2009). A high quantity of projects will constrain organizational resources (Levinthal & Wu, 2010), and subsequently limit the time and ability of organizational members to learn from the quality of past experience. To test our hypothesis we conduct an empirical analysis on the use of fuel cell patents. These fuel cell patents are the result of R&D projects that have received government funding from the U.S. Department of Energy. As part of the funding contract; extensive progress data for each project has been recorded. This progress data contains project scores for each project given by a jury of peers, which allowed us to model the quality of past experiences.

Results from our empirical analysis confirm that firms learn more from low and high quality projects and have difficulty learning from average quality experiences. Contrary to our prediction the analysis seems to show that the quantity of innovation project experiences does not negatively moderate the innovation performance that the quality of experience provides. Upon further inspection of this moderation effect it shows that a high quantity of project innovation experience only has a negative moderation effect for average quality innovation project experiences. Innovation performance becomes lower when firms are learning from a high quantity of average quality project experiences. Therefore we find that the ability to learn from an average quality innovation project experience becomes even further constrained in the case of a high quantity of innovation project experience.

Organizational members face cognitive constraints and require more time when learning from average quality experiences and will require more time to do so. As such cognitive and resource constraints may explain the lower innovation performance of learning from average quality experiences.

Our research contributes to organizational learning theory by providing a new perspective on the

heterogeneity of experience. The quality of experience is an important differentiation between

experiences and will result in different types of learning. Central to our analysis is the following research

question: What is the effect of the quality of past experience on the innovation performance of

organizations?

(4)

3 2. Theoretical background

2.1. Organizational Learning and the generation of innovation experience

Organizational learning consists out of processes of knowledge creation, retention and transfer (Argote &

Hora, 2017). To manage organizational learning, the primary role of the organization is that of integrating the knowledge of all organizational members (Kogut & Zander, 1992; Grant, 1996; Turner & Makhija, 2006). With its organizing principles, the organization can decide on the allocation of organizational members to certain activities (Grant, 1996). This will ultimately determine in what activity experience is gained and what lessons will be learned. Lessons from past experience may subsequently be transferred and used to improve performance in new innovation projects.

What organizational members have already learned is retained within the knowledge base of an organization. The knowledge base contains assumptions and expectations about cause and effect relationships that structure the way that organizational members perceive reality, and thus it structures their actions. It represents the „organizational lens‟ through which they view the world (Madsen & Desai, 2010). To better reflect reality and increase performance, assumptions can be refined through lessons learned from experience. We define this experiential learning as the process by which organizational members review their personal and shared assumptions about reality to reflect the lessons learned from experience (Madsen & Desai, 2010). These improved assumptions assist organizational members in finding solutions from previous projects and applying them to problems they encounter in doing their jobs (Darr, Argote & Epple, 1995). As organizations are striving to increase performance, they will come to reflect these reviewed assumptions about reality within its routines, processes and culture (Argote &

Spektor, 2011). Organizational learning is thus a firm shaping process in which the organization uses past experience to improve performance.

2.2 Competitive advantage through experience

Improvements in performance are most likely to grant a competitive advantage if they originate from a process of experiential learning. This is because the evolutionary nature of experience makes it hard for competitors to imitate these improvements (Kogut & Zander, 1992; Barnett, Greve & Park, 1994; Ingram

& Baum, 1997). Experience gained from innovation projects tends to be especially inimitable for two reasons.

First, innovation projects tend to differ from each other substantially. Each innovation project has

different goals and thus requires different skills to be combined in order to achieve these goals (McKee,

1992). An innovation project can therefore be seen as a unique learning experience. The lessons learned

from experience gained in rare and unique events cannot be easily replicated outside its context (Lampel,

(5)

4 Shamsie, Shapira, 2009). Second, to be able to learn from an innovation experience the organization will often require some kind of prior experience with a similar technology to even be able to identify, acquire and integrate the lessons of its current innovation experience (Cohen & Levinthal, 1990; Zahra & George, 2002). Two organizations may learn completely different things from the same experience because their prior experiences are different. Additionally, the reliance on prior experience can lead to path dependency, meaning that the lessons that organizations take away from an experience tend to be similar to what they have already learned in the past (Holmqvist, 2004). Organizational members from both the competitor and the focal firm tend to rely on what they already know in trying to learn something new from experience. Path dependency thus poses a cognitive constraint to both competitors and the focal firm. This cognitive constraint will determine what a firm can learn and makes it harder for competitors to imitate the lessons learned from innovation experience.

2.3 Valuable learning opportunities from different types of experience

To gain a competitive advantage, it is important for organizations to choose innovation projects that will provide experiences with valuable learning opportunities. However, it may be difficult for firms to envision the learning opportunities an experience can offer. After all, innovation activities are at their core uncertain projects (Fleming, 2001). Prior research leads us to believe that firms will benefit from choosing to opt for a high quantity of innovation projects when trying to learn from experience (Cyert &

March, 1963). The logic behind this choice is that repetition of organizational tasks familiarizes organizational members with the experience associated with that task. When an experience becomes familiar to organizational members it becomes more salient and reusable which enables learning. This increases the ability of organizational members to learn from that experience through repetition (Cyert &

March, 1963). However, the quantity of experience is not a prerequisite for organizational learning to occur. An important learning opportunity can also originate from the content of an experience. Because innovation projects have different performance goals and expertise requirements, their contents will vary and provide the organization with different learning opportunities (McKee, 1992).

Organizations must therefore evaluate the content of an experience to determine whether or not

the experience will trigger deep reflection and positively impact innovation performance. An experience

can be said to have an impact on organizational performance when it motivates organizational members

to change assumptions, provides meaningful lessons and alters subsequent behavior (Madsen & Desai,

2010). Whether an experience is able to meet these conditions and impact organizational performance

depends on the salience and reusability characteristics of its content. When the salience and reusability of

an experience is high, a firm is able to learn from it and apply these lessons to increase innovation

(6)

5 performance. Salience and reusability can positively reinforce each other‟s effect on innovation performance. Therefore, it does not particularly matter which of the two characteristics is the most distinctive feature of a given experience. For example, when an experience has high reusability characteristics this facilitates further application of the lessons derived from it, In turn, by application the lessons become more meaningful and salient to the organization. This process allows for a deep understanding of the experience that generates increased innovation performance (Katila & Ahuja, 2002).

Vice versa, organizational learning studies have considered that experience becomes more useful to the firm when it is more salient. For example; Nerkar & Roberts (2004) found that technological experience similar to prior experiences increased innovation performance. Thus, when the similarities between experiences increase, the experience becomes more meaningful and salient to the organization. A salient experience motivates further interpretation and reflection, which leads to a deeper understanding of the experience and increases reusability for future innovation.

In trying to learn from a project innovation experience, organizational members must first interpret that experience and determine how actions have led to the outcome of the innovation project (Lant, 1992; van de Ven & Polley, 1992). To do so, organizational members must assess whether the outcome was positive or negative relative to their performance aspirations (Lant, Milliken & Batra, 1992;

van de Ven & Polley, 1992). The interpretation of the success or failure of an experience is therefore dependent on organizational members‟ performance aspirations for the experience. Research has theorized that organizational members generally aspire to the lowest acceptable performance (Cyert &

March, 1963). An astounding success or embarrassing failure is an obvious deviation from these satisficing performance aspirations. When an experience performs away from prior expectations to such an extent, it may be considered a rare or unusual event and trigger a deep reflection (Baum & Dahlin, 2007; Lampel et al, 2009). Such an experience will have high salience and as a result, organizational members become motivated to learn from that experience. It is important to note that organizations will learn differently from both success and failure experience (March, Sproull & Tamuz, 1991; Baum &

Dahlin, 2007; Kim, Kim & Miner, 2009; Madsen & Desai, 2010; Deichmann & Van den Ende, 2014).

The experience of failure is salient and reusable as the organizations correct wrong behavior and takes

care not to repeat unsuccessful activities (Deichmann & Van den Ende, 2014). The experience of success

is salient and reusable as it provides a frame of reference in which firms can select and reuse activities

that lead towards success (Kim, Kim & Miner, 2009; Anand, Mulotte & Ren, 2016). Since the logic

behind learning from failure and success experiences is dissimilar, the distinction between both types of

experience is important in explaining why characteristics of experience may affect learning. A project

with a low quality experience is regarded by us as a „failed‟ project, while in a high quality experience,

(7)

6 the project is regarded to be successful. We expect that both types of experience are characterized by high salience and reusability and that both types have a considerable impact on future innovation performance.

In the next section we further elaborate on these mechanisms, arguing that the competitive value of experience as a learning opportunity is likely to depend on the quality of experience.

2.4 Hypotheses

In this section we formulate hypotheses that will connect the quality of experience to innovation performance and elaborate on the cognitive constraints that may be imposed by high quantity. First, we argue that experiences with low and high quality projects provide a substantial learning opportunity to the firm, as they both provide salient and reusable experiences. Second, we argue that organizations are cognitively constrained and that a high quantity of experiences will negatively moderate the positive effects of quality experiences.

2.4.1 Quality of experience and its effect on innovation performance. Firms are able to learn from low quality projects because they signal to the organization that its conception of reality has led to low performance. In other words, failure draws attention to particular problems in conceptualization (Deichmann & Van den Ende, 2014). The experience of failure becomes a salient example of what goes wrong in innovation projects. This will motivate organizational members to review their assumptions about cause and effect relationships in order to reflect reality in a better way (Sitkin, 1992; Madsen &

Desai, 2010). Additionally the experience of a low quality project provides a meaningful lesson to employees, who will look for answers and solutions to why a project failed and did not perform to their expectations (Deichmann & van den Ende, 2014). After finding the cause of failure, employees will adjust their behavior accordingly and take care not to repeat the same mistakes. The motivation to pinpoint the reason of failure and adapt behavior with the goal of preventing another failure, means the salience and reusability of that experience is relatively high. A low quality innovation project experience will therefore have a substantial positive impact on future innovation performance.

Moving from low- to average-quality projects, we expect that organizations are less likely to benefit and learn from an average experience for several reasons.

First, organizational members will generally try to satisfice and label the lowest acceptable level

of performance as satisfactory (Cyert & March, 1963). When an average quality experience meets the

minimum performance requirements and is seen as a satisfactory experience, there is no reason for

organizational members to review their original assumptions and update their outcome expectations

(Lant, 1992 ; March & Shapira, 1992?; Madsen & Desai, 2010). This means that average quality

(8)

7 experience is less salient and provides less motivation to interpret the experience thoroughly. As a result of reduced salience; Organizations are less likely to learn from average quality innovation experience.

Second, the costs of interpreting an average quality experience and reusing it for future innovation projects are higher. Because average quality innovation satisfices minimal performance expectations, it is more difficult to pin point what went wrong and what went well during the innovation project. Organizational members are less able to discriminate between substantive issues and everyday noise, which introduces ambiguity about cause and effect linkages into the experience (Van de Ven &

Polley, 1992). Organizational learning studies have suggested that it is more difficult to draw appropriate inferences from ambiguous experiences (Van de Ven & Polley, 1992; Zollo, 2009; Argote et al, 2011;

Ghosh, 2014). Properly interpreting an average quality innovation experience will thus require more cognitive effort, which requires more time and increases the costs of interpretation and learning (Keegan

& Turner, 2001). It follows that it also becomes more costly to generalize and apply this experience to new innovation projects, which ultimately hurts the reusability of the innovation experience.

Third, when inappropriate cause and effect linkages are made, firms risk overestimating their abilities (Zollo, 2009; Ghosh, 2014). If they reuse the inappropriate cause and effect linkages, this might not only reduce the benefit of experience, it can even end up hurting future innovation performance.

Because organizations are less motivated to learn from average quality experience, and because the costs of interpretation are higher, it is tempting for firms to just move on and not try to learn from an average quality innovation experience. When they do try to learn from average quality experience, they run an increased risk of wrongly interpreting the experience. For these reasons we expect average quality innovation experience to be less salient and reusable. Consequently, firms stand to benefit and learn less from average quality project experience.

Whereas average-quality project experience is expected to yield mostly negative outcomes, we

argue that, beyond a certain tipping point, project experience quality will yield a positive impact on

innovation performance. Similar to learning from failure; firms can also learn from high quality

innovation project experiences. High quality innovation project experience challenges and motivates the

firm to discover and interpret the true cause of success. An astounding success can provide a frame of

reference and lead by example (Jensen, Szulanski, 2007; Gersick & Hackman, 1990; Deichmann & Van

den Ende, 2014). Within the uncertain process of innovation, examples from past experience are likely to

be appreciated for their uncertainty reducing qualities (Fleming, 2001). Success is more salient than

average experience because it is considered to be a rare event (Deichmann & Van den Ende, 2014). Rare

events are shown to trigger learning processes that subsequently impact behavior within the organization

(Lampel et al, 2009). Success is therefore likely to provide a salient experience to organizational

(9)

8 members. Because of the salience of such an experience the lessons learned are likely to be reused.

Indeed, organizational learning studies have shown that successful behavior is likely to be selected and repeated (Anand et al, 2016). The lessons learned from success are thus meaningful and salient. When these experiences are reused, they may impact organizational innovation performance.

Firms are using assumptions shaped by high quality experience to select desired behavior and predict success. The overall salience and reusability of high quality innovation project experience provides a valuable learning opportunity from which organizations can draw an example. We therefore expect high quality project experience to have a positive effect on innovation performance.

Regarding the overall effect of the quality of innovation project experience on innovation performance we hypothesize the following:

H1: The quality of its innovation project experience will have a U shaped non-linear effect on innovation performance.

2.4.2 Moderation effect of the quantity of experience. We expect that firms are less likely to benefit from the quality of innovation project experience when these experiences come in high quantity for two reasons.

First, when firms are producing a high count of high quality innovations, the experience of success becomes less rare. When experience becomes more common it reduces the salience of that experience for organizational members. As organizational members become familiar with success, they are likely to relate the experience of success to prior findings instead of trying to interpret the experience in a new way. In interpreting a high count of high quality experience, we therefore expect organizational members to conduct a „local search‟ through which they try and refine existing assumptions, whilst refraining from challenging their prior assumptions through deep reflection (Daft & Weick, 1984; Lant, 1992).

Second, learning from project based experience requires a significant cognitive effort, which demands time and reflection (Keegan & Turner, 2001). To enable reflection, an adequate amount of resources must be allocated to the project. However firms are constrained in their resource capacity and cannot allocate resources freely (Levinthal & Wu, 2010). When organizations have a high quantity of quality experiences, they must spread their resources over multiple product development activities which put limits on the time that is available for reflection. When there is not enough time for reflection upon the quality of experience, interpretation will not happen and learning opportunities are lost (Keegan &

Turner, 2001). The cognitive effort organizational members have to make to ensure that firms learn from

a high quantity of high quality experiences may just be too much for the constrained capacity of the firm.

(10)

9 If this is the case, the opportunity to learn from a high quality experience may be replaced by another high quality experience before the firm could take full advantage of the focal experience.

We expect that a high quantity of experience will negatively moderate learning from the quality of experience, in such a way that all degrees of quality will lead to lower results with high quantity.

H2: The quantity of innovation n experience will negatively moderate the impact of the quality of innovation experience on innovation performance.

3. Methodology

3.1 Sample

We drew our sample from a list of funded patents of the fuel cell sector within the time span of 1977 to 2014 provided by the U.S. Department Of energy (DOE). Multiple industries such as the car industry and chemical battery industry are represented by the fuel cell sector. The list of patents included patents registered to companies, universities, government labs and research institutes. The population from which we drew our sample thus represents a dynamic and diverse competitive environment.

To test our hypotheses, we rely on data sourced from the U.S. Department Of Energy (DOE) Hydrogen Fuel Cell Program. Specifically we used data from two sources: (i), the Annual Progress report of which the Annual Merit Review (AMR) and Peer Evaluation report is an important part, and (ii), the 2015 Pathways to Commercial Success (PCS) report. The AMR and peer evaluation report are part of the Annual Progress report, in which the progress of each Hydrogen fuel cell R&D projects that receive funds from the DOE‟s office of Energy Efficiency and Renewable Energy (EERE) and other DOE offices is discussed. In the AMR Peer Evaluation Report, each project is evaluated by a jury of peers with the primary goal to be able to score projects and compare them. DOE uses the results of this review to make funding decisions for the upcoming years. The evaluation consists of five aspects which are then weighed and compiled into a final score, indicative of the quality of the project.

The PCS report sources from a desire to see what outcome has resulted from the R&D projects

funded from DOE support through the office of EERE. A team from Pacific Northwest National

Laboratory (PNNL) was contracted to create the yearly PCS report. They have further identified and

documented the commercialized and commercially emerging hydrogen and fuel cell technologies and

products that resulted from funded R&D projects. They have sought to find out what projects have led to

a patent registration and subsequently what is being done with that patent. Their patent search has

identified 589 patents in total. These patents have resulted in different outcomes: Some of them are still

being used in research, some are used in commercial products or are commercially emerging, some

patents are being licensed and others are no longer in use. We intend to use these four categories to derive

a measure of innovative performance.

(11)

10 3.2 Matching technique

We started with the 589 patents from the PCS report and searched every patent number within Google Patents in order to retrieve extra information on their respective publication date, filing date, priority date, inventors, international patent classification codes and the contract number under which DOE funding was granted. Using this data, we started to match patents to projects from the Annual Progress report.

First, we checked the comments section of Annual Progress report to see whether or not any patent filing, publication or even patent number was listed. When this was the case a match on project identifier was made. Second, we continued to make matches between patents and projects based on a combination of the title of the patent, the authors, date, contract-number and organization that listed the patent. Once we found a match between the patent and a project we could generalize this project to other patents that were published under the same contract number for the same organization, but only for companies, research institutes and universities. We could not generalize based on contract number for government laboratories. Since government labs often start multiple projects under the same contract, making the match between patent and project based on contract number is therefore less reliable for government laboratories. Once we made all the possible matches between patents and projects, we updated the matches by using the project‟s fiscal year that was closest to the priority date of the patent, bringing the match as close to the point of invention as possible. Additionally, we used the Annual Progress report to add information on the start-up year of the project. In some cases no information on the start-up year was listed, in those cases we took the earliest listing of a project as an approximation to the start-up year. For example; if a project had no known start-up date and its first progress report was in 2005, we used 2005 as its start-up date.

After matching patent data with project data, we had effectively linked information from the

Annual progress report to the PCS report, thus showing what project led to which patent. From the 589

patents listed we were able to confidently match 312 patents from the PCS report to projects from the

Annual progress report. Making an important correction to the data, we used the information on the start-

up date of the project and checked whether the priority date of the patent had its origin within the time

bounds of each project. If the priority date listed was at an earlier point in time than the startup date of the

matched project, this indicated that the innovation was actually done outside of the project, and thus the

match was faulty. Because our earlier mentioned approximations to the start-up date of a project only

models projects to be younger, but never older, this correction applied a rigorous control to our data. After

correcting for these impossible matches, we were left with a population of 226 matches. Out of these

matches, 94 contained all the required information for our quality of experience variable. These 94

matches contained 5 patents from Catacel cororation, which as it turns out received its government funds

(12)

11 indirectly through the Edison Materials Technology Center (EMTEC). The project matched to the patents by Catacel, did not reflect the project evaluation of Catacel, but rather the evaluation of EMTEC and all its funded projects. We had to drop these 5 matches because we could not use them to model Catacel‟s project experience. As a result our sample consisted out of 89 observations which contained all the information on patent and prior experiences.

3.3 Dependent Variable

3.3.1 Innovation performance. For innovation performance, we relied on the status of the patent in 2015 as reported by the PNNL team whose job is to track technologies for the 2015 Pathways to commercial success report. The status of each technology is updated yearly as quoted from the PCS report (2015: p. 13): “In subsequent years, the PNNL team used a similar technology tracking process to identify new emerging and commercially available technologies and ascertain the current status of technologies identified in previous year”. As noted before; the status of each patent is categorized to either be licensed, used in commercial technologies, still being used in ongoing research, or not being used anymore at all.

Information on the contemporary use and application of an innovation is unique in literature on patents.

Patent data is often limited in modeling performance because it does not show the actual application of the patent. Scholars have approximated the use and application of a patent by measuring its impact by the number of forward citations (Hall, Jaffe & Trajtenberg, 2005; Kelley, Ali & Zahra, 2013), arguing that patents with high impact are an indication of high performance. Our data provides updated information on the application of each patent. On a higher level this gives us information about whether or not a patent has a use that benefits the organization, or remains unused. If a patent remains unused it means that either no practical application has been found or that it is a blocking patent, whose only use then is to prevent competitors from using that patent. Patents that are used are producing value to the organization whereas unused patents do not find a value or are not used to produce value but rather to block competitors from creating value. Our dependent variable uses this dichotomy to distinguish innovations that perform and thus directly contribute to the total innovation performance. We attribute a value of 1 to patents that have been licensed, are being used in commercial technologies, or are still being used in ongoing research. We attribute a value of 0 to patents that are not being used by the focal organization anymore.

3.4 Independent Variable

3.4.1 Quality of experience. From the AMR peer evaluation report, we gathered information on the scores that reviewers attributed to each project. Reviewers scored each project on five aspects, culminating in a final score that captures the overall quality of the project. The first aspect accounts for

„the relevance/potential impact on DOE program goals‟ and has a weight of 0.15 in the final score. The

(13)

12 second aspect concerns „strategy for technical validation and/or deployment‟ and has a weight of 0.20 in the final score. The third aspect concerns „the accomplishments and progress toward overall project and DOE goals‟ and is weighed 0.45 in the final score. The fourth aspect concerns „collaboration and coordination with other institutions‟ and is weighed 0.10 in the final score. The fifth aspect concerns

„proposed future work‟ and is also weighed at 0.10 in the final score. The final score sourcing from these five aspects is an accurate and objective representation of the quality of the R&D project before its completion, a rarity in empirical research on innovation. For our measure of the quality of experience; we used an average of the final score for every fiscal year within the project that preceded the patent. To further clarify our measure for the quality of experience within a project we have provided visualization in Figure 4 in the appendix, an explanation of the figure follows: Figure 4 shows Project 1 with a start-up date of 2006. Our goal is to measure the quality of experience prior to the patent of 2008. To do so, we look at the project years before the year of patenting and as such take the average final score of fiscal years 2006 and 2007 for our measure Quality of experience.

3.5 Control Variables

In this research, we argue that quality of experience will strongly affect innovation performance. However it is important to note that prior research has shown that innovation performance can be predicted by a range of other factors. In this section, we introduce control variables into our model that are likely to either hold predictive power over the performance of an innovation or influence the independent variable.

3.5.1 Quantity of experience. For the quantity of experience we took a count of the fiscal years preceding a patent in a project. The quantity of experience thus represents the amount of years a firm required before generating an innovative outcome from a project. Firms with a high quantity of experience have had multiple opportunities to learn before patenting.

3.5.2 Company. We measure whether or not an organization in our database is a company or a

different type of organization. Organizations may differ in their goals; where companies are striving for

profit, universities have a larger focus on advancing research. Universities are less concerned with

protecting their value and as such they may be less inclined to patent. In creating innovations,

organizations that are not striving for profit may not be as focused on creating patents that can be used in

a commercial product or can be licensed out. Whether or not an organization has commercial goals can

affect the performance of patents created. From the observations in our sample; 61 source from a

company, 11 from universities, 1 from a research institute, and 16 from government labs. We proceeded

to group the university, research institutes and government labs together because their commercial goals

(14)

13 are substantially different from those of a company. After making this arrangement, we added the control variable „company‟ to our model. For this measure we attribute a value of 1 to patents registered by a company. We attribute a value of 0 to patents that are registered by a government lab, research institute or university.

3.5.3 Subcontract. As some of the patents in our database were sourced from the subcontractor of a project, it could be that they had less initiative and control within the project compared to the main contractor. If this proves to be the case then it is possible that subcontractors were less able to utilize their prior experience due to a lack of project control. This would mean that their prior experience would be less important for innovation performance. To account for this variation we added a control variable in which we attribute a value of 1 if a patent came from a subcontractor. We attribute a value of 0 to patents that came from the main contractor.

3.5.4 Number of inventors. Collaboration between inventors is a prerequisite to exchange knowledge within a project. Bigger team size suggests there is increased human capital available for the team to make use of; meaning that there is a larger pool of knowledge available to inventors to draw combinations from. Working in teams has indeed been found to be beneficial to innovation performance and there is some indication that this positive effect increases with team size (Singh & Fleming, 2010). It is plausible that the positive effect of team size on innovation performance benefits from increased knowledge diversity between inventors allowing them to make novel combinations of knowledge (Hoisl, Gruber & Conti, 2017). That is, innovation projects may benefit from increased diversity with an increasing team size. Therefore, we include the number of inventors listed on each patent as a control variable for team size.

3.5.5 Technological breadth. Technological breadth is measured by counting the different product categories a patent combines as indicated by the International Patent Classification codes (IPC).

In organizational learning literature, technological breadth is a common concept. We include

technological breadth as a control variable into our model because research indicates that it can affect our

dependent variable of innovation performance. Technological breadth is shown to have an impact on

performance. For example, Ghosh et al (2014) showed that technological broad patents can have a

negative impact on innovation performance. Technological breadth is even argued to be a core

characteristic to breakthrough innovations (Kelley, Ali & Zahra, 2013). Technological breadth can

therefore impact our dependent variable significantly.

(15)

14 3.5.6 Recency of experience. Recency of experience is measured as the time between the measurement of experience and patent filing. As our data comes from the DOE Annual Progress report of 2015, younger patents may be biased in their innovation performance. Research has shown that knowledge gained from experience tends to decay quickly and becomes forgotten, only recent experience is found to be useful for increasing performance (Argote et al, 1990). Innovation projects may benefit more from recent high or low quality experience. Even though their high salience and reusability could mean that their value decays at a slower pace, the effect is still likely to be relevant. For these reasons we include recency of experience as a control variable.

3.5.7 Patent Subgrouping. Research has shown that technology strategies differ in performance across industries (Zahra, 1996). Environmental factors such as competitiveness and dynamism might have a moderating effect on the performance of different types of innovations (Jansen, Van den Bosch &

Volberda, 2006). Some product categories may be more susceptible to environmental changes in their industry. Our sample consists of patents associated with fuel cell technologies funded by the DOE. These patents can be divided into subgroups, after which our sample consists out of 45 fuel cell patents, 32 hydrogen production and/or delivery patents and 12 hydrogen storage patents. It could very well be that some subcategories are more prone to environmental factors and as a result perform differently. To control for such inter category differences we add patent subgrouping as a control variable.

3.5.8 Missing data. For our independent variable quality of experience we only average those past project scores that are available in our sample. For example; our control variable quantity might indicate that a project has ten previous years of experience but due to missing values we only average three of those years to approximate the quality of past experience. If only good project scores were registered in those three years, this would give us a biased view of the quality of a project. We therefore add a control variable that measures the difference between the quantity of experience and the available scores to show us the amount of missing data.

3.6 Method

In our research, we take a novel approach to measuring innovation performance and measure it by its

current use. A patent is said to be performing and creating value when it is actively being used by the

originating organization. When it is not being used, we argue that it is not creating value. As such our

conceptualization of innovation performance is categorical and is measured as a binary dependent

variable (DV): the patent can be either used (1) or not used (0). Our independent variable (IV) measures

the quality of experience through the use of interval data. As we predict a non-linear relationship between

(16)

15 the quality of experience and innovation performance, we use a logistic regression to estimate the probability of patent use. Data on the quality of experience is normally distributed in our sample and there is no heteroskedasticity in predicting the quality of experience. A binary logistic regressions is a fitting test for our research model considering our research question, the nature of our data and its distribution.

The binary logistic regressions allows for a further interpretation of a non- linear relationship between the quality of experience and patent use. We follow the three step process of Lind & Mehlum (2010) to confirm non-linearity: First we test if the quality of experience is negatively significant and if quality of experience² is positively significant. Second, we test if both sides of the U- shape are significant and take on the expected shape through a U- test. Third, we check if the point of inflection is within the data range so that the quality of experience can actually take on the values associated with the U- shape.

4. Results

4.1 Main results

In reporting our results we start by examining our variables through descriptive statistics and the correlation matrix provided in Table 1. In the correlation matrix, we can see that our independent variable quality of experience has a high correlation with companies. The correlation of 0.433 indicates that companies are associated with high quality experience in our sample. We suspect that this correlation can be explained by our relatively small size, a limitation on which we will elaborate later. Technological breadth has a negative correlation to the quality of experience of -0.220 indicating that broad patents from our sample are associated with lower prior quality of experience. This could be because organizations might be prompted, after experiencing low quality projects, to try something new and experiment with adding additional technological categories into their patent. The patent subgroup is negatively correlated at -0.248 to the quality of experience. Consequently some subgroups may have consistently experienced lower quality projects within our sample. This indicates that either their respective industries may not appreciate innovation to a similar extent or they would have lacked the resources to produce high quality experiences.

Table 2 shows our regression results. All research models used proved to be significant; their

explanatory power grows as we introduce variables in subsequent models as can be seen in the pseudo R2

values. In model 1 we can see that, including the quantity of experience, the control variables

technological breadth and missing data are found to be significantly related to the use of patents. In model

2 we introduce our independent variable, but main effect of Quality of experience remains insignificant in

model 2. Model 3 introduces the quadratic term for quality of experience. We find that the main effect of

quality of experience is negative and statistically significant (β = -200.81, p<0.05) and the quadratic term

of quality of experience² is positive and statistically significant (β = 34.73, p<0.05). Model 4 introduces

(17)

16 an interaction effect which is positive and statistically significant (β = 9.13, p<0.1) a finding that contradicts with our second hypothesis H2. Additionally model 4 shows that the negative and statistically significant effect of the quality of experience grows (β = -297.92, p<0.05) and quality of experience² gains a more positive effect (β = 46.89, p<0.05). Therefore we can say that model 3 and model 4 both indicate that quality of experience passes the first condition for a U- shape of Lind & Mehlum (2010). For the second condition we conducted a test for the U- shape which was found to be significant (t = 2.36, p<0.01). The lower bound was found to have a significant negative slope of -38.5 (p<0.01) and the upper bound had a significant positive slope of 20.38(p<0.01). This confirms that both sides of the U- shape are significant. We continued to test the third condition for a U- shaped effect and plotted the quality of experience within the margins of its minimum and maximum values to show that the turning point is within the data range, the resulting graph can be found in Figure 1. Model 3 and 4 therefore show that quality of experience passes all three U-shape conditions of Lind & Mehlum (2010). Hence, we can confirm our first hypothesis H1: “the quality of experience will have a U- shaped non-linear effect on innovation performance.”

To better understand the moderating effect of quantity, we continue to test the interaction effect over quality of experience² in model 5, which introduces an interaction effect with quality of experience².

The explanatory power of the model grows and we see that all our control variables become significant.

The effects found in the main effect of quality of experience and the quadratic effect of quality of experience² seems to have flipped shape to an inverted U- shape: The quality of experience is found to be positive and statistically significant (β = 791.72, p<0.01) and the quadratic effect of quality of experience² is negative and statistically significant (β = 75.16, p<0.01). The interaction effect with quality of experience is negative and statistically significant (β = -442.05, p<0.01). And the interaction effect with quality of experience² is positive and statistically significant (β = 75.16, p<0.01).

The shape flip indicates that the effect of quality of experience remains a U- shape as long as

Quantity takes on a meaningful value of 1 or more. Because the quality of experience is part of an

experience itself, an organization requires at least 1 quantity of experience to even be able to learn from

the quality of experience. It then follows that quality of experience always has a U- shaped non- linear

effect on patent use. To determine how quantity affects the quality of experience we test whether or not

the turning point shifts to the right or left. The test shows that there is no significant difference between

the location of the turning point before and after the shape flip (p = 0.321). The point of inflection thus

stays at the same value of quality of experience regardless of the quantity of experience. In model 5 we

observe that the U- shape is steepening based on the quality * the quantity of experience interaction and

the quality² * the quantity of experience interaction. We can then conclude that the moderation is a case of

multiplicative steepening in which the moderator of quantity strengthens the difficulties in learning from

(18)

17 average quality of experience (Haans, Pieters & He, 2016). We predicted that quantity of experience would affect the quality of experience negatively over the complete U-shape, yet we only find a steepening effect as can be seen in Figure 2. This means that we can reject our second hypothesis H2:

“The quantity of innovation experience will negatively moderate the impact of the quality of innovation

experience on innovation performance”. A high quantity of experience does not negatively affect high

and low quality experience but makes it even more difficult to learn from average quality of experience.

(19)

18 Table 1. (1.1) Description, Means, Standard deviations, minimum and maximum (1.2) Correlation Matrix

Description Count mean sd min max

Dependent Variable

Used Indicator that is 1 if the patent is being used for further research or is in a commercial project and/or is licensed.

89 .752809 .4338228 0 1

Independent Variables

Quality of exp. Average final score of fiscal years within a project before the point of invention.

89 3.032657 .3912281 1.505 3.583333

Control Variables

Quantity of experience Count of fiscal years within a project before the point of invention.

89 2.696629 1.228492 1 6

Company Indicator that is 1 if the organization is a company.

89 .6853933 .46699 0 1

Subcontract Indicator that is 1 if the patent has been applied for by a subcontractor

89 .1123596 .3175976 0 1

Number of inventors Number of inventors listed on a patent.

89 3.033708 1.855272 1 10

Technological breadth Number of IPC classes listed on a patent.

89 3.11236 2.698645 1 17

Recency of experience Number of years between the fiscal year of the project and the priority year of a patent.

89 -.011236 .9942377 -6 2

Subgroup Patent subgrouping for fuel cell technologies.

89 1.629213 .7132197 1 3

Priority Year Year in which the patent was first applied for.

89 2006.045 1.413491 2004 2009

Missing Data Amount of final score data missing in a project before the fiscal year of patenting

89 .7078652 .606822 0 2

1.1 Description, Means, Standard deviations, minimum and maximum

(20)

19

Used Quality

of exp.

Quantity of exp.

Company Subcontract No.

inventors

Technol.

breadth

Recency of exp.

Subgroup Priority Year

Missing data

Used

1

Quality of exp.

0.0511 1

Quantity of exp.

-0.150 -0.0409 1

Company

-0.175 0.433 -0.144 1

Subcontract

-0.311 0.136 0.147 0.234 1

Number of inventors

0.168 -0.102 -0.0900 -0.119 -0.279 1

Technological breadth

-0.0686 -0.220 0.0158 -0.189 -0.145 0.0465 1

Recency of exp.

0.0926 0.0660 -0.121 0.0836 -0.0490 -0.0809 0.0164 1

Subgroup

0.167 -0.248 -0.113 -0.115 -0.201 -0.0797 -0.0901 0.189 1

Priority Year

0.195 -0.121 0.205 -0.189 -0.187 0.00169 0.220 0.435 0.137 1

Missing data

-0.0778 0.0351 0.692 -0.0945 0.143 -0.118 0.0717 -0.0203 -0.0539 0.0584 1

Company -0.77

[0.82]

-1.01 [0.96]

-0.82 [0.99]

-0.40 [1.31]

1.55 [1.43]

Subcontract -1.15

[1.07]

-1.12 [1.01]

-1.38 [1.50]

-2.82 [1.77]

-6.07

^***

[1.81]

Number of inventors

0.24 [0.22]

0.25 [0.22]

0.57

^**

[0.23]

0.86

^***

[0.33]

1.55

^***

[0.41]

[0.84]

1.26 [1.01]

5.17

^***

[1.70]

5.50

^***

[1.51]

8.53

^***

[2.15]

Quality of exp. 1.05

p 0.00 0.00 0.00 0.00 0.01

Table 2. Regression results

Standard errors in brackets

Year and fuel cell division dummies in all models

* p < 0.1, ^** p < 0.05, ^*** p < 0.01

(22)

21 Figure 1. Margins plot Quality of experience

Figure 2. Margins plot Quality of experience for a Quantity value of 1 and 3

(23)

22 4.2 Robustness checks

To check if our results are robust we conduct another binary logistic regression in which we replace our independent variable . In table 3 the results of these regressions can be found.

We replaced our independent variable for a measure of prior quality experience across projects.

This measure takes the average final scores, from one organization, of one fiscal year, over all projects that were active in that year: the specified fiscal year is the year that was closest to the priority year of a patent. The measure shows what experience occurred right before the focal project that led to a patent. To further clarify the measure for prior quality of experience,a visualization of our measure is provided in Figure 5.1 and 5.2 in the appendix. An explanation for both figures follows: Figure 5.1 shows Project 1 as the project that was matched to the patent with a start date of 2005. The patent was applied for in 2007.

To measure prior experience closest to the year of patenting; we take the average final scores in 2006 of Project 1, Project 2 and Project 3. In Figure 5.2, Project 1 starts in 2007 and applies for a patent in the same year, thus there is no prior experience within Project 1. In this case prior experience that is closest to the year of patenting consists out of the average score in year 2005 for both Project 2 and 3.

This measure for prior quality of experience of an organization provides a good robustness check for two reasons: First, our measure for prior quality of experience within an organization uses the most recent experience. This means that there is less risk of decay in the usefulness of quality experience over time. Therefore we are likely to observe the undiluted effect of the quality of prior experience on innovation performance, directly after the experience was gained. Second, the measure for prior quality of experience accounts for experience from other projects within the same organization. A form of indirect learning is included in which organizational members in a project can learn from other organizational members allocated to a different project. Considering that our research aims to contribute to organizational learning literature, this measure checks if the prior quality of experience is transferrable across projects. If prior quality of experience can maintain its use for innovation performance across different projects, this indicates that, the organization as a whole can truly learn from experience.

The inclusion of indirect experience into our measure and generalization of both direct and indirect learning into one measure of prior experience also has a drawback. Whereas our main test only measures direct experience related to the same project, this new measure uses both direct and indirect experience, which comes from different projects conducted by different organizational members.

Learning from indirect experience requires vicarious learning. Vicarious learning could be a completely

different type of learning and affect innovation performance differently than learning from direct

experience (Gino, Argote, Miron-Spektor & Todorova, 2010; Banerjee, Prabhu & Chandy, 2015).

(24)

23 The results from our robustness check can be found in Table 3. Prior quality of experience has a

negative and statistically significant effect on patent use in model 3 (β=-149.41, p<0.01), model 4 (β=-

142.21, p<0.01) and model 5 (β=-139.12, p<0.05). Prior quality of experience² has a positive and

significant effect on patent use in model 3(β=25,88, p<0.01), model 4 (β=24.91, p<0.01) and model

5(β=24.41, p<0.05). These results indicate non- linearity and we continue to test the shape. The U- shape

is found to be significant (t = 1.4, p<0.1), the lower bound has a negative slope of -9.1 (p<0.1) and the

upper bound has a positive slope of 4.7 (p<0.1). We conducted a test for the U- shape which was found to

be significant (t = 2.36, p<0.01). The lower bound was found to have a significant negative slope of -38.5

(p<0.01) and the upper bound had a significant positive slope of 20.38(p<0.01). When we plot prior

quality of experience within the margins of its minimum and maximum values, we can see however that

the turning point is not within the data range as can be seen in Figure 3. According to the third condition

of Lind & Mehlum (2010) we can therefore not completely confirm the U- shape within the data range in

our robustness check. Our robustness check does not provide additional support for our first and second

hypothesis, even though the regression results do show signs of a U- shape outside the data range. We

will further interpret these findings in the discussion section.

(25)

24 Model Patent use (1) (2) (3) (4) (5)

Prior quantity of exp. 0.13 [0.13]

0.07 [0.17]

0.40

^*

[0.21]

2.35 [4.64]

4.63 [39.74]

Company -1.04

[0.70]

-1.23 [0.77]

0.19 [0.77]

0.18 [0.78]

0.21 [0.78]

Subcontract -1.18

[1.02]

-1.27 [0.99]

-4.39

^***

[1.05]

-4.44

^***

[1.11]

-4.40

^***

[1.05]

Number of inventors 0.15 [0.18]

0.18 [0.19]

0.44

^*

[0.23]

0.43

^**

[0.22]

0.43

^**

[0.22]

Technological breadth

-0.11 [0.08]

-0.09 [0.08]

0.13 [0.11]

0.14 [0.11]

Recency of exp. 0.13 [0.31]

0.21 [0.29]

-0.29 [0.40]

-0.26 [0.39]

-0.27 [0.36]

Missing Data -0.40

[0.30]

-0.31 [0.36]

-1.01

^**

[0.48]

-0.98

^**

[0.48]

-0.98

^*

[0.52]

Prior quality of exp. 0.94

[9.96]

Prior Quality * Quantity of exp.

-0.63 [1.44]

-2.11 [24.89]

Prior Quality² * Quantity of exp.

0.24 [3.91]

Constant 2.95

^**

[1.33]

-0.65 [3.27]

211.89

^***

[65.29]

199.24

^***

[60.98]

194.52

^**

[94.03]

Observations 121 121 121 121 121

Pseudo R

²

0.169 0.178 0.361 0.367 0.367

Wald chi squared 16.97 20.07 35.88 37.44 40.23 Log Likelihood -51.25 -50.70 -39.38 -39.05 -39.05

p 0.26 0.17 0.00 0.00 0.00

Table 3. Robustness check

Standard errors in brackets

Year and fuel cell division dummies in all models

* p < 0.1, ^** p < 0.05, ^*** p < 0.01

(26)