Open, Closed and Hybrid Crowdsourcing for the Best Ideas

(1)

Open, Closed and Hybrid

Crowdsourcing for the Best Ideas

Jan Wilbert Luth

(2)

Master Thesis, final version First Supervisor: Ren´e van der Eijk Second Supervisor: Thijs Broekhuizen

University of Groningen Faculty of Economics and Business MSc Strategy & Innovation Utrecht, February 22, 2013

(3)

Abstract

The relationship between hybrid crowdsourcing and the quality of the best ideas is investigated as the best ideas are the most important outcomes of an idea competition next to the best implemented ideas. A best idea on the one hand consists of measures that determine its external value for the market as well as internal measures determining the implementation chances of the idea. Chosen is to meassure all ideas of an actual hybrid competition at NS (Netherlands Railways) on added value, novelty, feasibility and elaboration.

Furthermore three parties are actively involved in an innovation contest, the crowd, the initiator and the intermediate responsible for the process. Of these three parties, 11 variables have been selected influential on best idea quality. And in three blocks (of control, crowd and process variables) are subsequently three of those variables tested in multiple forms.

(4)

Executive Summary

The testing of the hybrid form of crowdsourcing for ideas is the core of this re-search. Hybrid crowdsourcing starts closed, where people work alone and closes open, where people can build on each others work. All ideas were submitted online on ideecentraal.nl and were measured by 4 experts and 4, partly over-lapping experts did the final measurement of the 26 best ideas. Here are some results:

Closed phase Open phase Total Duration 3 weeks 4 weeks 7 weeks

Visitors 1731 731 2463

Registrations (active solvers) 122 (46) 28 (22) 150 (64)

Log-ins 280 252 532

Ideas produced 109 60 171

Messages posted on ideas 2 82 84 Winning ideas (preselected) 9 (19) 0 (7) 9 (26)

Table 1: Results specified by phase and total competition

The most interesting results is that the first period by far has more ideas, implying that open crowdsourcing and its idea build-up does not function. This could be the case as people were not easily able to experience a flow experience through the ideas and were confronted with much data.

Furthermore as the goal is implementation of ideas, chosen is to look broad on factors influencing best ideas quality. Factors attributed to the crowd, such as its size could improve. The platform as mentioned can be improved, however also seekers or organizers elements could be stimulated. By using less, and highly specific questions the communication is more straightforward resulting in better ideas. Also the process could be focussed on this, by for example only selecting some ideas the crowd might improve in the second round, instead of focussing on all ideas and thereby actually lacking focussing.

(5)

List of Tables

1 Results specified by phase . . . 3

2.1 Measures of added value and its definitions . . . 18

2.2 Sources of motivation . . . 23

2.3 Elements of diversity, all positively influencing best idea quality . 24 2.4 Different influences of the stated problem on best idea quality . . 26

2.5 Mentioned factors of visibility in scientific literature . . . 28

2.6 Feedback functions and sources . . . 29

3.1 Operationalizations of factors influencing best idea quality . . . . 33

3.2 Factors influencing best idea quality applied . . . 35

4.1 Available research data . . . 39

5.1 Regression outputs . . . 47

(9)

List of Figures

1.1 Idea competitions in the innovation process . . . 11

2.1 Conceptual model of factors influencing best idea quality . . . . 31

3.1 Conceptual model of factors in one contest . . . 36

5.1 Pageviews of Idee Centraal by day . . . 45

5.2 Other daily fluctuation metrics of Idee Centraal . . . 45

6.1 Two screenshots of Idee Centraal . . . 50

(10)

Chapter 1

Introduction

Contests have always proven their value for innovation. In 1714 for exam-ple British Clockmaker John Harrison solved a contest by making an accurate portable timepiece, earning himself an enormous reward of near £20.000. The challenge, to come up with a method to determine longitude at sea, was initi-ated after a series of incidents in which one costed nearly 2.000 lives and four warships. Eventually did it led to the development of a whole new generation portable timepieces, thereby justifying the significant investments completely (Che and Gale, 2003).

Another well-known example, that spawned the golden age of steam locomotion, is the Rainhill Trials from the Liverpool and Manchester Railway of 1829. The winning vehicle ”the Rocket” reached an astonishing speed of 46 kilometers per hour and earned the maker a reward of £500 (Fullerton et al., 2002). Interesting here is that the current exploiter of the famous Manchester - Liverpool route (Netherlands Railways, owner of Northern Rail) organizes a unique innovation again, this time to investigate the system of open and closed crowdsourcing for ideas. Open and closed crowdsourcing here stands for the current online idea competitions that are able to vary in the disclosure or exposure of submitted ideas. Will the submitted solutions be publicly visible or not during the contest?

If submitted ideas are visible, then it could be beneficial because contestants can be stimulated, motivated and inspired by earlier solutions. Also it could harm idea quality, because it decreases the variety of the submitted ideas which negatively influences the quality of the best ideas (Girotra et al., 2010). When the submitted ideas are not visible then it could have both effects the other way around; contestants will not be inspired while on the other hand will also not be narrowed in their search for ideas.

(11)

unprecedented; online idea generation platforms are either open, for example My Starbucks Idea (mystarbucksidea.force.com), closed, for instance P&G Connect + Develop (www.pg.com/connect_develop/), or have both open and closed idea generation available, but not in one single challenge, such as Inno-centive (www.innoInno-centive.com).

In the current crowdsourcing literature the topic of hybrid idea generation is still missing. The nearest crowdsourcing-research has been done by Blohm et al. (2011a) and is about the influence of the idea-submitter (group or individual) on idea-quality. Another topic considered in this research that is still uncovered in most crowdsourcing literature is the broader innovation process. While many authors focus on the creation process and some on the selection process (Girotra et al., 2010), the implementation aspect of ideas is often unmentioned. Remark-ably, because successful implementation of creative ideas in an organization is what innovation is about (Amabile, 1996).

1.1 Research questions

This research will combine elements of crowdsourcing, idea contests, innovation and collaboration by investigating the hybrid structure of idea generation for crowdsourcing. A hybrid structure is unprecedented in practice, however has been recommended in crowdsourcing and brainstorming literature (Gegenhuber and Hrelja, 2012; Girotra et al., 2010). The main goal of this research is to determine the value of hybrid crowdsourcing by the following research question:

What is the influence of hybrid crowdsourcing for ideas on the quality of the best ideas?

The dependent variable in this research, best idea quality, has been chosen be-cause it largely determines the success of an idea generation initiative (Boudreau et al., 2011; Girotra et al., 2010; Poetz and Schreier, 2012). The first subquestion focusses on this factor:

1. What determines the quality of an idea and how can it best be measured consistently?

The hybrid structure for idea crowdsourcing can be further decomposed in mul-tiple factors influencing idea quality. Furthermore are there also other factors influential in a crowdsourcing competition and because a laboratory setting, where one factor is relatively easily isolated, behaves significantly different than a real life crowdsourcing competition (Yang, 2012, p. 10, 18-19) the decision has been made to first investigate all important factors influencing best idea quality and then to focus on the factors characterizing hybrid idea generation. Therefor subquestion 2 and 3 are:

2. What are the most important factors influencing the quality of the best ideas in a crowdsourced idea competition?

(12)

After these factors are considered the empirical part of the research will begin. This is based on an online idea competition executed in two rounds at Nether-lands Railways (NS). The experiment starts with a round that has foreclosed ideas and continues with a round where all ideas are insightful. For both rounds the quality of the best ideas will be measured and analyzed. Hereby it becomes possible to investigate the effect on the quality of the best ideas by comparing the two phases, closed and open, stating the fourth subquestion:

4. What is the difference in terms of best idea quality between the hybrid, closed and open form of crowdsourcing?

The difference between the hybrid and closed phase is easy to investigate as the competition starts with a closed phase and finishes as hybrid competition. For the open phase it is more complicated because the starting point of the open phase is a little different than that of an open crowdsourcing contest. However based on the multitude of factors developed, the difference between the open and hybrid form can be investigated.

Furthermore conclusions on how future idea competitions can be organized best in the broader innovation process might be drawn as well. This information would be very interesting as Blohm et al. (2011a) signal a gap in the literature when it comes to transforming ideas into actual innovation. The ”innovation gap” is also mentioned by Schepers et al. (1999): ”people who have the idea are very often not the ones who can turn it into a business”. An innovation contest can be a valuable tool to bridge the gap towards a more comprehensive innovation process (Schepers et al., 1999).

1.2 Research boundaries

Considering the model of idea contests in the innovation process (see figure 1.1), this research will focus on the first phases of innovation, idea generation and selection. The implementation phases are not considered mainly in this research, however will be handled in the recommendations, because by actually executing a crowdsourced idea competition, also valuable information on that topic will be derived because of the interconnected nature of the innovation process. It is important to keep an integrated view of the process, because the ultimate outcome of an innovation process is not the best idea identified (as used in for example Girotra et al., 2010), it are the implemented ideas that generate value.

Figure 1.1: Idea competitions in the innovation process (Schepers et al., 1999).

(13)

crowd-sourcing and brainstorming literature and also literature on idea tournaments will be used. And although every factor influencing idea generation and the influence of factors on best idea quality are considered, the main research will be on the hybrid model of crowdsourced idea generation. The factors of the hybrid model are placed into the perspective of other factors influencing best idea quality to create an integrated view on the topic and to optimally validate the results of this study.

Therefor the research will be grounded on the idea generation process of crowd-sourcing, investigates all the factors considered as (more or less) significantly influential on best idea quality and focuses in more detail on the factors con-cerned with hybrid idea generation.

1.3 Reading guide

This thesis will be structured as follows, first the literature necessary to answer the research questions is handled in chapter 2. The start will be broad, by introducing the field of research and it then narrows down in defining best idea quality (section 2.2) and the factors influencing it (section 2.3). Chapter 2 ends with a conceptual model based on all the theoretical relationships handled in the literature earlier.

After this extended literature research, two further analyses are made on the conceptual model. First the relevant factors for hybrid crowdsourcing are oper-ationalized and secondly are all factors that remain stable in one competition left out of the conceptual model because these factors will also remain stable in the case study as it also contains one competition.

The chosen methodology, to continue, is discussed in chapter 4 and will be followed by the results (chapter 5), a discussion about the results in respect to the research questions (chapter 6) and finalized by conclusions (chapter 7) and management recommendations (chapter 8).

(14)

Chapter 2

Literature

The literature chapter of this thesis can be divided in three major sections that become more specific as more topics are handled. The start will be a review of the research field relevant for this thesis. The second part is more of a zooming in on the quality of the best ideas by handling constructs that could define best idea quality and discussing it, and the final part (sections 2.3 – 2.7) handles the specific factors relevant for a crowdsourcing idea competition that could influence the quality of the best idea.

2.1 Field of Research

In this section the relevant topics will be handled to form an overview of the field of research this thesis will fit in. The topics handled are innovation as the broadest field this thesis focusses on, innovation contests and crowdsourcing as the mechanisms to generate ideas that could spur innovation and finally best idea quality and collaboration in idea generation because that are the two variables this research is directed at.

2.1.1 Innovation

(15)

2.1.2 Innovation contests and crowdsourcing

In an innovation contest, a firm (the seeker) facing an innovation-related prob-lem posts his probprob-lem to a population of independent agents (the solvers) and then provides an award to the agent that generates the best solution (Terwiesch and Xu, 2008). It is a process that starts with the creation of many innovation opportunities followed by a filtering step to select the most promising opportu-nity from among those candidates (Terwiesch and Xu, 2008). Literature often ends with the generation of good (or the best) ideas and not with realization or optimization of the realization chances. Possibly better ideas are easier to implement. It is important to make the distinction between the two phases, idea generation (including selection) and implementation while understanding its interconnectedness.

In history contests have always been a popular mechanism to encourage innova-tion (Boudreau et al., 2011) and it is no surprise that these contests found their way to the internet making use of crowdsourcing. Crowdsourcing is the out-sourcing of tasks, once performed by an enterprise, to a large crowd in the form of an open call (Howe, 2006, 2008). With crowdsourcing it becomes possible to reach an enormous crowd (Terwiesch and Xu, 2008), while with conventional idea competitions the number of contributors is limited because of the need to save on the costs for conducting and evaluating the competition (Fullerton and McAfee, 1999, p. 576). Online, crowdsourced idea competitions are the main idea generation mechanisms for this thesis and the dependent variable of this research is the quality of the best idea identified because this largely determines the success of an idea generation initiative (Boudreau et al., 2011; Girotra et al., 2010; Poetz and Schreier, 2012).

2.1.3 Collaboration in idea generation

Collaboration is a factor that retrieves attention in both the higher and lower aggregations of innovation processes. On a high fundamental level the question with whom to innovate is often asked. Options are to work in one unit, across units or external, by collaborations with parties outside the firm (Hansen and Birkinshaw, 2007). Also whether or not should the innovation be hierarchi-cal managed is questioned in collaboration issues (Pisano and Verganti, 2008). Two variables on this higher aggregation level of collaboration are the opening of an idea generation process (who will be involved) and the form of manage-ment (flat or hierarchical) (Pisano and Verganti, 2008). For idea generation by crowdsourcing these two questions are answered, it is a form of collaboration with externals (open) that is hierarchical managed. Advantages are the large number of broad solutions and control in both the direction of the innovation and capturing value (Pisano and Verganti, 2008).

(16)

voluntarily, meaning that submitters can choose to add content to promising ideas (making it a collaboration) or submitting their own idea. This voluntarily idea build-up makes it harder to investigate the connection between idea quality and its collaborative origin because it could well be that individually created quality ideas received contributions with little extra added value that makes it a collaboration in theory, while in practice it is still the work of one person. In contrary to the findings of Blohm et al. (2011a) are findings of Girotra et al. arguing that idea build-up (in brainstorming) does not lead to better ideas, but in fact to worse-quality (Girotra et al., 2010). The direct conclusions of this research therefor, that collaboration of customers in crowdsourcing lead to better ideas, have to be validated by more research.

The field of brainstorming on its turn does contain multiple publications on col-laboration. Brainstorming, an idea generation technique, originated by Osborn in 1939, is characterized by 4 ground rules: criticism is ruled out (1), ”Free wheeling” is welcomed (wild ideas are welcome) (2), quantity is wanted (3) and combination and improvement are sought (4) (Taylor et al., 1958). Osborn claimed that working in face to face groups would increase the number and quality of ideas compared to working individually. Interestingly, the current consistent finding is that nominal groups, where people first work individually and later come together, can generate more and better ideas than an equal amount of people working in face to face groups (Nijstad et al., 2002; Diehl and Stroebe, 1991; Sutton and Hargadon, 1996). The difference in effectiveness is explained by free riding (decreased motivation because contributions in a group are less identifiable), evaluation apprehension (fewer production caused by fear of negative evaluations) and production blocking (the attribute of productivity loss to the convention that in groups only one can have the word) (Diehl and Stroebe, 1987). In a crowdsourced idea competition the contributor has a large amount of time to generate and submit it’s idea, often varying between 4 and 26 weeks (Blohm et al., 2011a, p. 109) making free riding and production block-ing less likely. In brainstormblock-ing one or more sessions are attended lastblock-ing for minutes or hours, making it a (time and place) synchronous idea generation mechanism, while crowdsourcing for ideas on the other hand is typically asyn-chronous (Girotra et al., 2010, p. 12-13; Shirani et al., 1999, p. 142). However evaluation apprehension could still be an aspect influencing crowdsourcing be-cause of the openness of the internet and the fact that the best idea generators might prefer closed networks (Pisano and Verganti, 2008).

Collaboration can further be translated in specific operational aspects, such as idea visibility, accessibility, prioritization and idea sharing, that are all handled in section 2.6.

(17)

2.2 Best idea quality

The success of idea generation initiatives largely depend on the quality of the best opportunity identified (Girotra et al., 2010; Boudreau et al., 2011; Poetz and Schreier, 2012). In innovation tournaments the extremes matter and one would prefer for example one outstanding idea and 99 bad ideas over 100 merely good ideas (Girotra et al., 2010). However also other measures have been used in the past assessing the success of an idea generation initiative. In this section I will therefor highlight some relevant findings of this topic by discussing idea quality and its constructs. The goal of this section is to provide the fundaments to answer the first subquestion of this thesis: ”What determines the quality of an idea and how can it best be measured consistently?”

2.2.1 Defining (best) idea quality

Idea quality is a vague construct (Blohm et al., 2011a) and although companies search for the best ideas (Girotra et al., 2010; Boudreau et al., 2011; Poetz and Schreier, 2012) and try make those ideas explicit so the knowledge of the ideas can be understood (Bj¨ork and Magnusson, 2009), what a company does or is able to do with an (explicit) idea, in the end determines its value. Because idea quality is measured before a company decides to implement an idea, we consider idea quality as the prediction of the net future value gained by implementing the idea. To determine this value we define two elements. The first element is the intrinsic value of the idea, and the second is the connection with the firm that has to implement and commercialize the ideas. The latter in general is determined by a construct defined as the firm’s absorptive capacities (Cohen and Levinthal, 1990). The first can be measured by internals and externals and is referred in this thesis as the external measure of idea quality. The second will be referred internal idea measurement, because the assessor needs to have knowledge of the firm that searches ideas to implement. It is in line with brainstorming literature that often mentions creativity as a combination of originality (external) and feasibility (internal) measures (Diehl and Stroebe, 1987, 1991; Rietzschel et al., 2006, respectively p. 500, 395 & 246)

After determining these two elements of the quality of an idea, further questions to consider are: What are appropriate constructs for evaluating idea quality? How can these constructs be applied reliably? And how should measures of individual ideas be aggregated to support comparisons across individuals or groups? (Dean et al., 2006)

(18)

and are based on extended literature studies of both authors. Blohm et al. (2011a) for example used 20 studies concerning idea evaluation to count if they applied one of the four concepts. From these 20 studies, 19 used feasibility as a factor, 18 applied novelty, 14 times is added value (relevance) cited and elaboration was mentioned 13 times.

The four factors furthermore correspond well to the defined external (market) and internal (company) separation of measures of idea quality. Firstly feasibility and elaboration can be considered as internal measures of idea quality. And secondly, novelty and added value can be seen as external measures. The two external measures are interesting to compare. Novelty states that new value might be added because the idea has not been commercialized before, however it does not indicate how high the expected value will be when executed well, here added value comes in. It makes both elements contributing in signaling the external value of an idea. Furthermore some ideas are novel because no company is able to implement the idea or the implementation chances have never been considered as positive. If the latter is the case, then the two internal factors on each turn arrive as crucial for the future success of the idea. Can the idea be implemented and how well is it devised for implementation?

To conclude this part about the relation of the four quality factors, before con-tinuing on each factor separately, are the first two (external) concepts correlated because of the ability and practicality on the profit-side (the demand) and are the second two (internal) factors correlated because of the ability and practical-ity on the investment-side (the supply). Together enabling interesting insights on the value of the idea and also directly affecting the core of business, which is the convention of supply and demand on a marketplace, involving many sub-parts enabling one or more transactions that create value.

2.2.2 Novelty

For the first concept, novelty, we recognize a close relation with originality and unexpectedness. Novel ideas are unique, original, not expected from previous information, unmet needs or proposed solution (Ang and Low, 2000; Girotra et al., 2010; Riedl et al., 2010). Other attributes of novelty are ”surprising, imaginative and uncommon” (Riedl et al., 2010).

The origin of novelty is in the fact that ideas have to be new, to the organization or to the perspective of the customer to create value that was not created if the idea was not implemented. Garcia and Calantone (2002) translated this newness in their model to operationalize innovations in innovations that are new to the company and new to the industry, further specified on the dimensions of marketing and technology. The concepts of new to the customer or market and world are not used by Garcia and Calantone because innovations that are new to the industry are also new to the market and innovations that are new to the world are considered a sub-category of radical innovations (i.e. innovations new to the company and industry from a technological and marketing perspective).

(19)

innova-tions are implemented (commercialized) ideas, the view on this typology can be considered backwards orientated. Novelty in the sense of Garcia and Calantone is used as a measure to separate different kinds of innovations, while novelty is further relevant because it can be linked to a larger variety of solutions, enabling better solutions as well as distinction (Hirschman, 1980).

To conclude, novelty enables a larger variety of ideas and thereby also better ideas. It is used by many authors as an important concept to distinguish in-novations (Garcia and Calantone, 2002). And while novelty on its own does not provide full information on the future success of an idea, it does help as an indication of idea potential.

2.2.3 Added value

In the novelty-concept one of the attributes is that an idea should be unexpected, unmet needs or proposed solution, while innovations on the other hand are frequently targeted (Francis and Bessant, 2005) and proposed to be goal based (Litchfield, 2008). Although appearing to be contradictory, for added value a well defined focus definitely simplifies its measurement because when ideas meet the pre specified goals it by definition adds value (Poetz and Schreier, 2012). Meeting prior stated requirements is often used in recent literature, however there are other definitions and they are all more are less different, table 2.1 presents a brief overview.

Measure Definition Source

Market potential not further specified F¨uller et al. (2006, p. 68) Effectiveness extent of problem solving Barki and Pinsonneault (2001) Purchase intent intent to purchase based on idea Girotra et al. (2010, p. 2) Business value idea utility for com. organizations Girotra et al. (2010, p. 7) Customer benefit ability to solve underlying problem Poetz and Schreier (2012, p. 250) (Strategic)

tangible and vital problem solving Riedl et al. (2010); Relevance Blohm et al. (2011b) Meaningfulness value contribution Ang and Low (2000)

Table 2.1: Measures of added value and its definitions

In addition Francis and Bessant (2005) made a more concrete structure for targeting innovation, they argue that a company should focus on one topic at the time and present a model consisting of four P’s, all standing for one specific direction the firm could aim its innovative activities on. It consists of the firm’s products, processes, position (in respect to the firm or its products) and the firm’s paradigm (divided in an inner and outer directed paradigm that can be chosen).

(20)

2.2.4 Feasibility

After focussing on the novelty and value of the idea, which are both fairly external measures of idea quality, we now return at a more internally related concept: feasibility. Feasibility, is very often used as a measure for idea quality and captures the ease of which an idea can be transformed into a commercial product as well as the fit between the idea and organizer (Blohm et al., 2011a). It refers to the organizer’s strategy, capabilities and resources from an internal perspective, however it is also important regarding the image of the organizer to the outside world (Blohm et al., 2011a, p. 111).

Dean et al. (2006) emphasize feasibility as workability and subdivides it (similar to Blohm et al.) into implementability and acceptability (Dean et al., 2006, p. 661), which is an interesting remark because really good ideas that have no breeding ground will eventually not be implemented and are less valuable than ideas that marginally have commercial value and are implemented. As explained in the introduction chapter of this thesis few has been written on the implementation phase of ideas in crowdsourcing, brainstorming and other idea generation literature. The subdivision into ”want to” (acceptability) and ”able to” (implementability) measures is therefor an interesting distinction that can be considered possibly in more phases than only the idea evaluation phase.

2.2.5 Elaboration

Elaboration can be seen as the extent that an idea is complete, detailed and well understandable (Dean et al., 2006, as cited in Blohm et al., p. 111). And while customer generated ideas are often not very elaborated (Riedl et al., 2010, p. 4), elaboration is still a relevant factor in determining the best ideas because of two reasons. Firstly can elaboration tell us something about the developing or the building on others ideas (Paulus, 2000), which is one of the fundaments of hybrid idea generation. And secondly is elaboration important because unclear, vague, incomplete ideas or ideas that contain unclear causality are less useful and valuable than the more specific ones (Dean et al., 2006, p. 662). This argu-ment is important as the main goal of idea generation initiatives is to actually implement and commercialize ideas. The chance of successful implementation of an elaborate idea is higher than of a less elaborated equal idea and in this sense is elaboration also a kind of concept indicating the matureness of an idea (Franke and Hienerth as cited in Blohm et al., 2011a, p. 111).

(21)

2.2.6 Conclusion

Many factors can be chosen to define idea quality and in this research is chosen to start with the internal and external side of an idea and to adapt a well cited and used framework on it. The internal side is the dimension that determines the fit with the initiating company and the external side is connected to the market. For the internal supply side is chosen to discuss feasibility and elaboration and on the external demand side are novelty and added value chosen as measures.

The concept of novelty is difficult to interpret because of the vague terms of originality and unexpectedness that are close to it. A way to handle those terms is to connect newness or novelty to a specific actor, such as a customer, company or an industry. Furthermore is it an interesting factor because it indicates a larger variety of that enables better solutions and distinction from other solutions. The last is relevant when the goal is to capture value on the market.

While novelty is about a variety of ideas and distinction, added value is more linked to the goal based aspect of innovation. Those elements can be contra-dictory and also leaves room for many operationalizations. Table 2.1 provides an overview of the options that can be selected. Interesting is that a very goal based research question makes added value more easy to measure and (as dis-cussed in section 2.5.2) also provides better ideas, while novelty of course can still be measured, and even can be added to the broadcasted problem.

(22)

2.3 Factors influencing best idea quality

Successful implementation of new programs, new product innovations, or new services all depends on a person or a team having a good idea. The social environment assumably can influence these persons in both the level and fre-quency of their creative behavior (Amabile, 1996). A crowdsourcing platform can also be seen as (a large part of the) social environment and thereby has to be designed carefully. Furthermore is it important to know the main factors influencing best idea quality because, while the focus is on the influence of the hybrid form on the quality of the best ideas, other factors can also play a role in the results as the case-part of this study involves a crowdsourced idea competi-tion in the field, which could have significantly other outcomes than laboratory research for idea generation (Yang, 2012).

The goal of this section is to provide an answer on the second subquestion of this research: What are the most important factors influencing the quality of the best ideas in a crowdsourced idea competition? Which will be done by handling factors influencing best idea quality, roughly structured by a framework of (Es-tellés-Arolas and González-Ladrón-de Guevara, 2012), and will be finalized by creating a conceptual model and framework in the next chapter. The framework will recapitulate factors characteristic for hybrid, closed and open crowdsourc-ing and will thereby handle subquestion 3 of this thesis: Which of the factors from subquestion 2 can be considered as characteristic for hybrid crowdsourced idea generation?

(23)

2.4 Crowd

The three characteristics extracted by Estellés-Arolas and González-Ladrón-de Guevara (2012) towards the crowd are: who forms the crowd (1), what has it to do (2) and what it gets in return (3). The first question will be handled by the topics size and diversity of the crowd, the second question about the specific tasks of the crowd are answered in all three parts that discuss influential factors on idea quality (sections 2.4, 2.5 and 2.6). The third question is handled by discussing the influence of different motivational factors.

2.4.1 Size

Innovation contest organizers typically invite and encourage widespread entry (Boudreau et al., 2011). Crowds can be very large and the size of the crowd is an important factor determining the number of outstanding ideas (Boudreau et al., 2011). Three effects are recognizable according to Boudreau et al., firstly does adding competitors harm the motivation of individual crowd members. It takes place over the whole distribution of outcomes in terms of quality and is the largest for the highest percentiles. Secondly, the highest scores on each term do become higher when extra contributors are added (Terwiesch and Xu, 2008). Boudreau et al. also refer to it as the parallel path effect; an innovation contest can be seen as a large search process with multiple paths to solutions. Adding extra competitors increases the chance that competitors choose new ap-proaches/paths in their search process which also enhances the chance that the ultimate solution or maximum value outcome is one of these solutions. Thirdly, a moderating effect will take place depending on the complexity of the challenge. If a problem is high uncertain then the negative incentive effect of adding more competitors diminishes and the parallel path effect increases.

The size of the crowd is a variable close to the number of ideas generated, although not completely because the idea generation technique can also influence the number of ideas (Boudreau et al., 2011), as well as of course the size of the crowd can. Furthermore, because the variance in idea quality across people is large (Girotra et al., 2010), a larger crowd implies a larger variety of ideas with a larger chance of extreme values (Boudreau et al., 2011). The number of produced ideas by an individual is further used by many researchers as an indication of performance, next to mean idea quality and best idea quality (Girotra et al., 2010; Walter and Back, 2011; Kavadias and Sommer, 2009), and remarkably not as source of higher idea quality.

(24)

2.4.2 Motivation

Motivation is a concept that can influence creativity through internal and ex-ternal factors (Amabile, 1996; Blohm et al., 2011a; Leimeister et al., 2009) that are on each turn influenced by internal and external incentives (Walter and Back, 2011). In crowdsourcing, a common external incentive is price money, generally starting at e1000 (Blohm et al., 2011a, p. 109), however too much price money can limit creative behavior and creates a perverse effect (Walter and Back, 2011). More motivators exist in learning opportunities, other direct compensations, self marketing (profiling options) and social motives in the form of appreciation by peers or the organizer (Leimeister et al., 2009).

Furthermore the prospect of realizing one’s own idea (for example by granting seed money), and being recognized for that accomplishment is the strongest motivator of most people (Schepers et al., 1999). Other extrinsic motivators lay in the degree of feedback (B¨orjesson et al., 2006; Schepers et al., 1999; Leimeister et al., 2009; Walter and Back, 2011) and delayed rewards such as career opportunities (Walter and Back, 2011). Finally how to structure the incentives is also relevant. For example choosing a ”winner takes all” policy has other implications and is more wise in certain situations then in others (Terwiesch and Xu, 2008; Boudreau et al., 2011). Another example is to change the award structure in a more performance contingent to make each solver invest more in the competition (Terwiesch and Xu, 2008).

Motivator Effect Source

price money limited stimulating Walter and Back (2011) learning opportunities stimulating (internal) Leimeister et al. (2009) other direct compensations limited stimulating Leimeister et al. (2009);

Walter and Back (2011) self marketing (profiling) stimulating (internal) Leimeister et al. (2009) peer appreciation stimulating (external) Leimeister et al. (2009);

Schepers et al. (1999) prospect of realizing own idea stimulating (internal) Schepers et al. (1999) delayed rewards (e.g. career opp.) stimulating (internal) Walter and Back (2011)

organizer appreciation/feedback stimulating (external)

B¨orjesson et al. (2006); Schepers et al. (1999); Leimeister et al. (2009); Walter and Back (2011)

Table 2.2: Sources of motivation

2.4.3 Diversity

(25)

for solutions enlarges the chance of finding better solutions (Boudreau et al., 2011).

Creating a very diverse crowd is also congruous with the lead-user theory of Von Hippel (1986). Lead users are users whose present strong needs will be-come general in a marketplace months or years in the future (Von Hippel, 1986). Von Hippel identified user understanding and their needs as essential for the development of new or improved products in addition to technology-driven in-novations. When combining this theory with the parallel path effect, we can conclude that (lead-)users are able to identify their own novel paths and that they are therefor very welcome in a crowd. Together with outsiders and pro-fessionals on the area, (lead-)users can form the diverse crowd containing high chances for generating the best ideas.

Diversity comes also in other forms, the etnical background of the team mem-bers for example, when being diverse, has a positive influence on idea quality (McLeod and Lobel, 1992). Other dimensions of heterogeneity are advanta-geous, leading to ”original and high quality problem solutions” and being based on both differences in perspectives and attitudes as well as on differences in ideational ability (McLeod and Lobel, 1992, p. 227). For example Kavadias and Sommer (2009) show that team structure can influence the quality of the best ideas and Boudreau et al. sees an explanation of it in the origin of innova-tion, as being the recombination of existing knowledge and ideas, people from various backgrounds also have various knowledge bases to build upon (Boudreau et al., 2011, p. 5). It is an extension of the view that the amount of innovation potential increases because more parties are actively involved (Leimeister et al., 2009).

Elements of diversity Source

Outsiders/marginal participants Lakhani et al. (2006)

Broad crowd Jeppesen and Lakhani (2009) Lead users Von Hippel (1986)

Etnical backgrounds McLeod and Lobel (1992) Team structure Kavadias and Sommer (2009) Involvement many parties Leimeister et al. (2009) Various knowledge bases Boudreau et al. (2011)

Table 2.3: Elements of diversity, all positively influencing best idea quality

(26)

2.5 Initiator

For the initiator two characteristics are captured by Estell´es-Arolas and Gonz´ alez-Ladr´on-de Guevara (2012): who is the initiator (1) and what does the initiator get in return for the work of the crowd (2). Factors influencing idea quality related to the organizing firm are the seekers brand strength, market maturity and the broadcasted problem. The first two factors have a direct connection with the initiator (1) and the broadcasted problem directly enables the results the initiator is seeking for (2).

2.5.1 Seekers brand strength & market maturity

Starting with the first two factors, research has described the direct connection with more and better ideas; strong brands attract more and more motivated solvers and an initiative that exists for a longer period enables better ideas because of the possibility to contact earlier solvers (Walter and Back, 2011). Furthermore the factor described at the end of the previous section when han-dling the work of Cady and Valentine, team experience, can also be considered as a form of market maturity, directly influencing the quality of the best ideas. Because the more familiar the solvers are with the solving process, the better the problems can be solved, resulting in higher quality ideas.

2.5.2 Broadcasted problem

The broadcasted problem contains many variables that all could influence the outcomes of a crowdsourced idea competition. The nature of the broadcasted problem for example can explain the differences in achievements between labora-tory and real world generated solutions (Terwiesch and Xu, 2008, p. 3). Toubia on its turn noticed the difference between the random and the structured view of innovation. The random view assumes an anarchy of thoughts and is on the base of idea generation tools such as brainstorming. The structured view has led to the development of systematic idea generation approaches (Toubia, 2006, section 3.4). A crowdsourced idea competition is more in line with the structured view, while some choices can fit the random view. For example the broadcasted problem can be generally (random), or specifically (structured) stated (Leimeister et al., 2009). The more structured the task (in combination with significant external rewards), the higher the chance of high quality and quantity ideas (Walter and Back, 2011). Also the narrowing of the desired out-comes can be beneficial. On this topic does Litchfield (2008) also argue in favor of specific goals to align when brainstorming. This goal-based view starts with goals that are set by the seeker and can be maintained by feedback-processes (see also section 2.6.3).

(27)

stated in one longer integrated question (Dennis et al., 1996). Other aspects of the problem that vary, however not (yet) confirmed as influencing idea-quality or quantity, are the required submission type, the description length of the prob-lem and the degree of elaboration of the outcome (Piller and Walcher, 2006; Ebner et al., 2008; Leimeister et al., 2009; Walter and Back, 2011).

Furthermore the complexity of the problem has an important moderating role on the quality of the best idea (Boudreau et al., 2011; Yang, 2012). The greater the set of knowledge components or domains involved in addressing an innovation problem, the higher the expected uncertainty or variability of outcomes, both in failure and breakthrough (Fleming, 2001). Complexity is a factor that can be linked to many other factors, such as the difference between team and individual performance (Taylor, 1995), the number of competitors or solvers (Boudreau et al., 2011; Yang, 2012), solver diversity (Kavadias and Sommer, 2009; Paulus, 2000, p. 250) and the submission speed of a solver (Yang, 2012, p. 54). Possibly also incentives can be linked, empirical research on this connection however has not been found.

Problem aspect Connection Description

Nature of the problem Explains lab. vs. real world Terwiesch and Xu (2008) Structured vs. random view Originates ideation approach Toubia (2006)

Well structured problem Higher quality solutions Walter and Back (2011) Litchfield (2008) Problem division in parts More solutions Dennis et al. (1996)

Higher complexity More variability (top ideas)

Boudreau et al. (2011) Yang (2012)

Fleming (2001)

Table 2.4: Different influences of the stated problem on best idea quality

2.6 Process

Estellés-Arolas and González-Ladrón-de Guevara (2012) mentioned three char-acteristics for the process: the process type (1), the call type (2) and the used medium (3). This section will discuss factors directly influencing the idea gen-eration process that are not strongly connected to the properties of the seeker. These factors are: contest duration, visibility of solutions, provided feedback and either or not working in multiple stages.

2.6.1 Duration

(28)

tends to focus participants attention and thereby increases the number of ideas, while also causing participants to overlook subparts, leading to lower quality solutions (Dennis et al., 1999). Nonetheless, because of the relatively long time frame of online idea contests, and its asynchronous character, we don’t expect any stimulating effects of time pressure on the number of submitted ideas.

2.6.2 Solution visibility

The element of solution visibility or accessibility of peer contributions, to con-tinue, is one that has not received much attention in crowdsourcing research yet. However as explained in the introduction section, it is a significant design element in which crowdsourcing contests can differ. A crowdsourcing platform’s idea access can be closed because of privacy aspects, to ensure diversity or simply because there is no necessity to open it (Geiger et al., 2011). The idea-diversity argument is also used by Yang arguing against visibility of ideas during a con-test because it enables imitation and lowers the diversity of the ideas, which is associated with lower innovativeness (Yang, 2012, p. 146,147).

Geiger et al. (2011) separates four different forms or levels of accessibility: none, viewing, assessing and modifying. These levels can be seen as operationaliza-tions of collaboration for crowdsourcing. None indicates no interaction of so-lution information between contestants. The view characteristic means that all contributions are visible to any potential contributor. Assess implies that contributors have explicit mechanisms to express their opinions on individual contributions and finally modify stands for the alteration or even deletion of each other’s contribution in order to correct, update or otherwise improve them. It is the highest level of accessibility and provides high collaboration (Geiger et al., 2011).

In addition to the modifying property, Blohm et al. (2011a) mentions the wiki technology as a tool for collaboration. Participants can view and edit the ideas of others and are thereby able to improve an existing idea. It can be seen as an attempt to encourage cognitive stimulation by idea sharing; the effect that individuals in a group can generate ideas that would not have been created when working alone (Dugosh et al., 2000). This positive effect takes place in idea generation settings both during and after idea exposure and is moderated by the attentional set of the participant and the content exposed to (number of ideas, presence of irrelevant information) (Dugosh et al., 2000). Idea sharing however could also cause a negative effect. This so called cognitive interference happens when ”stimulus ideas” activate an image that is at odds with a person’s train of thoughts (Nijstad and Stroebe, 2006). It can lead to a shorter train of thoughts (someone does not think an idea through till the end), loss of potential ideas and increasing switching between semantic domains (idea directions); reducing the depth of idea production because rapid associations within a domain are prevented (Nijstad and Stroebe, 2006, p. 537).

(29)

other words it is the combi- or recombination of ideas with ones own ideas that can generate new ideas.

Two elements are influencing this combination according to Javadi et al. (2011): visibility and prioritization. Visibility, generally in line with the earlier men-tioned ”viewing”, defines the extent of information that is presented on the screen and is defined by the portion of the idea pool that is presented on the screen at any given time without extra effort (e.g. clicking) (Javadi et al., 2011, p. 23-24). Prioritization on its turn, is defined as the criterion used for idea or-dering, while technically no prioritization occurs when the ordering is randomly executed or based on chronological order. The most commonly used prioritiza-tion is collective evaluaprioritiza-tion of the group and remarkably this is also one of the few applicable real-time methods for synchronous idea generation systems (e.g. brainstorming) (Javadi et al., 2011, p.27-28). Javadi et al. cited the, also in the initiator-section mentioned, theory of Litchfield (2008) as promising source of prioritization. Litchfield (2008) argues in favor of specific goals to align when brainstorming. This ”goal-based view” improves external evaluation of ideas and comes with a small cost because some valuable ideas are possibly not recog-nized in the narrow scope. However in most organizational contexts these losses are a small price for improved focus (Litchfield, 2008, p. 662). The goal-based view enables prioritization by the crowd itself, only specific goals are required to make it effective.

Operationalizations/aspects of solution visibility Source

None/Viewing/Assessing/Modifying Geiger et al. (2011) Wiki Technology Blohm et al. (2011a) Number of presented ideas/Presence of irrelevant info Dugosh et al. (2000) Visibility/Prioritization Javadi et al. (2011)

Table 2.5: Mentioned factors of visibility in scientific literature

Working with specific goals (in the contrary to general ones) is closely related to working with more complex rating mechanisms. Easy rating mechanisms like ”thumbs up/down” are remarkably similar outperformed by more complex and focussed (specific) mechanisms (Riedl et al., 2010). Furthermore, while contemporary sorting mechanisms often allow sorting based on chronological order, views, likes and categorizations, possibly an equally smart system can be used as in Customer Research Management for product promotion. Here automatic recommendation systems keep track of customers preferences and bring new products under the clients attention for add-on selling (Bodapati, 2008, p. 77).

2.6.3 Provided feedback

(30)

et al., 2009, p. 206). Furthermore can it be seen as idea rating and does it strengthen trust in the quality of the content for further processes (Leimeister et al., 2009, p. 216). It can also harm the quality of the ideas and, when used right, does it improve performance (Litchfield, 2008, p. 657). Schepers et al. also recommend the of use feedback processes during idea contests and are in favor of a contact person during all phases of the competition, including implementation phases. They further argue that even a small gesture as an acknowledgement of the receipt of a submission can serve as an encouragement to the author, resulting in better contributions (Schepers et al., 1999, p. 29). In line with it do Piller and Walcher mark automatic feedback systems and the mentioned peer feedback to improve efficiency and idea quality (Piller and Walcher, 2006, p. 310 & 312). Yang adds that the best feedback consists of encouraging words with detailed improving suggestions (Yang, 2012, p. 29) and that feedback can function as a powerful tool to increase effort in the appropriate direction, preferred by seekers (Schepers et al., 1999; Yang, 2012, p. 19).

Interestingly, in a major crowdsourcing platform, feedback has been sent to only 8,9% of the contestants of 1461 contests (Yang, 2012, p. 31 & 165). And as this feedback is commonly send to the preferred solutions, it is also an indication of the solver’s chance to win, stimulating them to increase the quality of their idea(s) even more (Yang, 2012, p. 23).

Feedback functions Source

Gain learning experiences Leimeister et al. (2009) Idea rating Leimeister et al. (2009) Strengthen trust in quality Leimeister et al. (2009)

Improve quality/efficiency

Schepers et al. (1999) Piller and Walcher (2006) Leimeister et al. (2009)

Encouragement/stimulation Yang (2012) Schepers et al. (1999) Harm idea quality Leimeister et al. (2009)

Automatic feedback Schepers et al. (1999) Piller and Walcher (2006) Peer feedback Leimeister et al. (2009) Seeker feedback Leimeister et al. (2009)

Increasing effort in the preferred direction Yang (2012) Schepers et al. (1999) Indicate win-chances Yang (2012)

Table 2.6: Feedback functions and sources

(31)

2.6.4 Multiple stages

The final topic of this section, concerning elements influencing best idea qual-ity, is to either or not work with multiple stages. Few research has been done directly at the influence of it, however some theoretical work has been done. The topic often returns in the field of brainstorming (see for example Girotra et al., 2010) and there is also an interesting recommendation in the conclusion of an article about motivation caused by the number of competitors in idea contests. Here, Terwiesch and Xu suggested to design a multi-round contest in which the first round contestants are only asked to undertake little investment and then the best contestants subsequently are chosen for a second, ”private” round of idea generation in which they are expected to work harder. The the-ory is that by bringing down the number of submitters, the chances of winning of one contestant increases, which drastically improves solver-effort (Terwiesch and Xu, 2008, p. 1542). Multiple rounds are thus used to overcome an under-investment problem by the contestants. Multiple stages can further be used as a research method, for example as a component to investigate solver behavior in idea tournaments (Tsetlin et al., 2004).

Another perspective on multiple rounds has been given by Gegenhuber and Hrelja (2012). These authors theoretically build a model of hybrid idea gen-eration in line with the one used in this research. First they discuss three (brainstorming-) forms of idea generation, Nominal Groups, Nominal Group Technique (NGT) 1 _{and the Hybrid model and later they combine the best}

el-ements, while focussing on idea selection. The model from Gegenhuber and Hrelja (2012) similarly constructs an open and closed phase (labelled as public and private contests) which is used to determine the access to others ideas. Still everyone can entry the competition and the model also works with two phases in which the crowd can contribute (out of 5 phases total before the winners are selected). And although there is evidence on the brainstorming aspect that a hybrid idea contest performs better than interactive groups, and that some other research papers are cited to justify the direct adoption of elements from the field of brainstorming to web based idea generation, no field research has been done to confirm this performance in the crowdsourcing field.

2.7 Theoretical model of influential factors

To recapitulate all the factors discussed from section 2.3, a conceptual model of the spoken theoretical factors and its connections has been created (figure 2.1). This section handles the choices made and discusses the probability of the model.

Starting with the crowd is the size of the crowd a proven variable that influ-ences the quality of the best ideas, more people adds more competition which is negative for the average crowd as the motivation decreases, however positive for

(32)

Figure 2.1: Conceptual model of factors influencing best idea quality

the quality of the best ideas because the chance to find an extreme value contri-bution increases. Furthermore many factors do influence the size of the crowd and in this model two are concretized. Market maturity influences crowd size as it becomes more easy to contact earlier solvers and also the brand strength of the seeker is influential when comparing multiple competitions. Problem com-plexity is a factor that moderates the influences of crowd size, because the more complex a problem is the larger the variability in expected outcomes which in-creases the chance of an extreme value solution. Secondly a reduced negative incentive effect has been found when increasing complexity or problem uncer-tainty (Boudreau et al., 2011, p. 18-19). This incentive effect is the effect that a larger crowd causes lower winning chances, thereby reducing the motivation of individual solvers. However there are many more elements determining the motivation of the crowd, described in section 2.4.2.

(33)

ma-turity also influences the quality of the best ideas because of the learning curve of contestants. Problem structuring has a positive effect when the problems are more specifically stated (or better structured).

For the process factors also some easy relationships can be observed such as a longer duration results in better high quality ideas and more feedback also results in a more motivated and a better functioning crowd. Multiple rounds can have a moderating role on the negative incentive effect of many competitors by reducing the number of competitors in the final rounds. And visible solutions are expected to motivate the crowd as well to generate better solutions.

(34)

Chapter 3

Conceptual model

From the first conceptual model based on the literature that describes possible relationships this chapter now focusses on the factors that influence best idea quality directly relevant for this research. First all the factors of the conceptual model are generally operationalized (table 3.2) and discussed. Secondly the operationalization helps indicating which factors only apply on the whole contest and which differentiate per phase. And thirdly the elements relevant for hybrid crowdsourcing competition are highlighted and handled. The chosen approach helps answering the research questions and further builds on the model from the literature (section 2.7) with a new conceptual model (figure 3.1) that indicates regular factors as well as factors characteristic for hybrid crowdsourcing.

3.1 Best idea quality operationalized

Factor Operationalizations Crowd size number of solvers

Crowd diversity number of nationalities gender ratio employee % Crowd motivation degree of elaboration ideas generated per person Market maturity platform age number of past challenges Seekers brand strength customer loyalty scores brand value

Problem complexity single-/multi-domain problems Problem structuring generally/specifically stated problems Contest duration number of weeks open for idea submission Solution visibility none/viewing/assessing/modifying

Multiple rounds yes/no number of rounds Focussed feedback total feedback % top idea feedback %

Table 3.1: Possible operationalizations of factors influencing best idea quality

(35)

num-ber of solvers, its diversity and motivation can all be different based on the dynamics of the crowd on that time. The measurement furthermore of the the first is simple, the second is less clear because multiple measures for diversity exist and the third, crowd motivation is even more complex. Motivation is a factor that is based on internal and external incentives of which the internal incentives are hard to change and for example a measure as price money is not straightforward in its results (Walter and Back, 2011). However these are all inputs of motivation, the output would be better ideas, but are those ideas then also for example more elaborate or are it more ideas? The connection between motivation and elaboration has been made also in the literature (Javadi et al., 2011, p. 18) and more and better output as a consequence of motivation makes a logical causality. Although it always remains a relative factor, unskilled highly motivated solvers might for example still deliver low quality solutions.

Continuing on the factors related to the initiator is the measurement logically outlined. Market maturity reveals itself in the lifetime of the contest platform or the number of challenges executed on the platform. Seekers brand strength can be structured by Aaker’s four dimensions of brand equity: loyalty, perceived quality, associations, and awareness (Aaker, 1996). Here I choose to include brand value and customer loyalty as these measures are calculated frequently in business. Furthermore are loyal customers also the customers that would do something in return for the value they perceive of the seeker. To conclude the initiator factors are problem complexity and structuring aspects that can be measured by defining single or multi domain problems (see for example the work of Boudreau et al., 2011) and can problems be very specific or very generally stated, keeping the solution space either clear or vague (see section 2.5.2).

Arriving at the process factors is contest duration measured in weeks because the submission phase of an online idea competition usually takes between 4 and 26 weeks (Blohm et al., 2011a, p. 109). For visibility is chosen to work with the operationalization of Geiger et al. (2011) as it is the most rich concretization of solution visibility and collaboration in one. Multiple round are either or not the case, on the exact number of rounds has not been found any literature, however this can be chosen and investigated as well. And the feedback given can for example be measured by the percentage of feedback that is given, or more in line with Yang (2012), by the percentage that is given to the top tier ideas. Finally even subtle feedback actions such as automatic submission confirmations, as advised by Schepers et al. (1999), can be considered to distinguish multiple competitions.

3.2 Crowdsourcing modes applied

After stating the factors influencing best idea quality in a conceptual model (figure 2.1) and operationalizing them further for the best understanding, now we arrive at a more specific section of this thesis: the key characteristics of the hybrid, closed and open models of online idea generation; presented in table 3.2.

(36)

Closed Open Hybrid

Contest duration Relatively short Relatively short Relatively long Solution visibility No Yes Partly

Multiple rounds No No Yes Focussed feedback No Possibly Possibly

Table 3.2: Factors influencing best idea quality generally applied for closed, open and hybrid crowdsourcing

considering the three forms of crowdsourcing. The other elements can differ as well across different competitions, however can also remain more or less similar; they are not dependent on the crowdsourcing mode and therefor not included in table 3.2. In the case study, however are they mentioned because this research makes use of a focussed, integrated approach as the real world behaves dissimilar to laboratory settings (Yang, 2012).

The four elements differing are all part of the ”process” group in the concep-tual model and literature and the biggest difference is expected to be in the solution visibility as that largely determines if users are able to work together. The contest duration will differ, and will be expected to have its influence on the number of generated ideas and thereby on the quality of the best ideas. Multiple rounds can have its influence on the motivation, however the context Terwiesch and Xu (2008) hinted on is not the direction that hybrid crowdsourc-ing is directed at. And focussed feedback can always be given by the organizer, however peer-feedback cannot be send when the contest is in its closed state.

Finally the duration between the three crowdsourcing modes by definition will not differ or have the difference of more or less a factor 2 and can in that case have a significant influence in the outcome of the contest. However when comparing the effects of the discussed crowdsourcing modes it is desirable to assimilate these periods. Therefor would it possibly be wise to first compare the open and the closed phase of hybrid crowdsourcing before comparing the open with the hybrid form. If the open phase will perform better than the closed phase, then another experiment is desired containing at least two contests (open and hybrid), in which the durations of these contests are assimilated. In the other scenario, conclusions can be supported by only the first contest and a second contest can be simply used for extra deepening.

3.3 Conceptual model for single hybrid contest

(37)

has its influence on the diversity as well as the motivation. The last aspects, feedback and visibility are the two process elements that do differ during the contest as the open phase has both elements possible and the closed phase does not.

(38)

Chapter 4

Methodology

In this research the question is asked what the influence of hybrid crowdsourcing is on the quality of the best ideas. Hybrid crowdsourcing is a form of idea generation that has not been used in practice before and therefor is it relevant to compare to regular crowdsourcing techniques such as the open en closed technique. Interesting are questions concerning the differences in (best) idea quality between both phases. Does idea buildup take place? And what is the effect of it?

Because hybrid crowdsourcing works in an open and closed phase and these phases are very comparable to open and closed crowdsourcing, the results can easily be generated by simply comparing the best ideas of both phases. Three possible scenarios can arrive; the best ideas of the first (closed) phase have sig-nificant better quality, those of the second (open phase) are more valuable or they remain fairly similar. Here, I will not anticipate further on these hypothe-ses. However because of the real-life setting of the chosen research-case, more factors come in play that have to be taken into account as well. In the literature section these factors are already introduced and theoretically analyzed, in this methodology chapter I will introduce them in the chosen research model.

The set-up of this section will be first to introduce the hybrid crowdsourcing competition held at Netherlands Railways (NS) in section 4.1, secondly to ex-plain the possible variables available through this research (section 4.2) and finally the chosen variables and the techniques for analyzing are discussed in section 4.3.

4.1 Data collection

(39)

ideas. The competition started with a phase of three weeks in which almost no interaction between contestants was able and continued with a phase where all ideas became visible enabling co-creation.

The total competition thereby took 7 weeks and had three specific challenges to be solved. For both phases, the first of 3 weeks and the second of 4, solvers could earn prices for each of the three challenges per period. 500 euro for the number 1 in a challenge and two times 200 euro for the numbers 2 and 3. In total (3 challenges times 2 phases) 5400 euro price-money could be earned. A bonus-price was the invitation of top contestants to the NS headquarters for a company presentation and further working on their ideas.

In total 171 ideas are submitted by 64 contestants of which 110 ideas were composed in the closed phase and 60 in the subsequent, open phase of the contest. The ideas were delivered in two forms, a short version that all ideas had with a maximum of 250 words (correlating to about 1 minute reading, considering a 240 words per minute average reading speed) and a long version of max. 4 A4 people could add to their short version. The latter long version was used in 16 submissions. Chosen is to enable both versions to give participants the submission-option they prefer.

In the first evaluation four experts of three different departments, Commerce, Traveller Information and a department that organizes traveller-related aspects in minor and major train incidents (2 experts), were involved. Each expert judged all ideas on a 5 point-scale and were asked to keep the four properties from literature (novelty, added value, elaboration and feasibility) in mind when evaluating the ideas. The chosen properties are in accordance to research of Blohm et al. (2011b) and Dean et al. (2006) and are discussed in more detail in section 2.2 and further.

The four properties were also used for the second evaluation in which the best 26 ideas were ranked and this time each idea received 4 grades from 1-5; one grade for each property. For all four categories a concrete operationalization of the possible scores has been made and communicated to preclude calibra-tion difficulties and ideas that improved during the competicalibra-tion were evaluated twice, once for each phase-state. The experts that evaluated the second phase were chosen on their capacities to realize the ideas that were evaluated and were involved as early as possible in the process to enhance the actual implementa-tion chances of the ideas. The group consisted of the head of transport of NS (Netherlands Railways), the responsible manager of all mobile activities of NS, an outside expert of YES!Delft (an innovation platform that helps young en-trepreneurs of Delft University) and a manager from the ”incidents” department that was also involved in the pre-selection described above.

Open, Closed and Hybrid Crowdsourcing for the Best Ideas