CROWDSOURCING FOR
INNOVATION
A comprehensive literature review and exploratory
research
Master thesis University of Groningen Faculty of Economics and Business
Msc. Business administration: strategic innovation management
Ytsen van der Meer S1999842
ytsenvdmeer@gmail.com
First supervisor: Dr. W.G. Biemans Second supervisor: Prof. Dr. J. Surroca
June 20, 2016
ABSTRACT
TABLE OF CONTENT
CROWDSOURCING FOR INNOVATION 1
ABSTRACT 2 TABLE OF CONTENT 3 INTRODUCTION 4 METHODOLOGY 6 Methodology description 6 Data collection 6 Data analysis 7 Data synthesis 8
Crowdsourcing in practice: exploratory interviews 9 RESULTS OF COMPREHENSIVE LITERATURE REVIEW 11
Descriptive analysis 11
Paper topic and innovation phase 13
SYNTHESIS OF EXISTING LITERATURE 15
Figure 5: Theoretical framework 15
Innovation phases 16
Roles of crowdsourcing during innovation 16 Problems of crowdsourcing during innovation 17 Advantages of crowdsourcing during innovation 19
CROWDSOURCING IN PRACTICE: FINDINGS 20
DISCUSSION 23 Crowdsourcing in general 23 Fragmentation 23 Ideation 24 Development 25 Testing 26 Marketing 27
LIMITATIONS, IMPLICATIONS AND FURTHER RESEARCH 28
Limitations 28
Managerial implications 28
Further research 29
SOURCE LIST 31
APPENDIX A — Roles of crowdsourcing 38
INTRODUCTION
A decade ago, Chesbrough (2006a) promoted the paradigm of Open Innovation by noting “companies can no longer afford to rely entirely on their own ideas to advance their business” (Chesbrough, 2006b, p. 2). Open Innovation is a paradigm that assumes that firms are able to use external ideas next to internal ideas, as well as both external and internal paths to market, as they try to be innovative regarding their technology (Chesbrough, 2003). Crowdsourcing is a part of Open Innovation and can be defined in the following way: “Crowdsourcing means outsourcing to the crowd” (Schenk and Guittard, 2011, p. 94). Since this article focuses on crowdsourcing for innovation, a more precise definition is: “Crowdsourcing [..] takes place when a profit oriented firm outsources specific tasks essential for the making or sale of its product to the general public (the crowd) in the form of an open call over the internet, with the intention of animating individuals to make a contribution to the firm's production process for free or for significantly less than that contribution is worth to the firm. Firms engage in crowdsourcing to inexpensively mobilize the creative work of sometimes highly skilled persons as a resource for the generation of value and profits” (Kleemann et al., 2008, p. 6). Crowdsourcing allows organizations to increase their direct communication with consumers, which provides access to the knowledge of a large and important knowledge source (Huang et al., 2014). The rise of crowdsourcing and the increasing use of the web 2.0 opens many opportunities for companies who want to exploit the power of the crowd. Crowdsourcing and Open Innovation can thus be attributed to the same paradigm: knowledge is appropriated, and the opening of a firm’s research and development (R&D) may turn into a source of competitive advantage (Schenk and Guittard, 2009). The main difference between crowdsourcing and Open Innovation is that Open Innovation focuses entirely on innovation practices, whereas crowdsourcing may be used for alternative processes as well (Schenk and Guittard, 2009). As such, crowdsourcing can be seen as a method within Open Innovation.
Previous studies have shown that crowdsourcing has potential to significantly add to innovation. Among others, Poetz and Schreier (2012) have found that users are able to develop new product ideas with higher relevancy as well as consumer benefit, compared to expert producers. Additionally, Füller et al. (2006), found that user participation in ideation and testing can enhance innovation within firms. Furthermore, companies may gain competitive advantage over others if they promote consumer participation within NPD (Fuchs and Schreier, 2011). All in all, it seems promising to investigate the functions of crowdsourcing, as it can be seen from previous literature that incentives exist to use crowdsourcing in the innovation practice.
few years, crowdsourcing has increasingly received researchers’ attention (Goncalves et
al., 2014). Although taxonomies of crowdsourcing exist, they either focus on one aspect of
crowdsourcing, such as task complexity (Nakatsu et al., 2014) or on specific crowdsourcing platforms (Brabham, 2012). This has led to a situation in which there exist a variety of definitions and taxonomies, which all look at crowdsourcing from a different point of view, existing literature can thus be said to be fragmented across several literature fields. On the one hand, existing work concerning problem solving focuses mainly on the early phases of innovation, such as innovation contests in which “a firm facing an innovation-related problem posts this problem to a population of independent agents and then provides an award to the agent that generated the best solution” (Terwiesch and Xu, 2008, p. 1529). On the other hand, existing literature on new product development (NPD) (e.g. Djelassi and Decoopman, 2013), crowdsourcing motivations (Brabham, 2010) and crowdsourcing in general (Schenk and Guittard, 2011) has been written from the point of view of all phases of the stage-gate model (Cooper, 1990).
Taking into account that available literature has increased substantially over the past few years, the literature field can benefit from a comprehensive literature review. Moreover, to my knowledge, no research yet exists that groups different forms that crowdsourcing may take during the phases of innovation and new product development. Additionally, Estelles-Arolas and González-Ladrón-de-Guevara (2012, p. 198) propose as an interesting avenue for further research: “it would be interesting to undertake a study of all the terms that are linked regularly with crowdsourcing to establish similarities and differences, with the objective of profiling the concept and defining a theoretical framework.“ Additionally, Hossain and Kauranen (2015) pose that future literature studies should delve into the pros and cons of crowdsourcing. This way, previous studies show that there yet remains much unclarity regarding the concept of crowdsourcing, and the ways it may be used for innovation practices by firms.
The focus in this research will thus be on identifying distinct forms of crowdsourcing during several phases of the innovation process. In doing so, the roles that the crowd may perform will be distinguished from existing literature. Moreover, problems and advantages of crowdsourcing will be identified with the objective to provide a theoretical overview consisting of the roles, problems and advantages that may occur during the use of crowdsourcing in a firm’s innovative efforts.
METHODOLOGY
Methodology description
This study will be written in the form of a comprehensive literature review. A comprehensive review uses a heuristic search process, as opposed to an explicit algorithm, to search for literature (Crossan and Apaydin, 2010). Since an unrestricted search in the Business Source Premier database of peer-reviewed journals containing the keyword crowdsourcing only produces 438 articles, and literature reviews as well as meta-analyses are rare, systematically reviewing literature may not yet add significantly to the field of crowdsourcing, mainly because existing work is fragmented across different fields of literature. A comprehensive literature review, however, may be effective since it does not require an algorithm in order to search for and select literature. The process of a comprehensive literature review generally consists of three sequential steps: data collection, data analysis, and data synthesis (Crossan and Apaydin, 2010).
Data collection
The data used in this study was collected by searching for various keywords in a single database. The database used in this study was the Business Source Premier database. Business Source Premier is the industry’s most used business research database, providing full text for more than 2,300 journals, including full text for more than 1,100 peer-reviewed titles. Keywords included in the initial search were ‘crowdsourcing’, and similar notations of the word, such as ‘crowd-sourcing’, or ‘crowd sourcing’. No criteria regarding the publication dates were given to the initial search, however it was found that all selected articles from the Business Source Premier database were published after 2010.
The initial consideration set of this paper includes peer-reviewed journal articles only from the chosen database. Peer-reviewed journals were deliberately chosen since they can be considered as a validated source of information (Podsakoff et al., 2005). Often, literature reviews select their dataset based on citations, as highly cited paper serve as a proxy for the accumulation of knowledge (Saha et al., 2003). Since many publications were published recently, papers have not yet had time to accumulate citations. Therefore, this review regards articles that are peer-reviewed as validated sources of knowledge. Keywords used in the initial search were ‘crowdsourcing’, ‘crowdsource’, and ‘crowdsourced’ and other notations of the phenomenon, as described above.
set of papers focused on individuals social and intellectual capital and how this affects, for instance, team performance (Dissanayake et al., 2015). As this is not the aim of this research, those papers were left out as well. Third, a substantial amount of papers focused on mobile sensing and the relation to crowdsourcing, and fourth, a sizable amount of papers focused on data collected from Amazon Turk (e.g. Daaly and Nataraajan, 2015), centering on specific facets of that website, or on semi-crowdsourcing websites such as Youtube and Wikipedia where the crowd contributes to the website, but does not directly deliver innovations.
After 400 papers were excluded, the researcher used a snowballing technique in order to obtain more relevant data (Van Aken et al., 2006). Snowball sampling entails that the references of key articles are used for identifying additional literature. This technique led to the discovery of another 27 peer-reviewed articles, except for Howe (2006; 2008). Although Howe’s articles (2006; 2008) are not peer-reviewed, the researcher felt that it was necessary to include them, since they have been cited 2,865 and 1,344 times respectively. A second reason to use snowball sampling was due to the fact that the data set consisted of all articles published after 2010. Using snowball sampling, this paper identified papers that were published from 2002 an onwards, leading to a more dispersed data set, including not only recent papers that are likely to be based on similar assumptions.
Data analysis
The goal of this paper is to provide a comprehensive analysis of the literature, and a conceptual overview, rather than an unequivocal descriptive consolidation. Methodologically, this paper is thus limited to descriptive methods, rather than empirical results in the analysis of the collected data. Since 41 out of 65 articles are either theory building articles, theoretical articles, or literature reviews, the nature of the data is rather qualitative. Therefore, a qualitative data analysis technique has been used to analyze the data. The chosen data analysis technique was pattern-matching and explanation building (Yin, 1994). Finding patterns in literature is not an exact science and therefore the goal is to find obvious matches and mismatches, in which “even an “eyeballing” technique is sufficiently convincing to draw a conclusion” (Yin, 1994, p. 110).
After the four phases of the innovation process during which crowdsourcing may be used emerged, the following task was to identify roles that crowdsourcing could take during each of the distinct phases. In doing this, all papers were read thoroughly, using the same process used while scanning the literature, to ensure a structured process and enable the researcher to retrace the information from all available literature.
Data synthesis
The synthesis of existing literature “is the primary value-added product of a review as it produces new knowledge based on thorough data collection and careful analysis” (Crossan and Apaydin, 2010, p. 1157). Based on the data collection and analysis described above, the researcher identified roles that crowdsourcing may take during the innovation process. Furthermore, the problems that may occur while using crowdsourcing in the innovation process were identified, as well as the advantages of using crowdsourcing. The identified roles, problems, and advantages were mapped onto a theoretical model, consisting of the sequential steps of the innovation process.
Using ‘open coding’ (Glaser and Strauss, 1967), where the researcher reads through the data to create tentative labels for gross matches, reading all literature in the data set lead to the identification of several roles within each of the distinct phases that emerged earlier. Using techniques from open coding principles (Neumann, 2003) categories were developed to group the identified roles of crowdsourcing within the distinct phases of crowdsourcing. For instance, papers that explored ideation contests (e.g. Khasragi and Aghaie, 2014) were linked to papers that focused on idea generating communities (Bayus, 2013) under a category ‘idea generation’ in the phase of ideation. Similarly, whereas Brabham (2012) notes that the crowdsourcing model is suitable for marketing and public relations, Djelassi and Decoopman (2013) note that crowdsourcing is both a marketing promotion tool as well as a process through which companies apply open innovation. Therefore, this research labeled these two authors, among others, in a category labeled ‘promotion’ in the phase of marketing. During the execution of the study, new categories were added when another role was identified, until the researcher found that no additional categories were necessary, in line with the open coding principle (Glaser and Strauss, 1967).
Consequently, the same analysis was performed for the identification of problems that may occur while using crowdsourcing in the innovation process, as well as the potential advantages of using crowdsourcing. Evidently, while re-reading the text, occasionally the researcher found data that was usable in earlier stages of the process. Since the research already had already distinguished categories of, for instance roles, it was manageable to incorporate new findings in the analysis.
concise. The theoretical model was developed after the researcher had categorized the roles, problems, and advantages of using crowdsourcing during innovation. Thereafter, tables were developed containing either a definition or example of the role, problem or advantage that was distinguished. Thereafter, a head counting technique was used: “The head counts are simply a matter of counting” (Slichter and Kraemmergaard, 2010, p. 491) e.g. how many different journals have published articles concerning the category in question. This way, it could be found whether categories of roles, problems and advantages were highly supported, generally supported or narrowly supported.
Crowdsourcing in practice: exploratory interviews
In addition to the extensive literature review, this study aims to explore crowdsourcing in practice. On the one hand, these exploratory interviews were performed in order to control literature, and find out whether certain aspects may be confirmed by these managers. On the other hand, interviewing managers from practice give this review a practical dimension, by obtaining relevant information that may lead to additional and useful insights, as well as interesting examples perhaps not yet described in literature.
Seven managers at six different crowdsourcing intermediaries were interviewed. Intermediaries were chosen because they generally will have more knowledge about crowdsourcing, as they have performed multiple crowdsourcing assignments for clients, as opposed to firms who mainly have crowdsourced single tasks to the crowd. This increased experience vis-a-vis firms that have only crowdsourced one or a few assignments leads to believe that there is more to obtain from managers at crowdsourcing intermediaries.
The researcher used a semi-structured interview scheme, which is best used when only interviewing someone once (Bernard, 1988). The semi-structured interview included open-ended questions, followed by probing questions (Brown and Eisenhardt, 1997). This provides the researcher the opportunity to follow relevant topics that initially do not fit to the scope of the research. Moreover, semi-structured interviews guarantee comparability across interviews (Yin, 1994). The topics that were discussed with the participants concerned the roles the crowd performed in the crowdsourcing activity, which problems the intermediary or their client firms came across and what the managers of the intermediaries saw as definite advantages of crowdsourcing.
Table 1: Firm description Firm Company description
Firm 1 Marketing-based firm that uses the crowd in marketing efforts, for example by harnessing their ideas, or lets the crowd interact intensively with the client to develop ambassadorship for the client.
Firm 2 Ideation-based firm that owns platform on which client organizations can pose questions and problems that the crowd may answer.
Firm 3 Marketing-based idea generation firm that uses the crowd to come up with new ideas for marketing campaigns.
Firm 4 Problem solving firm that uses crowd to provide data-insights to client firms.
Firm 5 Platform that uses the crowd to obtain ideas to improve the neighborhood.
RESULTS OF COMPREHENSIVE LITERATURE REVIEW
In this section a descriptive analysis of the initial consideration set of 438 articles will be provided, providing the reader with figures and tables that describe the consideration set and dataset. Additionally, this research will provide a breakdown of articles in the dataset by paper type, year of publishing and paper topic.
Descriptive analysis
Since 2002, the Business Source P r e m i e r d a t a b a s e h a s e n t e r e d publications that contain the word crowdsourcing or some annotation of the word. As can be seen from figure 1 on the right, initially, there only have been published a few articles a year about crowdsourcing. After Jeff Howe (2006) published his highly cited paper “the rise of crowdsourcing”, the database has seen a sharp increase in publications about crowdsourcing. From 2006 until 2013, the figure nearly doubled each year.
Although we chose not to focus exclusively on highly cited papers, twenty out of sixty-six articles already accumulated over 100 citations or more. Just six articles included in the dataset had not yet accumulated any citations, and all of those articles were published in the year 2015, thus having the least time to do so. Since some of the papers have accumulated over 1,000 citations, it does not make sense to include the average number of citations of the dataset , but it is notable to mention that the median of citations is 20 citations per paper.
In the dataset used for this research, the trend in Figure 2 is fairly similar to the trend seen in figure 1. At the very least, this tells us that the dataset seems representative considering the latest trends in the field, as the dataset consists of articles from several years, cumulatively adding few articles, however in the final years when more articles about crowdsourcing were published, the dataset included more articles of those years. 0 35 70 105 140 2002 2004 2006 2008 2010 2012 2014 Figure 1: Articles published per year in consideration set
Within the dataset, additional interesting findings were identified (Figure 3). First, among the dataset, several papers (8 per cent) could not be specified to belong to a certain type of paper. Furthermore, four literature reviews (8 per cent) are part of the dataset. Here, it must be noted that several of these literature reviews only concerned ‘defining a definition’ (Estelles-Arolas and González-Ladrón-de-Guevara, 2012), or were a taxonomy of crowdsourcing (Nakatsu et al., 2014). Moreover, because literature reviews are based solely on existing literature, no categories of roles, problems or advantages were based singly on these reviews, implying that these literature reviews only represent a supporting role in establishing said categories. Theoretical papers represented over a quarter of all papers (36 per cent). The second-largest share of papers were empirical papers, theory testing (29 per cent) having a slightly larger share then theory building papers (20 per cent).
This research also identified the key topic of each of the papers in the dataset. Seven main topics were identified, as can be seen in figure 4.
Figure 3: Paper type dataset
8% 8%
36% 29%
20%
Theory building Theory testing Theoretical Literature review Unspecified
Figure 4: paper topic dataset 2% 27% 15% 11% 29% 5% 12%
Problem solving Open Innovation Crowdsourcing Crowdsourcing work Motivations NPD
The largest amount of papers, nineteen in total (29 per cent) considered crowdsourcing in general — describing for instance, a typology or taxonomy of crowdsourcing (e.g. Colombo et al., 2013; Hosseini et al., 2014). The second largest group (27 per cent) consisted of articles focusing on the use of crowdsourcing in new product development (e.g. Djelassi and Decoopman, 2013; Fuchs and Schreier, 2011). Fifteen per cent of the papers specified the motivations of the crowd, whereas a total of eight papers (12 per cent) discussed the use of crowdsourcing in organizational problem solving (e.g. Afuah and Tucci, 2012; Chiu et al., 2014). Eleven per cent examined the crowdsourcing of micro tasks, for instance at amazon mechanical Turk and its use for innovation (Conley and Tosti-Kharas, 2014). Three papers (5 per cent) discussed open innovation, and the function of crowdsourcing within open innovation and one paper discussed crowdsourcing-based business models.
Paper topic and innovation phase
Similarly, this research attempted to identify for each paper the phase of new product development it described. Organizing papers by both the topic and the phase of new product development it described, a table was developed in which can be clearly shown which phases and topics have (not) accumulated most published articles. Moreover, the table clearly shows gaps in the existing literature, which will be discussed below.
Table 2: Topic and phase
Ideation Ideation and Development
Ideation and Marketing
Testing All stages Problem solving Lee et al., 2015; Piezunka and Dahlander, 2015; Terwiesch and Xu, 2008 Afuah and Tucci, 2012; Afuah and Tucci, 2013; Bloodgood, 2013; Chiu et al., 2014; Luttgens et al., 2014 Open innovation Baldwin and Von Hippel, 2011
From table 2 it can be seen that the use of crowdsourcing has not yet been described from the perspective of distinct and different phases in innovation or new product development and thus clear fragmentation. Whereas ideation, and ideation and development are most supported by existing literature, testing and marketing are less supported. Furthermore, although approximately twenty per cent of the literature describes all phases of innovation or new product development, it must be acknowledged that within these papers, there often still exists a focus on either ideation, ideation and development or ideation and marketing.
NPD Bayus, 2013; Martinez, 2015; Schlagwein and Andersen, 2014; Simula and Ahola, 2014; Wikhamn, 2013 Huang et al., 2014; Luo et al., 2015; Poetz and Schreier, 2012; Simula and Vuori, 2012 Malhotra and Majczchrak, 2014; Hoyer et al., 2010; Fuchs and Schreier, 2011 Djelassi and Decoopman, 2013; Mladenow et al., 2014; Boudreau and Lakhani, 2013; Dahan and Hauser, 2002; Fuller et al., 2006 Motivations Goncalves et al., 2015; Khasraghi and Aghaie, 2014; Leimeister et al., 2009; Moussawi and Koufaris, 2013; Zheng et al., 2011 Boons et al., 2015; Kosonen et al., 2015; Schulten and Schaefer, 2015; Howe, 2006b Brabham, 2010; Brabham, 2012 Crowdsourcing work Ren et al., 2014 Conley and Tosti-Kharas, 2014; Ford et al., 2015; Kleemann et al., 2008 Kittur et al., 2008 Barnes et al., 2015
Ideation Ideation and Development
Ideation and Marketing
SYNTHESIS OF EXISTING LITERATURE
The following part will present the theoretical model and how it should be interpreted, thereby synthesizing existing literature into one comprehensive overview. Should ambiguities exist, Appendix A, B and C propose explanations of constructs presented in the theoretical model. Additionally, it can be seen from the Appendices which authors support the statements presented.
Innovation phases
The next part presents the phases that were distinguished after analyzing the literature. Estelles-Arolas and González-Ladrón-de-Guevara (2012) conclude “that crowdsourcing will be a participative distributed online process that allows the undertaking of a task for the resolution of a problem”. However, Estelles-Arolas and González-Ladrón-de-Guevara (2012) do not elaborate on which tasks or which problems may be solved by the use of crowdsourcing. This paper is interested in finding out whether crowdsourcing can be used during different phases of the innovation process. In doing so, this research has distinguished within papers whether those papers discuss the use of crowdsourcing in a single-, or in multiple stage(s) of the innovation process. Some of the papers included in the dataset had already classified crowdsourcing platforms by function such as Vukovic (2009), distinguishes: (1) design and innovation, (2) development and testing, (3) marketing and sales, and (4) support. Further, Mladenow et al. (2014) distinguish between the stages of the well known stage-gate model formulated by Cooper (1990).
Other articles focused on innovation contests and idea generation, e.g. (Bayus, 2013; Colombo et al., 2013; Simula and Ahola, 2014). Some articles focused on both the generation of ideas, and the development of those ideas into concepts e.g. (Afuah and Tucci, 2012; Bloodgood, 2013; Ford et al., 2015; Poetz and Schreier, 2012). Only one article focused solely on the use of crowdsourcing in the testing stage of innovation (Kittur
et al., 2008). Surprisingly, none of the articles in the dataset focused exclusively on using
crowdsourcing during marketing processes. Then again, many of the articles in the dataset did consider crowdsourcing to be of use in marketing processes, but those articles also considered either ideation and marketing, or all stages to be of importance, (e.g. Boons et
al., 2015; Kosonen et al., 2015; Malhotra and Majzchrak, 2014; Hoyer et al., 2010). As can
be seen from the above analysis, crowdsourcing can clearly be used during four distinct phases of the innovation process. These phases have some overlap with Cooper’s (1990) stage-gate model and consist of 1) ideation, 2) development, 3), testing and 4) marketing. Roles of crowdsourcing during innovation
Ideation
Development
Throughout the phase of development, four roles were identified. Less literature exists that focuses on this stage of innovation. Therefore, categorized roles only have a few supporting authors per category, for example co-creation is only supported by Mladenow et al. (2014), Djelassi and Decoopman (2013) and Kohler (2015) (Appendix A). Again, the crowd may be engaged by providing feedback upon developed concepts (e.g. Chiu et al., 2014). Moreover, the company may ask customers to suggest product concepts (e.g. Djelassi and Decoopman, 2013). Through the use of internet, firms are increasingly enabled to reach their customers, in doing so, companies may provide customers with the means to develop new products for them through the internet (e.g. Nakatsu et al., 2014). Lastly, firms may also choose to open up their factory and develop new products alongside their customers in a co-creation process (e.g. Mladenow et al., 2014).
Testing
When solely looking at the available literature, existing authors generally do not deem crowdsourcing to be suitable for the testing phase of the innovation process. Only five out of sixty-five articles mention the possibility of using crowdsourcing for testing purposes. Three out of five of those articles mention the possibility to ‘outsource’ the testing work to the crowd by means of using the internet (e.g. Kittur et al., 2008), whereas the other two focus specifically on companies opening up their doors in order for consumers to test their products (e.g. Mladenow et al., 2014).
“The company opens its doors to consumers (from 50 to 200 people) on the first Thursday of each month to test products.” (Djelassi and Decoopman, 2013, p. 686)
Marketing
Conversely, marketing is supported by larger amounts of publications. Most prominently supported are promotion, during which the company asks its customers to help a hand in promoting the company, brand or new product (Boudreau and Lakhani, 2013), as well as advertisement, during which consumers develop marketing campaigns for firms (Howe, 2006). Similar to the ideation phase, several authors acknowledge how the crowd may be used to vote on, for example, advertisements in order to pick the one that suits best to the firm’s customer (e.g. Prpic et al., 2015).
Problems of crowdsourcing during innovation
Early phases
The ideation stage is most described in literature and thus has identified several problems that may occur during, for example, idea generation contests. On the one hand, “crowdsourcing relies on voluntary participation and the reaching of a “critical mass” of contributors cannot be guaranteed” (Schenk and Guittard, 2011, p. 103). On the other hand, when too many consumers generate ideas for the firm, a large part of those ideas can be expected to generate ‘noise’ that diverts attention from higher quality ideas (e.g. Simula and Vuori, 2012). Similarly, contributions of consumers may not be in line with the expectations of companies, as some consumers may be unable to express and transfer their needs for new ideas leading to a misrepresentation of needs and thus wrong products being brought to market (Howe, 2006). When consumers are able to articulate and pass on those needs “co-creation initiatives can require firms to grapple with tricky questions around the ownership of intellectual property” (Hoyer et al., 2010, p. 289). Furthermore, with large amounts of generated ideas, firms may come across problems in selecting among ideas (e.g. Piezunka and Dahlander, 2015).
One problem mentioned specifically during the development phase is the resource intensiveness of crowdsourcing. The process of crowdsourcing “requires additional commitment, contributions and capabilities of key individuals” (Lüttgens et al., 2014, p. 367).
Lastly, three problems may occur during both early phases. First of all, firms may come across internal resistance from middle managers who don’t understand why firms choose for external ideation and development (e.g. Spradlin, 2012). Second, consumers may feel as if they are exploited by firms, when rewards for participating in crowdsourcing are perceived to be unfair (e.g. Djelassi and Decoopman, 2013). Finally:
While user generated ideas score higher in terms of innovativeness, users score
“somewhat lower in terms of feasibility.” (Poetz and Schreier, 2012, p. 245)
Latter phases
In the latter phases, literature discriminates between three potential obstacles that firms should try to avoid while using the crowd in their innovation process. Similar to the ideation phase, it may be troublesome to attract sufficient contributors to the cause (Schenk and Guittard, 2011). Furthermore, Rosen (2012) and Brabham (2008a; 2008b; 2010) note that the crowd that participates in this phase of the innovation process may not be as diverse as the average consumer, leading to an unrepresentative crowd. Finally, crowds of consumers may generate noise, or even worse, send in unsuitable submissions, called ‘crowdslapping’ (e.g. Howe, 2006).
“since crowdsourcing also involves sharing sensitive market research information with the crowd that could be advantageous to competitors” (Mladenow et al., 2014, p. 84).
Advantages of crowdsourcing during innovation
In total, the literature recognizes seven advantages of using crowdsourcing during innovation. Some pertain to the early phases of innovation, whereas others pertain to all phases of innovation.
Early phases
One of these advantages is the access to an infinite pool of resources (e.g. Bayus, 2012). Another advantage is that the use of, for example crowdvoting, simplifies the decision making process for companies (e.g. Huang et al., 2014). During development, the company does not have to deal with the principal/agent problem any longer, since the company “only pays for the results, and not for the design process” (Mladenow et al., 2014, p. 82). Implied here is that during outsourcing, the agent’s product may turn out not to be as the principal expected it. Furthermore, the use of crowdsourcing increases competition (Maiolini and Naggi, 2014). Competition, in turn leads to higher quality (Porter, 1998). This does not just affect internal business, seeing as Boudreau and Lakhani (2012) also found that increasing the number of competitors in a crowdsourcing contest leads to an increased chance of an extreme-value contribution.
All phases
Increased customer loyalty is an advantage that firms may derive from the use of crowdsourcing during ideation, testing or marketing (e.g. Bauer and Gegenhuber, 2015). Finally, supported by many authors, crowdsourcing in any phase of the innovation process stimulates the quality of the products, as well as lowers cost, and increases efficiency (e.g. Schenk and Guittard, 2011; Poetz and Schreier, 2012; Mladenow et al., 2014).
“The advantages of crowdsourcing are that it gives firms access to a potentially huge amount of labor outside of the firm which can complete necessary tasks often in a fraction of the time and at a fraction of the cost than if the same activities were conducted in-house” (Whitla, 2009, p. 25).
CROWDSOURCING IN PRACTICE: FINDINGS
Crowdsourcing in general
An initial interesting finding is that in the Netherlands, it appears as if crowdsourcing has not yet reached its potential. Multiple firms that were interviewed acknowledged that they started out as crowdsourcing consultancy firms, but since demand was low, they had to shift their focus towards using the crowd in one specific phase of innovation, for instance ideation (firm 2) or marketing (firm 1). This finding was acknowledged by briefly speaking with a manager at Brandfighters, one of the largest Dutch crowdsourcing intermediaries for marketing efforts, who unfortunately would not participate in an interview. Brandfighters did not want to participate on account of their shift from a crowdsourcing intermediary towards a situation in which they could either crowdsource an assignment, or outsource someone in their pooled crowd to a specific assignment, thus somewhat taking over the role of an employment agency.
Roles
All identified roles crowdsourcing may take during ideation were used by one (or more) platforms that were spoken to. Idea generation was most present, since it was the core business of firm 2 and 3 and to some extent 5. Trendwatching was least present, only being performed by the crowd of firm 4, who according to the manager were the eyes and ears of the market for firms. Firm 3 made extensive use of evaluation and feedback of each other’s ideas, noting that associations from people with diverse backgrounds produce synergies and make the best ideas.
Surprisingly, firm 2 didn’t merely use their idea generation platform for just that, but also for testing assumptions.
“After we have developed a certain idea posed on the platform, we usually form
certain assumptions, for which we can pose an additional question on the platform to check whether we are right or not.” — Manager firm 2
Moreover, managers at firm 2 and 6 acknowledge that market research firms use crowdsourcing in reaching large masses of customers through consumer panels.
During development, three managers expressed doubts about firms being able to fully crowdsource that phase:
“At our firm, we believe in controlled involvement, where we, as experts, outline the
campaign, and the crowd can fill it in.” — Manager firm 1
Firms 2 and 6 agree, by specifying that for product development it is necessary to have a certain amount of specific knowledge, and therefore, firm 6 usually attracts ‘experts’ with certain types of knowledge that are intrinsically motivated to enlist.
especially looking at factors such as autonomy, meaningfulness and skill use (e.g. Moussawi and Koufaris, 2013; Zheng et al., 2011). Multiple interviewees noted that next to intrinsic and extrinsic motivations, the design of the crowdsourcing project is also highly important.
“A clear objective should be formulated, as it sets the mindset of the participants, and puts them on the same page.” — Manager firm 2
“It ensures that the crowd knows exactly where the principal wants to go [..] because you need to set boundaries, otherwise the focus will become too wide.” —
Manager 2, firm 3
Problems
Two problems that were identified in the literature were also mentioned by several interviewees. First, noise generation is a problem that most of the firms have had to deal with.
“Only one per cent of the ideas presented is actually a good idea that the company
can, and is able to do something with.” — Manager firm 2
“The creative process is all about quantity, the more ideas you get, the larger is the
chance that one of those ideas is great, but a fair amount of poor quality ideas also have to be assessed.” — Manager 1, firm 3
From these statements we may also imply another problem that was identified in literature, namely the resource intensiveness of crowdsourcing for innovation. Second, NIH resistance was a problem that managers at firm 1 and 3 warned for. Additionally, managers at company 1 and 5 both mentioned their awareness of crowdslapping, but commented that they had not yet seen it in practice. All firms that used crowds to propose ideas had their crowds sign non-disclosure agreements to prevent against IPR lawsuits.
Two additional problems were found by doing the exploratory interviews. These problems were not yet mentioned in the literature, and are therefore important findings of this review.
First, the manager at firm 1 has encountered a problem he calls saturation. Saturation occurs when a certain type of crowdsourcing project has been performed several times, and thus leads to a drop in the level of participation. This way, it can be seen as interrelated to the problem of participation.
“Duplication of crowdsourced activities has the effect that people are less likely to
be involved in the activity.” — Manager firm 1
“Lay’s maak de smaak received 700,000 submissions with flavors for their new
chips in their first edition. During their second run with the same campaign, they only received 300,000 submissions. Thereafter, McDonalds did the same thing with a new burger, and they, after investing heavily in it, only received 100,000 submissions from the crowd.” — Manager firm 1
Second, disappointing results are a problem that may result from the firm’s failure to motivate customer to gather on the platform. Firms should think about the motivation of the customer to respond to the problem, and promote the existence of the platform. Should disappointing results still occur, there may still be ways to save the campaign, because there has been participation from the crowd, possibly leading to increased loyalty or perhaps a better identification of consumer needs.
Advantages
Here, all but increased competition were mentioned by one or more managers at the crowdsourcing intermediaries, likely missing because it is of less concern to the intermediary. One notable finding from the interviews is the stress that managers placed on the ease of use of crowdsourcing. For example, one manager mentioned:
“The ideas are existing somewhere already, they just need to be picked up the right
way somehow.” — Manager firm 2
Ease of use was not mentioned in the literature, but can easily be brought under resource access, since crowdsourcing enables allows “access to large numbers of people to benefit from the wisdom of crowds” (Ford et al., 2015, p. 379). Another finding one manager noted is that by using external experts:
“Less group dynamics will form [..] Groupthink, for instance, is very harmful.” —
Manager firm 6
Groupthink can be defined as “a psychological drive for consensus at any cost that suppresses dissent and appraisal of alternatives in cohesive decision making groups” (Janis, 1972, p. 4). The manager believes that less group dynamics will form because individuals have no prior ties before working together on a crowdsourcing project.
DISCUSSION
Crowdsourcing in general
An initial point of discussion concerns the presumable relationship between the role that the crowd performs and the intensity of that role. Certainly, this review is not as generalizable as empirical theory testing, but it can be seen from the appendices that the more simple tasks (e.g. idea generation, evaluation/feedback or crowdvoting) generally have more supporting authors than the more labour or resource intensive tasks such as co-creation or (online) NPD (Appendix A). This may be linked to the problem that may occur during development, namely the resource intensiveness of crowdsourcing, explained below.
In line with avenues for further research from Estelles-Arolas and González-Ladrón-de-Guevara (2012), this review has identified roles, problems and advantages of crowdsourcing for innovation. Although Euchner (2013), Simula and Vuori (2012) and Aitamurto (2015) among others had already identified several problems related to the use of crowdsourcing, this review extends and aggregates 65 articles in the classification of problems and presents some of these findings in the light of practical examples obtained from conducting 7 exploratory interviews with crowdsourcing intermediaries.
51 out of 65 articles were written between 2011 and 2015 (Figure 2), thus it can be said that the field is still young and immature. Articles have only had several years to accumulate citations, and the median number of citations of the dataset is 20 citations per paper, therefore it can be said that the average knowledge contribution of the articles in the dataset is relatively high, with referral to Loebbecke and Leidner’s (2012, p. 432) statement: “citations act like an expert referral”.
Another point of discussion concerns the definition of crowdsourcing. The majority of articles in the dataset use Howe’s (2006) definition to explain the concept of crowdsourcing. Although this definition is sufficient to understand the concept, it does not take into account different points of view (Estelles-Arolas and González-Ladrón-de-Guevara, 2012). Moreover, while using this definition, much ambiguity remains as to who forms the crowd, what the crowd may do and how they get compensated. While Estelles-Arolas and González-Ladrón-de-Guevara (2012) have developed an all encompassing definition, many still use Howe’s (2006) definition, which seems rather underdeveloped compared to the former.
Fragmentation
sense that possibly could add to this work. Although some authors have written literature reviews on crowdsourcing (Estelles-Arolas and González-Ladrón-de-Guevara, 2012; Nakatsu et al., 2012), they mainly concern the search for an exhaustive definition of crowdsourcing, or the development of a classification of crowdsourcing based on characteristics of the process (Schenk and Guittard, 2011) or motivational reasons for participation in crowdsourcing (Doan et al., 2011). Therefore, a single overview containing all the roles, problems and advantages of crowdsourcing during innovation will contribute to the field by consolidating a great amount of knowledge on crowdsourcing into a theoretical framework. Further, although some selection bias may have occurred, this review has followed a transparant and rigorous review method, as well as performed a synthesis of existing literature.
As stated earlier, existing research is fragmented across several dimensions. Although nineteen papers in the dataset are empirical theory testing papers (Figure 3), within these papers there exists high variation among the topics discussed. For example, both Bayus (2013) and Martinez (2015) discuss the use of crowdsourcing in a new product development context, focusing on the initial phase of innovation, namely ideation. However, whereas Bayus (2013) discusses the quality of serial ideators’ ideas, Martinez (2015) explores solver engagement as a key determinant of the quality of solvers’ ideas. Alternatively, Lee et al. (2015) and Piezunka and Dahlander (2015) both examine the use of crowdsourcing in a problem solving context, within the ideation phase, but whereas Piezunka and Dahlander (2015) review how distant search affects the use of crowdsourcing, Lee et al. (2015) explore the feasibility of adopting crowdsourcing for problem solving, because of this fragmentation across dimensions, generalizability overall is difficult. Even when publications have the same topic — e.g. problem solving, and focus on the same phase — e.g. ideation, the scope of both papers may still be largely different, as can be seen from the example above. Since empirical papers on crowdsourcing focus on different topics, their findings are difficult to generalize, clearly explaining the lack of meta-analyses in the data as well as consideration set.
Ideation
Strikingly, for this stage of innovation all roles, problems and most advantages that were identified in the literature review were also mentioned by the managers during the interviews. From these two quotes, resource access, decision making and quality can be retrieved:
“Internally, firms will be enabled to identify customer needs, leading to higher
quality as the firm produces what the customer wants — thus leading a reduction of failed products on the market.” — Managers firm 1 and 2
the articles considered the ideation process in their paper, the findings of this stage are most developed. Still, only having executed seven practical interviews, this shows that the interviews generally succeed in confirming findings from the literature.
Development
Interestingly, no literature in the dataset focuses singly on the use of crowdsourcing in the development phase of innovation (table 2). Admitting that considerable research has been published on the use of crowdsourcing in the phases of ideation and development, or papers that discuss all phases of innovation, not one single publication in the dataset focuses singly on the development phase. It is possible that this is due to one of the problems that may occur during this specific phase, as is mentioned by Lüttgens et al. (2014, p. 367). It is possible that scarce research exists here because of the resource intensiveness of crowdsourcing during development, leading to a situation in which crowdsourcing simply does not occur frequently during development. Another explanation may be the occurrence of negative spillovers, more likely to arise during development, since the company has to open up their firm and share susceptible market information with the crowd that may be transferred to competitors. The likelihood of negative spillovers is greater during later stages of innovation, since the later stages require more two-way interaction between parties (Aitamurto, 2011). As seen in transaction cost economics, two-way interaction between parties is more intensive, thus the favored mode of governance is a joint venture (Chen and Chen, 2003). A final explanation could be that the use of the crowd while developing for innovation is better described in co-creation literature, as both are interchangeably used in literature (Chiu et al., 2014, p. 41).
A somewhat related point of discussion is concerns loyalty as an advantage of crowdsourcing. Loyalty is described as a definite advantage of crowdsourcing in the initial, and latter two phases of innovation, but not in development, where, as just stated: two-way interaction is higher, thus more likely also leading to loyalty. It could be that it is offset by the resource intensiveness in this phase, however further research is necessary to find out.
Additionally, although definitely described by some as roles of crowdsourcing (Appendix 1), others argue that there are some clear distinguishing factors between crowdsourcing and co-creation, as well as (online) NPD. Chesbrough and Brunswicker (2014, p. 20), for example, note that:
Co-creation is “the involvement of consumers in the generation, evaluation and testing of novel ideas for products” whereas they see crowdsourcing only as a tool for problem solving.
Some authors discuss the the actual occurrence of crowdsourcing during development. Kleemann et al. (2008, p. 11), for example discusses how “Calls by established firms for participation in the design or configuration of new products represent one of the most prevalent forms of crowdsourcing being used currently. These vary in intensity from simple opinion polls to elaborate schemes for the collaborative development of actual products by users”. Over the following paragraphs Kleemann et al. (2008) gives examples of ideation and designing practices, but does not describe any process that directly involves consumers’ developing products. Therefore, it seems as if involvement during the development stage occurs through design, but not actual physical development. Supplementary to this, Aitamurto (2011) debates the use of crowdsourcing during the development phase. Aitamurto (2011, p. 5) notes that in many cases crowdsourcing indicates one-way communication during which customers propose ideas or other relevant information to a specified task. On the contrary, Djelassi and Decoopman (2013) note that Michel and Augustin have opted for active participation of customers in their development process, having them come up with new product prototypes and concepts for packaging. In practice, the one manager questions the ability to use the general crowd for co-creation.
“Take for instance, AkzoNobel. You can’t ask the general crowd to develop a new
type of paint, because they will have no idea how to develop a new type of paint, you need to work together with experts to develop new types of paint.” — Manager
Firm 1
The contribution of crowdsourcing to development thus lies in obtaining designs and product concepts, of which actual development is not done by consumers, but rather in-house.
Testing
research firms use crowdsourcing for the (online) testing phase, by using consumer panels to reach large amounts of consumers willing to respond to surveys.
“Because these firms are able to do it at a lower price, we outsource these
projects to those firms.” — Manager firm 2
Marketing
Remarkably, in the dataset were no publications focusing specifically on using crowdsourcing in marketing efforts. This seems odd, having observed several examples of the use of crowdsourcing to give a boost to the marketing campaign. This absence may be explained by looking at the theoretical framework. “Marketing can be seen as relationship management: creating, developing and maintaining a network in which the firm thrives” (Gummesson, 1987, p. 11). During ideation, this paper identified, among others, idea generation, evaluation and feedback, and crowdvoting. All three of these roles that crowdsourcing takes during an innovation process enable a company to establish, develop and manage relationships with customers, that can easily be transferred to the marketing of these ideas, evaluations or votes. As pointed out a decade ago by Howe (2006) crowdsourcing initiatives driven by consumers are in essence marketing campaigns. Looking at the literature in our dataset, this claim seems legitimate, as all of our papers that elaborate on marketing efforts also describe the earlier ideation process. From a practical standpoint, this view is confirmed. The manager at firm 6 acknowledges that it is not just about receiving the customer’s ideas, but also about them:
“Becoming dedicated to the client firm and becoming ambassadors for their brand.” — Manager firm 6
LIMITATIONS, IMPLICATIONS AND FURTHER RESEARCH
Limitations
This study acknowledges several limitations. First, due to reasons of time and space the study only included the Business Source Premier database, however a rigorous snowballing approach (Van Aken et al., 2012) was performed in order to minimize selection bias. Consecutive research could choose to use several databases and perform a similar study. Second, several articles in this database addressed the concept of mobile crowdsourcing. Articles on this type of crowdsourcing were purposefully omitted. “Mobile crowdsourcing is an emerging technology that utilizes agent-participatory data for decision making” (Chen et al., 2015, p. 157). Although it can be related to both evaluation and feedback as well as trendwatching, as both could easily be performed by the use of a mobile device, those papers mainly focus on providing companies with real-time data instead of innovative ideas, which is not the scope of this paper. Third, it was also purposefully chosen to leave out articles focusing on semi-crowdsourcing efforts such as Youtube and Wikipedia. Although these semi-crowdsourcing websites may indirectly be beneficial for innovation, this study did not take it into account to ensure that the focus would stay on the roles of crowdsourcing in the innovation process, rather than indirect links between crowdsourcing and innovation efforts. Finally, because of the large dataset and the use of the eyeballing technique (Yin, 1994), it was attempted to develop connections that were either neglected or underdeveloped earlier, however, there is a possibility that the eyeballing technique may have failed to capture other connections.
Considering the explorative interviews that were performed next to the literature review, some limitations also have to be acknowledged. First, only 7 explorative interviews were performed. Because this number is relatively low, findings should be seen as experiential and tentative. Moreover, even though firms are all intermediaries, connecting clients with the crowd, some of them are far apart with regards to, for example the stage-gate model (Cooper, 1990). Findings should thus be looked at from an exploratory context. Second, as just stated this study did not include articles that addressed mobile crowdsourcing, however one of the participating interviewees was a manager at a company that provides data-insights through through the crowd (firm 4), which may seem contradictory. Therefore, no definitive conclusions were made based singly on that interview.
Managerial implications
Moreover, it may help firms become aware of potential problems they may come across during the deployment of the crowd. Next to the problems illustrated in the theoretical framework, according to the manager at firm 1, firms should be aware of saturation since it will lead to a lower rate of participation as seen in the previously mentioned Lay’s and McDonalds example. Firms shouldn’t simply copy a competitor’s strategy, but rather develop a crowdsourcing campaign that is original and ensures a high level of participation. As Xu et al. (2012) have already found that there exists a positive (though complicated) relationship between the use of crowdsourcing and firm performance, managers may also become aware of additional possible advantages of crowdsourcing for innovation. By being aware of which roles the crowd can perform, which problems may occur and what advantages may be derived from the process, a suitable crowdsourcing for innovation strategy can be developed that fits the firm’s need.
Another implication is the double observation of the resource intensiveness of crowdsourcing for innovation. On the one hand from how many authors supported a specific role, and thus how well-developed that role already is (appendix A), and on the other hand from the observed problem of resource intensiveness (appendix B). It seems as if simpler crowdsourced tasks are generally more supported than roles that are more intensive. Thus, from a practical standpoint, inexperienced firms could consider starting with relatively simple crowdsourcing tasks in the beginning of the innovation process. A good start could be crowdvoting between in-house developed products. Then, as they gain experience, firms can move on to more complex tasks, such as co-creation. Additionally, crowdsourcing these less complicated tasks may increase the level of participation, likely leading to increased loyalty.
Finally, already described in other articles (Moussawi and Koufaris, 2013; Zheng et
al., 2011) this study places importance on the design of crowdsourcing tasks for firms.
According to several managers from practice, the right objectives have to be set, because too broad a focus leads to an unclear goal and the quality of input from the crowd will decrease. According to managers from practice, it is more beneficial to focus on a few objectives, rather than several at the same time.
Further research
As noted, several articles in the dataset had identified problems that may occur while crowdsourcing for innovation (e.g. Aitamurto, 2015; Euchner, 2013 and Simula and Vuori, 2012). How these problems may be avoided, however is, to my best knowledge, not yet researched. A case study approach, similar to the one used by Djelassi and Decoopman (2013) may resolve this lack of research on how to avoid and/or remedy the problems of crowdsourcing for innovation.
Although categorized as a problem in this review (Appendix B), there has been given limited attention to problems with intellectual property rights (IPRs). Articles that do discuss problems with IPR focus their concern only in the ideation stage. Future work could focus on how to organize crowdsourcing, when to use it and when to avoid using it.
This research specifically chose to use crowdsourcing intermediaries for the exploratory interviews, because they were more likely to provide a broader knowledge base and thus additional insights to this review. Subsequent research could choose to use firms that have crowdsourced assignments to intermediaries to see whether these firms see different roles, problems and advantages of crowdsourcing for innovation.
One possible advantage, not yet described in literature was mentioned by the manager at firm 6. He stated that it is likely that less group dynamics will form in crowdsourced project groups, because group members do not get to know each other as traditional groups would. In this context, it might be interesting to find out how fast, for example groupthink (Janis, 1972) occurs in crowdsourced groups.
SOURCE LIST
Afuah, A., & Tucci, C. L. (2012). Crowdsourcing as a solution to distant search. Academy
of Management Review, 37(3), 355-375.
Afuah, A., & Tucci, C. L. (2013). Value capture and crowdsourcing. Academy of
Management Review, 38(3), 457-460.
Aitamurto, T., Leiponen, A., & Tee, R. (2011). The promise of idea crowdsourcing–benefits, contexts, limitations. Nokia Ideasproject White Paper.
van Aken, J., Berends, H., & Van der Bij, H. (2012). Problem solving in organizations: A
methodological handbook for business and management students. Cambridge
University Press.
Baldwin, C., & von Hippel, E. (2011). Modeling a paradigm shift: From producer innovation to user and open collaborative innovation. Organization Science, 22(6), 1399-1417. Barnes, S. A., Green, A., & Hoyos, M. (2015). Crowdsourcing and work: individual factors
and circumstances influencing employability. New Technology, Work and
Employment, 30(1), 16-31.
Bauer, R. M., & Gegenhuber, T. (2015). Crowdsourcing: Global search and the twisted roles of consumers and producers. Organization, 22(5), 661-681.
Bayus, B. L. (2013). Crowdsourcing new product ideas over time: An analysis of the Dell IdeaStorm community. Management Science, 59(1), 226-244.
Bernard, H. R. (1988). Research methods in cultural anthropology (p. 117). Newbury Park, CA: Sage.
Bloodgood, J. (2013). Crowdsourcing: Useful for problem solving, but what about value capture?. Academy of Management Review, 38(3), 455-457.
Boons, M., Stam, D., & Barkema, H. G. (2015). Feelings of Pride and Respect as Drivers of Ongoing Member Activity on Crowdsourcing Platforms. Journal of Management
Studies, 52(6), 717-741.
Boudreau, K. J., & Lakhani, K. R. (2013). Using the crowd as an innovation partner.
Harvard business review, 91(4), 60-69.
Brabham, D. C. (2008a). Crowdsourcing as a model for problem solving an introduction and cases. Convergence: the international journal of research into new media
technologies, 14(1), 75-90.
Brabham, D. C. (2008b). Moving the crowd at iStockphoto: The composition of the crowd and motivations for participation in a crowdsourcing application. First monday,
Brabham, D. C. (2010). Moving the crowd at Threadless: Motivations for participation in a crowdsourcing application. Information, Communication & Society, 13(8),
1122-1145.
Brabham, D. C. (2012). A model for leveraging online communities. The participatory
cultures handbook, 120.
Brown, S. L., & Eisenhardt, K. M. (1997). The art of continuous change: Linking complexity theory and time-paced evolution in relentlessly shifting organizations. Administrative
science quarterly, 1-34.
Chen, H., & Chen, T. J. (2003). Governance structures in strategic alliances: transaction cost versus resource-based perspective. Journal of World Business, 38(1), 1-14. Chen, P. Y., Cheng, S. M., Ting, P. S., Lien, C. W., & Chu, F. J. (2015). When
crowdsourcing meets mobile sensing: a social network perspective.
Communications Magazine, IEEE, 53(10), 157-163.
Chesbrough, H. (2003). The logic of open innovation: managing intellectual property.
California Management Review, 45(3), 33-58.
Chesbrough, H. (2006a). Open innovation: a new paradigm for understanding industrial innovation. Open innovation: Researching a new paradigm, 1-12.
Chesbrough, H. W. (2006b). Open innovation: The new imperative for creating and
profiting from technology. Harvard Business Press.
Chesbrough, H., & Brunswicker, S. (2014). A fad or a phenomenon?: The adoption of open innovation practices in large firms. Research-Technology Management, 57(2),
16-25.
Chiu, C. M., Liang, T. P., & Turban, E. (2014). What can crowdsourcing do for decision support?. Decision Support Systems, 65, 40-49.
Colombo, G., Buganza, T., Klanner, I. M., & Roiser, S. (2013). Crowdsourcing
intermediaries and problem typologies: an explorative study. International Journal of
Innovation Management, 17(02), 1350005.
Conley, C., & Tosti-Kharas, J. (2014). Crowdsourcing content analysis for managerial research. Management Decision, 52(4), 675-688.
Cooper, R. G. (1990). Stage-gate systems: a new tool for managing new products.
Business horizons, 33(3), 44-54.
Crossan, M. M., & Apaydin, M. (2010). A multi‐dimensional framework of organizational innovation: A systematic review of the literature. Journal of management studies,
Dahan, E., & Hauser, J. R. (2002). The virtual customer. Journal of Product Innovation
Management, 19(5), 332-353.
Daly, T. M., & Nataraajan, R. (2015). Swapping bricks for clicks: Crowdsourcing longitudinal data on Amazon Turk. Journal of Business Research, 68(12), 2603-2609.
Dissanayake, I., Zhang, J., & Gu, B. (2015). Task Division for Team Success in
Crowdsourcing Contests: Resource Allocation and Alignment Effects. Journal of
Management Information Systems, 32(2), 8-39.
Djelassi, S., & Decoopman, I. (2013). Customers' participation in product development through crowdsourcing: Issues and implications. Industrial Marketing Management,
42(5), 683-692.
Doan, A., Ramakrishnan, R., & Halevy, A. Y. (2011). Crowdsourcing systems on the world-wide web. Communications of the ACM, 54(4), 86-96.
Dolan, R. J., & Matthews, J. M. (1993). Maximizing the utility of customer product testing: beta test design and management. Journal of Product Innovation Management,
10(4), 318-330.
Estellés-Arolas, E., & González-Ladrón-de-Guevara, F. (2012). Towards an integrated crowdsourcing definition. Journal of Information science, 38(2), 189-200.
Euchner, J. (2013). The uses and risks of open innovation. Research-Technology
Management, 56(3), 49-54.
Ford, R. C., Richard, B., & Ciuchta, M. P. (2015). Crowdsourcing: A new way of employing non-employees?. Business Horizons, 58(4), 377-388.
Fuchs, C., & Schreier, M. (2011). Customer empowerment in new product development.
Journal of Product Innovation Management, 28(1), 17-32.
Füller, J., Bartl, M., Ernst, H., & Mühlbacher, H. (2006). Community based innovation: How to integrate members of virtual communities into new product development.
Electronic Commerce Research, 6(1), 57-73.
Glaser, B. S., & Strauss, A. (1968). A.(1967). The discovery of grounded theory. Strategies
for qualitative research. London: Weidenfeld and Nicolson.
Goncalves, J., Hosio, S., Rogstadius, J., Karapanos, E., & Kostakos, V. (2015). Motivating participation and improving quality of contribution in ubiquitous crowdsourcing.
Computer Networks, 90, 34-48.
