M ODDING S UCCESS

(1)

M

ODDING

S

UCCESS

Does the openness of toolkits improve the quality of video game mods?

January, 2017

University of Groningen Faculty of Economics and Business

MSc BA Strategic Innovation Management Master thesis

(2)

Abstract

This paper investigates the influence of toolkits on user generated content in the PC video game industry within an open innovation context. Moreover, it analyses the effect of support in a moderating role. Data was gathered via an online survey distributed among mod makers (user innovators). No evidence was found that openness influences the quality of mods. Results suggest that mod team quality plays a significant role in determining the quality of a mod directly. This study also found that despite its negative direct impact, official support strengthens the positive influence of mod team quality. Support from a game developer significantly enhanced the positive effect of high quality mod teams on mod quality. However, this effect does not exist with low quality mod teams. Video game developers are recommended to create a supporting infrastructure and facilitate mod teams without interfering with the user content creation process itself.

Keywords: Open innovation, toolkits, video game industry, user generated content, modding Wordcount: 10.972

Acknowledgements:

(3)

1 INTRODUCTION

Products and services are no longer always developed in isolation, but more often with comprehensive consumer input. Organizations increasingly look towards these consumers in helping develop the product (von Hippel, 1986; von Hippel, 2005). Through concepts such as open innovation (Chesbrough, 2003) and co-creation (Prahalad & Ramaswamy, 2000) firms look outward and seek to capitalize on consumer knowledge and innovation in hopes of identifying, and subsequently capitalizing on market demands (Pisano & Teece, 2007).

In the video games industry, this practice is known as ‘modding’. It is the result of innovative users being giving the means to generate their own content, specific to cater to their needs and interests. User innovations created through modding are called mods, a generalized term for all forms of user edits that alter or add content to the original game. Users who engage in the practice of modding are called modders.

Through the use of toolkits organizations seek to enable users with the capability to innovate new products or services (von Hippel & Katz, 2002; Jeppesen, 2005). This enables manufacturers to capture innovation from outside the firm (Jeppesen & Molin, 2003). User communities can be considered strategic assets for stimulating innovation outside the firm. One of the greatest advantages of outsourcing innovation is the ability to capture specific information pertaining to the needs of the market through users as opposed to in-house market research which is susceptible to imperfect knowledge transfer due to specific user knowledge getting lost in translation (von Hippel, 1994).

Within the software industry, it is perhaps open source software (OSS) which is most often viewed through an open innovation lens (West & Gallagher, 2006). While some similarities can be observed, modding and mods are a distinct concept from OSS. The latter being defined as an ongoing project with multiple contributors, whereas the former generally follows a more discrete project structure with a well-defined end state. During the mid-2000s user generated content saw a massive surge with web 2.0 related applications, thus further spurring academic interest in user innovators and their motivations (von Krogh et al., 2012). Thus, while open source software has been researched extensively, open source in the context of video games only has a comparatively limited amount of peer reviewed literature dedicated to it (Arakji & Lang, 2007; Prügl & Schreier, 2006; Parmentier & Gandia, 2013; Koch & Bierbamer, 2016).

(5)

This paper investigates how video game developers through the openness characteristics of their toolkits can offer modders the means to create better user generated content. Additionally, it examines how modders themselves contribute to the quality of the user generated content. It draws on innovation management literature and the prevalence of toolkits in open innovation to determine this relationship. Accordingly, the following research question was formulated.

• Does the openness of the toolkits improve the quality of video game mods and what role can

developers play in supporting modders with taking advantage of said openness?

There are a number of instances where mods have superseded the original game in terms popularity. Most notably Counter Strike, a total conversion modification developed by users for the game Half Life (Postigo, 2007). Nevertheless, the overall majority of mods tend to be smaller in size and scope intended to marginally improve the base game (Arakji & Lang, 2007; Postigo, 2007). Thus, from a managerial perspective, investigating the effect of modders on the quality of video games as measured through the quality of user generated content produced will offer publishers and developers insight in how to increase product quality.

Moreover, another way of capitalizing on this open source protentional of modding is appropriation of the mod IP. Jeppesen (2004) identified two unique strategies with regard to user created content. Developers can choose to appropriate these mods or adopt a “laissez faire”, hands off approach. Tens of thousands of smaller user modifications exist for download on sites such as ModDB and Nexusmods. Currently, most developers choose the latter for dealing with the majority of modifications. It is worth noting however, that both strategies can be run concurrently.

Perhaps the most striking example of value appropriation with regard to NPD was the inception of an entirely new genre within real time strategy games. Massive online battle arena’s, or MOBA’s originated from toolkits and game developers partially opening up their game content to end users. Its immense popularity led to the publisher Valve acquiring all intellectual property rights pertaining to the DOTA modification. Interestingly the mod itself was built on Warcraft III, property of Blizzard, a competitor of Valve (Arakji & Lang, 2007; Baldwin, et al., 2006).

(6)

It stands to reason that a video game with a large and creative community will enhance the value of the brand. Thus, from a managerial perspective, investigating how to better support the modding community through improved toolkits will offer publishers and developers insight in how to effectively harness user generated content to complete their set objectives.

While there is currently some peer reviewed empirical research on open innovation in the video game industry, no real concentrated efforts have been made on linking openness of the original game with the quality of user innovations that can be spawned by an active modding scene. In other words, there is currently a gap of knowledge with how open exactly open source models and toolkits can contribute to a significantly better mod product.

The remainder of this paper is structured as follows. A theoretical background on relevant literature detailing the appropriate concepts related to open innovation, the video game industry and user generated content. Following this section is the research method, expanding on the methodology used in the study. Next, a section on the results of the analyses. The paper ends with a conclusion and discussion.

This research will therefore focus on the pc video game industry from 2002 -2016, as the eighth-generation consoles do not adequately support modding.

Modding games Manufacturing games

Publisher Developer Base game Modders Mods Toolkits

Community sites

Figure 1 Study framework

(7)

2 LITERATURE REVIEW

2.1 O

PEN INNOVATION

2.1.1 OPEN INNOVATION IN THE VIDEO GAME INDUSTRY

Ever since Henry Chesbrough championed his idea of open innovation a growing body of scientific literature spanning a variety of industries expanded upon the concept and implementation of this new paradigm (Chesbrough, 2003; Gassmann et al., 2010). It can be contrasted with five earlier innovation models.

Open innovation is characterized by an increased interaction between producers and consumers and specifically increased integration between collaborating companies. Chesbrough (2006) defines the concept as follows: “… the use of purposive inflows and outflows of knowledge to accelerate internal innovation, and expand the markets for external use of innovation, respectively”. One of the hallmarks of open innovation is the role of users as innovators and their proactive role in the innovation process (Bogers et al., 2010). Modding and modders can be seen as an extension of the user innovator concept within open innovation and can be defined as lead users, who foresee needs and actively change the product (von Hippel, 1986; Franke & Shah, 2003).

Related to the concept of open innovation is co-creation, a term popularized by Prahalad and Ramaswamy (2000). Similar to open innovation, co-creation puts great emphasis on customers being actively involved in the development process, providing innovation and ultimately adding value to the product or service. As such, co-creation is limited to the producer-customer relationship only, disregarding the broader framework of other stakeholders being involved in the innovation process and the formation of a community. Moreover, open innovation has a business model formed around it (Chesbrough, 2003). Although co-creation is a less comprehensive concept than open innovation, its inclusion is warranted due to its prevalence with regard to modding in the literature.

(8)

2.1.2 OPEN SOURCE INNOVATION

Open source is a subset of the concept of open innovation (von Krogh & von Hippel, 2006). Open source software (OSS) is defined as “software where users can inspect the source code, modify it, and redistribute modified or unmodified versions for others to use” (von Krogh et al., 2012). OSS can thus be placed within the open innovation framework as it involves several of its key tenants (i.e. user perspective). One important distinction between OSS and modding for video games lies in the redistribution of user innovations. Video game modding does not necessarily follow the redistribution of user content (i.e. any form of intellectual property). More often than not video game modders will want to retain ownership of their innovations. Nevertheless, some OSS studies have opted to include modding within their general framework. West and Gallagher (2006), identify three specific practices from OSS which also induce user innovation in a video game setting. These are (1) minimizing technical obstacles, (2) creating an infrastructure and (3) recognition for the contributors. While both OSS and modding rely on openness one distinct difference is that game modding itself is often closed. The development of mods itself is a closed process with modders being protective of their IP (Postigo, 2007; Postigo, 2008; Poor, 2014).

Baldwin and von Hippel (2011) argue that both open collaborative innovation and innovation by individual users will continue to increase in popularity and has the potential to dislodge the traditional producer innovation model in multiple industries. Indeed, in the videogame sector, developers seeking profit maximization are better off adopting open source innovation (Arakji & Lang, 2007).

2.2 M

ODS

Generally, the content produced by end users using toolkits within a video game context are called modifications or mods for short. Additionally, these user creators are often referred to as modders. Mods range from minor modifications to radically different games. Arakji and Lang (2007) differentiate between two distinct types; (1) partial conversion mods and (2) total conversion mods. The former is characterized by alterations to the game’s media files whereas the latter generally leads to the creation of entirely new IPs (games)2_{. Unofficial patches can be included under this definition as well. These generally entail user}

produced updates with bug fixes or other forms of optimization.

2.3 T

OOLKITS

One of the important aspects of open innovation is increased collaboration with the end users (Chesbrough, 2003). This can be accomplished through a variety of options such as lead users. In the software industry, this increased need for interaction is satisfied through the development and availability of so-called toolkits (von Hippel, 2001; von Hippel & Katz, 2002). Instead of focusing on understanding detailed customer needs,

(9)

toolkits are a measure for developers of outsourcing the collection of customer needs data, effectively repartitioning the development tasks. Parmentier and Gandia (2013) consider toolkits as a means of managing the boundary between the developers and users (Parmentier & Gandia, 2013; Bogers, Afuah, & Bettina, 2010). As a result toolkits can be considered strategic assets useful in maintaining and sustaining a business model focused on the user innovator. In order to be considered successful, von Hippel and Katz (2002) suggested that toolkits should adhere to the following five criteria: (1) Learning by doing trial and error, (2) Appropriate solution space, (3) User friendly toolkits, (4) Module libraries, (5) Translating user designs for production.

While the literature generally considers any set of tools enabling the user to create custom content as a toolkit (von Hippel, 2001; von Hippel & Katz, 2002; Parmentier & Gandia, 2013), for the purposes of this study it is necessary to further delineate what constitutes a toolkit with regard to mods. Therefore, while technically fitting the definition of a toolkit, level editors and mission editors specifically will not be considered toolkits in the context of this study. The reason being that user generated content produced with these editors is extremely limited in scope. Within a video game context toolkits are more commonly known as modding tools.

Von Hippel (2001) emphasizes the concept of sticky information in the context of toolkits and describes it as the "incremental expenditure required to transfer that unit form one place to another in a form usable by a given information seeker" (p. 248). Use of toolkits increases overall product development speed for two reasons. Firstly, it eliminates the need for producers themselves to obtain sticky information from the consumer, an inherently complex and expensive task. Secondly, it removes in its entirety the process of feedback and improving of the product (trial and error) by consumer and producer respectively. In effect, end users themselves are given the task of identifying their needs and means of fulfilling them. Furthermore, von Hippel and Katz (2002) also found that toolkits allowed for lower cost product development. Although, they do not consider the costs of continuously upgrading and supporting toolkits.

(10)

Toolkit software packages are building blocks with which end users can develop custom software suited to their own specific needs. End users are given the opportunity to innovate and design through iterative trial and error. Developers have to decide how much of their source code they will make available with regard to open source strategy. While generally the paradigm subscribes increased collaboration to facilitate knowledge bleeding over within the network, or innovation ecosystem (Chesbrough, 2003), full access to intellectual property rights will prohibit developers from appropriating the innovation and acquiring return on investment.

In a study conducted on Apache server software, Franke and von Hippel (2003) reveal that a high heterogeneity of need exists in server software. Moreover, users modifying their own software were found to be significantly more satisfied than non-innovating users. Additional findings include that the user costs associated with using toolkits cannot exceed expected benefits. Customer satisfaction is especially applicable in the video game industry as an active modding community can be a unique selling point for video games (Heineman, 2015).

Toolkits thus offer a means with which openness of video games can be measured. They allow developers to dictate the level of access available to the users. When a developer increases the openness of its toolkit for a particular game, a modder will have greater freedom in translating his own needs into custom developed content. A less restrictive, and thus higher quality toolkit will likely have a positive relationship with mod quality.

This openness differs between toolkits and influences the potential to improve the quality of mods. The greater this level of openness, the more flexibility it provides to the modders. Also, more modders are attracted leading to greater opportunity for improving mod quality. This results in the following hypothesis:

(11)

2.4 M

ODDER QUALITY

Modders also fit open innovation’s concept of organizations not being able to acquire all skilled personnel, highlighting the fact that this resource should be accessible through the innovation community network rather than an in-house asset.

Specific studies regarding modders are limited. While multiple studies (Postigo, 2007; Postigo, 2008; Poor, 2014) on modder characteristics and motivations have been conducted, very little literature exists with regard to open innovation or specifically successful exploitation of modding efforts within a value appropriation framework. Most research is limited to generalized open innovation concepts within the software industry or on the broader concept of communities in the video game industry (Burger-Helmchen & Cohendet, 2011). To date, three specific studies on modders, open innovation and the use of toolkits of note were conducted by Parmentier and Gandie (2013), Prügl and Schreier (2006) and Koch and Bierbamer (2016).

Modders often work in ad hoc teams, spanning geographical boundaries. Generally smaller mod projects are unstructured and lack formal planning. With increasing project size and scope (i.e. total conversions) mod teams become more formal and structured mod teams (Jeppesen, 2004). In fact, the organization behind total conversion mods tend to exhibit similar levels of professionalism as for-profit developers, following the same rigorous development practices (Nieborg & van der Graaf, 2008). Individual modders cover a large group of ages and levels of education, have experience working on multiple mod projects and generally have a strong sense of community with fellow modders. This heterogeneity within the group can help foster innovation given the diverse nature of the mod teams. Moreover, most evidence suggests modding is primarily driven by fun with a sense of community and helping others being significant factors (Arakji & Lang, 2007; Poor, 2014). This is contrasted by the motivations of OSS developers somewhat, who place more emphasis on demonstrating skills with regard to finding job opportunities (Lerner & Tirole, 2002).

(12)

Thus, when examining the literature on modders we can surmise that larger mod teams tend to display for profit development professionalism. Having access to a diverse group of individuals will allow for more knowledge dissemination, but more importantly it grants increased manpower (Alba & Hutchinson, 1987) Experience builds up as modders are active in multiple projects for a longer period of time. This leads to the following hypothesis:

H2: Higher quality mod teams will create mods of higher quality due to an increased level of competency in managing and executing modding projects.

2.5 S

UPPORT

Developers and publishers can try to actively support their communities. Within the context of toolkit quality and modders, is it the significance of the level of support which is relevant. While von Hippel and Katz (2002) clearly list a number of criteria for successful toolkits, external influences are ignored. As established earlier, modders form tight communities and are willing to offer assistance to each other (Burger-Helmchen & Cohendet, 2011; Poor, 2014; Postigo, 2008). Transfer of knowledge regarding effective toolkit usage can therefore easily be disseminated within such a network. Support can thus be categorized into official developer support and community support. The former can include developers and community managers engaging with the modding community via forums, workshops or other interactive events, whereas the latter includes the general user community that exists around a particular game.

This definition however lacks the inclusion of third party software. It differs from community support in that it specially refers to a software package used to complement or replace official toolkits. As a result, three distinct support categories can be formulated (1) official support, (2) community support and (3) third party support.

(13)

Official toolkits are neither mandatory or pervasive in modding. In fact, the majority of game developers do not officially support modding through a toolkit. This does not necessarily mean modding for these games is impossible or even discouraged. Depending on the game engine, developers can offer a level of openness rivaling or exceeding that of games with an officially released toolkit. Modders will often use third party mod tools to create content. What third party tools might lack in capability, they might make up in ease of use. Third party tools are developed from within the modding community, which allows access to what von Hippel (1994) describes as sticky information.

The three support types are hypothesized to positively interact on the relationship between our two main effects, toolkit openness and mod team quality and the quality of the mod. Accordingly, the following two hypotheses can be derived from this:

H3: The positive effect of toolkit openness on mod quality will be strengthened by higher levels of (a) higher levels of official support, (b) general community support and (c) third party support.

(14)

2.6 Conceptual model

These four hypotheses have been conceptualized in a framework depicted in figure 2. The model captures the two main effects of toolkit openness and mod team quality on mod quality. Support is defined as a moderator and the model denotes its possible interaction effects with the two main effects. Control variables are included to limit confounding effects.

Toolkit openness

Mod team quality

Community support

Official support Third party support Mod quality H1: +

H2: +

H3a: + H3b: + H3c: +

H4a: + H4b: + H4c: +

Control variables

Figure 2 Conceptual model

3 METHODOLOGY

Through use of a hierarchical multiple regression analysis we intend to investigate the relationship between the openness of toolkits and the quality of a mod. A major issue in this approach is the lack of literature on operationalizing the openness of the toolkit. In order to overcome this issue, we had to enlarge our scope and adopt a more generalized look on innovation literature. This approach yielded the result of a psychometric tool (KEYS) meant for assessing innovation in the workplace (Amabile et al., 1996). As the information regarding openness of the toolkit is not readily accessible in the form of an online database, these metrics will have to be obtained through a custom designed online survey. Several expert modders3_{were enlisted to}

help validate constructs based on existing literature. Von Hippel (2001) and von Hippel and Katz’(2002) concept of toolkits was specifically applied on a videogame framework. More formal reliability testing took place after collecting survey response data. Reliability of the constructs was estimated using Cronbach’s alpha. All hypothesized constructs scored adequately.

3_{("pogchamp", personal communication, December 6, 2016)}

(15)

3.1 Data collection & design

An online survey was distributed and ran for a period of two months. It was promoted on multiple websites which had large numbers of modders visiting. Moreover, respondents were encouraged to actively distribute the survey to team members or other modders in the general community. Respondents were asked to identify a mod they created with a good toolkit and one they created with a bad toolkit, mirroring the approach of Amabile et al. (1996).

Development of this questionnaire was based on existing literature on literature concerning modding. Openness pertaining to the toolkit was operationalized by initially using von Hippel and Katz’ (2002) five criteria. These were then submitted to several modders to determine both suitability and clarity. Once it was established these metrics could be presented clearly and interpretation would not be subject to errors, a preliminary survey was developed. This variant was then reviewed by a small group of expert modders and after minor adjustments was rolled out. Based on this initial feedback our total number of questions measuring the attitudes with regard to toolkits, mod teams and support was reduced from twenty-five to eighteen

Metrics were primarily measured using a five point Likert scale. While this data is ordinal in nature, it can be treated as interval with the associated distributional properties. Some questions were inverted to guard against response bias (i.e. halo effects, acquiescence bias, central tendency bias and leniency bias).

Respondents were presented with three distinct sections, namely a general introduction aimed at obtaining general demographical data, a section devoted to measuring their various attitudes towards the variables of interest and finally a section where the respondent had the opportunity to specify a mod they created with a good toolkit and a mod they created with a bad toolkit. It was assumed modders on average will have worked on multiple mods (Poor, 2014).

(16)

3.2 M

EASURES

3.2.1 DEPENDENT VARIABLE: MOD QUALITY

Product quality can be determined by proxy through expert scores metrics. This is a valid method used in the film industry (Elliott & Simmons, 2008; Eliashberg & Shugan, 1997) and has been used for studies involving the video game industry (Koch & Bierbamer, 2016). Video game reviews and aggregated scores are often codified on web resources such as Metacritic and GameRankings. Unfortunately, video game mods scores are not collected in a similar fashion, mostly due to the lack of a sufficient number of expert scores. Only the largest and most ambitious mods will likely attract professional critique in sufficient numbers.

While not exhaustive, ModDB does offer user scores for the more elaborate mods. There could be a risk of bias in these scores as there is no real control mechanism for rating mods other than possessing a free account. When ModDB scores are not available, secondary sources will be used. These include steam workshop ratings and ratings found on dedicated modding websites. This data will then be normalized in order to fit the ModDB scores. Lastly, if both ModDB scores as well as secondary sources are unavailable, the study will use scores indicated by the respondents themselves. While self-rating is an inherently biased method, its inclusion will be a necessity due to large number of smaller, less known mods.

3.2.2 INDEPENDENT VARIABLES

In order to ascertain the openness of the toolkit this research will adapt the general framework of the KEYS tool by integrating it with von Hippel’s literature on open innovation toolkits and supplemented by Parmetier and Gandia (2013). Our independent variables can be divided into the following three categories (1) toolkit openness, (2) modder quality and (3) Support. Additionally, a set of control variables will be listed. Our three constructs are listed in table 1.

Constructs

Toolkit Openness Control Features

Modder Quality Modder quality

Support Official support Community support Third party support

(17)

TOOLKIT OPENNESS

Our five items have been categorized into two variables, namely control and feature related. Construct Control

Learning by doing trial and error Enables users to carry out complete cycles of trial and error learning, thus

eliminating the costly process of having designs go back and forth between developer and user. Well-designed toolkits allow the user to have full control over the testing and implementing of new features. Within the context of video game mods this means modders are free to design and test their content on their own.

Appropriate solution space Solution space is the degree to which the user has the freedom to create the

designs they want to create. In digital applications, such as video games, this will necessarily be less strict compared to physical mediums. However, toolkits can offer restrictions in what is actually modifiable with regard to video game content. For example, a large solution space would typically entail opening up significant parts of the game engine, effectively giving modders the ability to create a radically different product (Arakji & Lang, 2007).

Construct Features

User friendly toolkits When users are able to successfully operate toolkits with a well-practiced design

language, the toolkit can be deemed user friendly. In video game design this can refer to a computer programming language (e.g. C, Java, Python, C++). Additionally, factors such as graphical user interface, tutorials, tooltips and terminology can be included in this variable.

Adequate documentation Depending on the complexity of the toolkit the proper documentation on how to

make effective use of its features can be decisive in its adoption. An interviewee reported lack of adequate documentation as an important factor in adopting a toolkit.

Translating user designs for production Compatibility ensures that user products can be used on the

producer’s production equipment without requiring revision by the latter’s engineers. Due to the lack of a physical production element, this factor is slightly different for video game mods. However, user generated content such as mods or plugins will have to be compatible with the game architecture. An example could be using standardized (proprietary) file formats.

Teamwork features Teamwork was a significant attribute of toolkits which was not covered by von Hippel

(18)

MODDER QUALITY

The following four items from our modder quality variable.

Team experience Another team related variable is the experience of the modding team. Teams with

knowledgeable individuals will lead to higher mod quality.

Team technical literacy Technical literacy encompasses the modder’s proficiency in programming languages

and general skills with software. To some extent this fits with Caves’ (2000) theory on the creative industries and the Motley crew principle (Caves, 2000).

Team creativity Creativity is an important factor in the ability to create high quality and unique products.

Teams who score highly on this dimension will more likely produce high quality mods.

Team commitment Due to voluntary nature of the modding process, modding teams will often struggle to

retain personnel. Members might leave due to lack of motivation or internal conflicts (Poor, 2014).

SUPPORT

Support includes the variables we hypothesized to act as moderators with regard to the two main effects of toolkit openness and modder quality. In total three sets of variables were defined based on literature and interviews with several expert modders: official support, community support and third party support. Construct Official support

Developers While the primary means of fostering and incentivizing user innovation is done through the

release and support of a toolkit, developers can also support the modding community through other means. Research has shown the importance of modder community interaction, both with the developers and among themselves (Nieborg & van der Graaf, 2008; Poor, 2014; Burger-Helmchen & Cohendet, 2011).

Community managers Often hired by the publisher, community managers serve an important role as a

dedicated to link to the gaming community. Community interaction is not necessarily suited to developers. A designated person can greatly influence the relationship between the developer and or publisher with the general gaming community.

Construct Community support

Modding community Support can also come from other modders or mod teams. Due to the sense of

(19)

General gaming community Regular fans of a particular game are classified as the general gaming

community. While they lack the technical expertise to assist modders with development issues, a supportive community can greatly motivate and influence modder disposition. Conversely, a negative community will have a negative influence on modders.

Module libraries If a product or service has a large user base engaged in creating custom content the

probability of truly unique innovations decreases significantly. Module libraries refers to a database of commonly featured designs (or elements thereof) which can be used by other users. Thus, when translated to a video game context this could encompass a selection of textures, sounds or prefab models.

Construct Third party support

Modding is an inherently complex activity which often forces users to use imperfect development solutions to produce new content. Modding tools supplied with official toolkits are often not identical to the tools used by the official developers. Therefore, in addition to the learning curve the overall process of producing new content will always be slightly less efficient. A common occurrence in video game modding is the development of third party tools, i.e. programs or plugins developed by the community themselves. It is important to note that these are not necessarily idiosyncratic to a specific game, but can span multiple games.

Complement If an official mod tool is present but is deemed lacking or it is required to import certain files

into the base game, modders will often resort to using third party tools in a complementary role.

Replace However, if the official mod tools are of insufficient quality (i.e. ease of use and functionality) third

party tools may completely replace them.

CONTROL VARIABLES

Team size As teams increase in size, their ability to produce more content will likewise increase. While

inefficiency in management could occur, generally larger teams have access to more diverse skillsets and capital (Arakji & Lang, 2007; Franke & Shah, 2003).

Game genre Similar to films and literature, videogames can be divided into several genres. Genre can have

(20)

Game franchise When games are part of a franchise there is a greater likelihood of an established base of

users migrating to every new released title. Parmentier and Gandia (2013) found that in the case Trackmania, the developers incorporated this fact into their business strategy. Each new title was expected to have the existing modders transfer over from the previous release. Moreover, in a study on blockbuster games, Cox (2013) also found statistically significant evidence on increased popularity of sequels in franchises.

Game age rating ESRB or PEGI rating systems can influence the popularity of the base game (Cox, 2013),

which in turn will have an effect on the popularity of any derivative mod works. However, there is some research on the UK film industry which could suggest that mature ratings could enhance the appeal of the product, a so-called ‘forbidden fruit’ effect (Collins, Hand, & Snell, 2002). Nevertheless, no strong evidence was found to support this claim. In his video game blockbuster study Cox (2013) did find a statistically significant link between age ratings and sales. Mature rated games sold better than unrated titles.

Modding contest Developers or publishers can incentivize modders by offering financial rewards for the

development and release of mods. Recent examples would be Bohemia Interactive’s Make ARMA not War contest4_{and Bethesda’s Fallout 4 modding contest}5_{. To our knowledge no empirical research has been}

conducted on the influence of modding contests on sales or the mod community for that particular game. However, it probably can be reasonably expected that cash prizes will increase modding activity and sales of the game.

Steam workshop Digital distribution has revolutionized the video game business model. Allowing consumers

easy access to a variety of titles. Of particular interest to this study is the presence of the steam workshop, an online database where users can upload and distribute mods. Its ease of use due to fluid integration with the base game and automatic updates does offer advantages compared to traditional mod distribution through fan websites. Indeed, Koch and Bierbamer (2016) found evidence of steam workshop support positively affecting user rating and the number of available user innovations. Consequently, it will be prudent to control the data for the presence of a steam release with steam workshop support.

Quality original game Modded content is by definition dependent on a base. Thus, the quality of said base

becomes an important factor influencing the overall mod quality. High quality games will often develop large communities, it is not inconceivable modders are attracted to the prospect of a large pool of talent. In other words, a high-quality video game will more likely offer the prerequisite conditions for successful modding. As such Metacritic ratings of the original game will be used to control for popularity.

4_{http://makearmanotwar.com/ (accessed November 2016)}

(21)

3.2.3 MEASUREMENT VALIDITY AND RELIABILITY

In total eighteen attitude measuring variables were included in the survey. In order to prepare the data for the regression analysis it was pertinent to reduce the existing variables down to a more manageable size. Fewer predictors will generally improve model accuracy and given a sensible interpretation of a principal component analysis, a significant amount of information can be retained in the newly formed variables. As most of the variables were grouped together conceptually based on existing literature, it was expected the latent variance would mirror these hypothesized groups (i.e. toolkit, modders, and support).

Our eighteen variables were examined for kurtosis and skewness prior to the factor analysis. Only nine variables yielded statistically significant values with regard to their kurtosis values. Average size of the

mod team displayed a usually high kurtosis value, indicating a leptokurtic distribution.

Varimax factor rotation was chosen as it allows for correlation between two components. A Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO) of > 0,4 and Bartlett's Test scoring significantly were used to determine whether data reduction is warranted. Rotated Eigenvalues were taken into account (> 1.0). Moreover, proposed factors were judged on whether they would fit together logically.

Based on the literature three groups of variables were defined, concerning: (1) toolkit quality, (2) support from the community and developers/publishers and finally (3) modders themselves. These facets were hypothesized to contribute to mod quality. In total this resulted in eighteen variables.

Data reduction for all three components was warranted as KMO scores were above 0.4 and Bartlett’s test for sphericity was significant at p < 0.00. The PCA procedure resulted in the following six factors:

Factors CB α

Toolkit openness Control (2 items) 0.80

Features (3 items) 0.67

Support Official support (2 items) 0.72

Community support (3 items) 0.60

Third party tools (1 item) 0.18

Modders Modder quality (4 items) 0.75

Table 2 PCA

(22)

4 RESULTS

4.1 D

ESCRIPTIVE STATISTICS

Table 3 provides an overview for the descriptive statistics of the variables used in this study. Control variables with multiple dummy categories were excluded. Included in the table are number of observations, lowest and highest observation, mean and standard deviation. Different between the number of observations can be attributed to invalid response on the survey. Dummy variables are listed with a ‘d’ notation. A full correlation matrix can be found in appendix A.

Descriptive Statistics N Minimum Maximum Mean SD

Mod quality 150.00 0.00 100.00 77.08 21.15 Franchise (d) 150.00 0.00 1.00 0.86 0.35 Modding contest (d) 150.00 0.00 1.00 0.37 0.49 Steam workshop (d) 150.00 0.00 1.00 0.67 0.47 MC base game 149.00 70.00 94.00 81.26 7.49 Team size 145.00 1.00 60.00 5.63 9.28 Control 145.00 1.00 5.00 3.28 1.05 Features 128.00 1.00 5.00 3.13 0.83 Official support 145.00 1.00 5.00 3.69 0.99 Community support 142.00 2.33 5.00 4.14 0.69

Third party support 143.00 1.00 5.00 3.70 0.90

Modder quality 149.00 2.00 5.00 3.68 0.73

(23)

4.2 R

EGRESSION ANALYSIS

Model 1 Model 2 Model 3 Model 4 Model 5 Controls

Genre included

Franchise included

PEGI rating included

Modding contest included

Steam workshop included

MC base game -.892 (.470)* -.802 (.473)* -.799 (.466)* -1.031 (.495)** -1.114 (.497)** Team size .591 (.229)** .393 (.241) .360 (.233) .389 (.237) .337 (.244) Toolkit Openness H1 Control 1.265 (2.349) 2.481 (2.281) 1.452 (2.460) .041 (2.527) Features .552 (3.069) .690 (3.151) 1.348 (3.352) 1.343 (3.348) Modders H2 Modder quality 7.521 (3.034)** 8.942 (2.941)*** 9.386 (3.106)*** 9.120 (3.162) *** Support Official support -5.598 (2.174)** -5.053 (2.300)** -4.861 (2.302)** Community support 4.474 (3.166) 4.900 (3.359) 5.839 (3.412)*

Third party support 3.958 (2.304)* 3.749 (2.371) 3.141 (2.420)

Interaction

H3a Control C * Official 2.761 (2.640) 3.496 (2.681)

Features * Official -.550 (2.831) -.416 (2.814)

H3b Control C * Community 4.160 (3.001) 3.866 (3.044)

Features * Community 2.772 (2.824) 2.846 (2.834)

H3c Control C * Third party 1.555 (2.660) 2.182 (2.707)

Features * Third party .792 (2.203) .149 (2.238)

H4a Modder quality * Official 5.202 (2.388)**

H4b Modder quality * Community -.111 (2.083)

H4c Modder quality * Third party .251 (2.278)

R2 _.138 _.193 _.282 _.323 _.362 R-Change .138 .055 .089 .041 .038 F-change 1.595 2.215 3.907 .891 1.690 Sig F-change .119 .091 .011 .505 .175 P value model .119 .056 .007 .018 .014 N = 111 Max VIF 5.564

Table 4 Regression model

Unstandardized coefficients are represented with standard errors in parentheses * p < .1, ** p < .05, *** p < .01

(24)

Our model is presented in five distinct iterations, each adding a specific category of variables in order to improve overall fit as measured by a significant change in R2_{values. Model 1 is limited to the inclusion of}

control variables and yields a non-significant R2 _{value of 0.138. Metacritic rating of the base game (β = -.892}

is shown to negatively correlate with mod scores but only at the p =.1 significance level. Team size (β = .591) however, correlates positively with mod scores.

In model 2 the main effects toolkit and modders were added. Both the model itself and the ΔF = 2.215 are non-significant. The analysis reveals no evidence for the support of hypothesis 1, which assumed a positive relationship between toolkit and mod quality. However, evidence supporting hypothesis 2 can be found as modder quality has a positive relationship with mod quality (β = 7.521; p = .02).

Model 3 introduced support as a variable. Both the model itself and the ΔF = 3.907 are significant. While support is hypothesized to be an interaction effect, we studied its direct effect on mod quality. In this independent role, official support is shown to strongly negatively correlate with mod quality (β = -5.053; p = .01)

The first set of interaction effects, support * toolkit were added in model 4. Although the overall model was significant, the ΔF = .891 is not. None of the interaction effects between support and toolkits were significant. Hence no evidence supporting hypothesis 3a, 3b and 3c is found.

Finally, model 5 added the support * modder interaction effects. Model 5 is significant overall, but the ΔF = 1.690 is not. One significant interaction effect was found, namely official support * modder quality (β = 5.202; p = .03). While the effect in itself is significant, the relationship is inverse. Hypothesis 4a is therefore rejected. Hypothesis 4b and 4c are rejected due to lack of sufficient evidence.

When plotted into a graph (figure 3) as advised by Aiken & West (1991), the exact nature of the interaction effect is revealed. Low official support does not appear to affect low and high quality mod teams. However, high quality support enhances the ability of high quality teams to create better mods.

Figure 3 Interaction effect

0 20 40 60 80 100 120

Low Modder quality High Modder quality

M

od

Qual

ity

(25)

5 DISCUSSION AND CONCLUSION

5.1 D

ISCUSSION

This study analyzed “openness” in toolkits for the PC video game industry and its effect on the quality of mods. No evidence was found that openness influences the quality of mods. Results suggest that mod team quality plays a significant role in determining the quality of a mod directly. This study also found that despite its negative direct impact, official support strengthens the positive influence of mod team quality. It would appear that as an interaction effect, official support combined with mod team quality as a moderator, allows high quality mod teams to create higher than usual quality mods.

This study uses a classification with three types of support. This is important to realize as the effects of them differ significantly. Whereas community and third party have no influence, official support has a negative direct effect, but strengthens the link between mod team quality and mod quality. This study contributes to the literature by emphasizing factors outside of the toolkit. No direct relationship between community and third party tools was found on mod quality and neither did these variables act as moderators between (1) toolkits a mod quality (2) modder quality and mod quality.

5.2 C

ONCLUSIONS

The primary focus of this study was to investigate the influence of toolkit openness on mod quality as well as determine the influence of support on this relationship. Through empirical research it sought to find significant evidence supporting this theory. While our final model has a predictive value of 36% (R2 _{= ,362)}

(26)

2014) that modders value the creative process, so overbearing support can quickly be perceived as interference.

Collaborative development and sharing knowledge, a practice so common in the OSS industry and a staple of the open innovation paradigm are thus at odds with video game modding culture. However, a distinction should be made between the exchange of user innovations and the protection of mod IP. As has been established modders are willing to help with the technical process of modding (Postigo, 2007; Postigo, 2008; Poor, 2014). It is therefore possible that publishers and developers seeking to appropriate mod IP risk antagonizing their mod communities, while publishers and developers adopting a more “laissez faire” approach will be able to enhance the value of the base game due to the presence of a content producing modding community.

Although the research shows official support has a negative direct impact, we have no real substantial emperical data to support our hypothesis of interferrence with the creative process. While this reasoning is supported by anecdotal evidence, future research should focus on investigating the precise nature of the relationship between support and mod quality.

Furthermore, as mod team composition is likely a significant contributor to mod quality, developers should make every effort to create an environment where modding teams can remain together. This is especially important in the case of high quality mod teams. Specifically, developers can direct their efforts on creating and facilitating a supporting infrastructure for mod teams. Forums can be maintained to facilitate mod team and general community interaction. Additionally, server hosting for the purpose of data storage or websites are methods of creating a supporting infrastructure. Again, this coincides with earlier theories on inducing user innovation in video gaming (West & Gallagher, 2006). Nevertheless, developers should be careful not to directly interfere in the modding process itself.

5.4 L

IMITATIONS AND FUTURE RESEARCH

Overall the sample size of 111 is very restrictive. Some unexpected results and lack of significant results can probably be attributed to some degree to the lack of observations. Future research should focus on obtaining a larger sample size.

(27)

As established earlier, using normalized experts scores is a good measure for determining product quality (Koch & Bierbamer, 2016; Elliott & Simmons, 2008; Eliashberg & Shugan, 1997). Its usage in this study was suboptimal for two reasons. First, ModDB scores are not expert scores. It is very likely some bias will slip in when peer reviewing takes place. Modders might not be entirely truthful when rating their peers. Modding communities tend to be small and interconnected, coupled with the high degree of amicability between modders (Poor, 2014) objective rating might be difficult.

Second, although great care was taken to ensure objective measurement of mod scores by using external sources, we at times had to report to self-reporting. Therefore, some bias is included in these mod scores. Part of this problem is inherent to the mod score metric. Compared to commercially released video games or films it is significantly harder to find a centralized source with normalized scores. Future research should therefore find a different metric or alternatively, increase the duration of the survey runtime. Furthermore, some support can be found for the role of third party tools as a predictor for mod quality. Combined with the data gathered from the interviews as well as the overall prevalence of third party mod tools in the general pc video game industry this warrants further investigation.

(28)

REFERENCES

Amabile, T. M., Conti, R., Coon, H., Lazenby, J., & Herron, M. (1996). Assessing the Work Environment for Creativity. The Academy of Management Journal, 39(5), 1154-1184.

Aiken, L. S., & West, S. G. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, London: Sage.

Arakji, R. Y., & Lang, K. R. (2007). Digital Consumer Networks and Producer-Consumer Collaboration: Innovation and Product Development in the Video Game Industry. Journal of Management

Information Systems, 24(2), 195-219.

Bagozzi, R. P. (2011). Measurement and Meaning in Information Systems and Organizational Research: Methodological and Philosophical Foundations. MIS Quarterly, 35(2), 261-292.

Baldwin, C., & von Hippel, E. (2011). Modeling a Paradigm Shift: From Producer Innovation to User and Open Collaborative Innovation. Organization Science, 22(6), 1399-1417.

Baldwin, C., Hienerth, C., & von Hippel, E. (2006). How user innovations become commercial products: A theoretical investigation and case study. Research Policy, 1291–1313.

Becoming an ideal co-creator: Web materiality and intensivelaboring practices in game modding. (2013).

new media & society, 16(2), 290-305.

Bengtsson, L., Lakemond, N., Lazzarotti, V., Manzini, R., Pellegrini, L., & Tell, F. (2014). Open to a Select Few? Matching Partners and Knowledge Content for Open Innovation Performance. Creativity and

Innovation Management, 24(1), 72–86.

Bogers, M., & West, J. (2012). Managing Distributed Innovation: Strategic Utilization of Open and User Innovation. Creativity & Innovation Management, 21(1), 61-75.

Bogers, M., Afuah, A., & Bettina, B. (2010). Users as Innovators: A Review, Critique, And Future Resarch Direction. Journal of Management, 19.

Burger-Helmchen, T., & Cohendet, P. (2011). User Communities and Social Software in the Video Game Industry. Long Range Planning, 44, 317-343.

Caves, R. E. (2000). Creative Industries: Contracts between Art and Commerce. Cambridge: Harvard University Press.

Chesbrough. (2003). Open Innovation: The new imperative for creating and profiting from technology. Boston: Harvard Business School Press.

(29)

Collins, A., Hand, C., & Snell, M. C. (2002). What makes a blockbuster? Economic analysis of film success in the United Kingdom. Managerial and Decision Economics, 23(6), 343–354.

Cox, J. (2013). What Makes a Blockbuster Video Game? An Empirical Analysis of US Sales Data.

Managerial and Decision Economics, 35(3), 189-198.

Eliashberg, J., & Shugan, S. M. (1997). Film critics: Influencers or predictors? Journal Of Marketing, 61(2), 68-78.

Elliott, C., & Simmons, R. (2008). Determinants of UK Box Office Success: The Impact of Quality Signals.

Review of Industrial Organization, 33(2), 99-111.

Faraj, S., Jarvenpaa, S. L., & Majchrzak, A. (2011). Knowledge Collaboration in Online Communities.

Organization Science, 22(5), 1224-1239.

Franke, N., & Shah, S. (2003). How communities support innovative activities: an exploration of assistance and sharing among end-users. Research Policy, 32(1), 157–178.

Franke, N., & von Hippel, E. (2003). Satisfying heterogeneous user needs via innovation toolkits: the case of Apache security software. Research Policy, 32(7), 1199–1215.

Frey, K., & Lüthje, C. (2011). Antecedents and Consequences of Interaction Quality in Virtual End‐User Communities. Creativity and Innovation Management, 20, 22-35.

Füller, J. M. (2009). Consumer Empowerment Through Internet-Based Co-creation. Journal Of Management

Information Systems, 26(3), 71-102.

Garcia, R., & Calantone, R. (2002). A critical look at technological innovation typology and innovativeness terminology: a literature review. 19(2), 110–132.

Gassmann, O., Enkel, E., & Chesbrough, H. (2010). The future of open innovation. R&D Management, 40(3), 213–221.

Gilmore, J. H., & Pine, B. J. (1997). The four faces of mass customization. Harvard Business Review, 75(1), 91-101.

Hars, A., & Ou, S. (2002). Working for Free? Motivations for Participating in Open-Source Projects.

International Journal of Electronic Commerce, 6(3), 25-39.

(30)

Jeppesen, L. B. (2005). User Toolkits for Innovation: Consumers Support Each Other. Journal of Product

Innovation Management, 22(4), 347–362.

Jeppesen, L. B., & Molin, M. J. (2003). Consumers as Co-developers: Learning and Innovation Outside the Firm . Technology Analysis & Strategic Management, 15(3), 363-383.

Koch, S., & Bierbamer, M. (2016). Opening your product: impact of user innovations and their distribution platform on video game success. Electron Markets, 26(4), 357–368.

Lerner, J., & Tirole, J. (2002). Some Simple Economics of Open Source. Journal of Industrial Economics,

50(2), 197-234.

MacKinlay, C. A. (1997). Event Studies in Economics and Finance. Journal of Economic Literature, 13-39. Nieborg, B. D., & van der Graaf, S. (2008). The mod industries? The industrial logic of non-market game

production. European Journal of Cultural Studies, 11(2), 177-195.

Ogawa, S. (1998). Does sticky information affect the locus of innovation? Evidence from the Japanese convenience-store industry. Research Policy, 26(7-8), 777–790.

Parmentier, G., & Gandia, R. (2013). Managing Sustainable Innovation with a User Community Toolkit: The Case of the Video Game Trackmania. Creativity and Innovation Management, 22(2), 195-208. Pisano, P. G., & Teece, J. D. (2007). How to capture value from innovation: Shaping intellectual property

and industry architecture. California Management Review, 50(1), 278-296.

Poor, N. (2014). Computer game modders’ motivations and sense of community: A mixed-methods approach. New Media & Society, 16(8), 1249-1267.

Postigo. (2007). Of Mods and Modders Chasing Down the Value of Fan-Based Digital Game Modifications.

Games and Culture, 2(4), 300-313.

Postigo, H. (2008). Video Game Appropriation through Modifications. Convergence, 14(1), 59-74.

Prahalad, C., & Ramaswamy, V. (2000). Co-opting Customer Competence. Harvard Business Review, 78(1), 79–87.

Prügl, R., & Schreier, M. (2006). Learning from leading-edge customers at The Sims: opening up the innovation process using toolkits. R&D Management, 36(3), 237–250.

Rothwell, R. (1992). Succesful industrial innovation: critical factors for the 1990s. R&D Management, 22(3), 221-239.

Ryzhkova, N. (2015). Does online collaboration with customers drive innovation performance? Journal of

(31)

Sen, R., Subramaniam, C., & Nelson, L. M. (2008). Determinants of the Choice of Open Source Software License. Journal of Management Information Systems, 207-240.

Shah, S. K., & Tripsas, M. (2007). The accidental entrepreneur: the emergent and collective process of user entrepreneurship. Strategic Entrepreneurship Journal, 123–140.

Shanka, V., & Bayus, B. L. (2003). Network effects and competition: an empirical analysis of the home video game industry. Strategic Management Journal, 24(4), 375–384.

Storz, C., Riboldazzi, F., & Moritz, J. (2015). Mobility and innovation: A cross-country comparison in the video games industry. Research Policy, 44(1), 121–137.

von Hippel, E. (1986). Lead Users: A source of novel product concepts. Management Science, 32(7), 791-805.

von Hippel, E. (1994). “Sticky Information” and the Locus of Problem Solving: Implications for Innovation.

Management Science, 4(40), 429-439.

von Hippel, E. (2001). User toolkits for innovation. Journal of Product Innovation Management, 18(1), 247– 257.

von Hippel, E. (2003). Open Source Software and the "Private-Collective" Innovation Model: Issues for Organization Science. Organization Science(14), 209-223.

von Hippel, E. (2005). Democratizing Innovation. Cambridge, MA: MIT Press.

von Hippel, E., & Katz, R. (2002). Shifting Innovation to Users via Toolkits. Management Science, 48(7), 821-833.

von Krogh, G., Haefliger, S., Spaet, S., & Wallin, M. W. (2012). Carrots and Rainbows: Motivation and Social Practice in Open Source Software Development. MIS Quarterly, 36(2), 649-676.

von Krogh, G., & von Hippel, E. (2006). The Promise of Research on Open Source Software. Management

Science, 52(7), 975-983.

West, J., & Gallagher, S. (2006). Challenges of open innovation: theparadox of firm investment inopen-source software. R&D Management, 36(3), 319-331.

(32)

APPENDIX A SURVEY

Access: https://docs.google.com/forms/d/e/1FAIpQLSfj5LEcbtxmSXqt66JzG_DZDzbb7zPDwUArIVwQgf0Cz02 DyA/viewform#responses General 1. Gender 2. Age

3. Which country are you from?

4. How many hours have you spent on modding per week? 5. How many years have you been active modding? 6. How many completed mods have you developed? 7. How many unfinished mods have you developed?

Toolkit

8. This toolkit allows me to be in complete control when implementing new features 9. This toolkit does not limit me in my creativity when creating mods

10. It is easy to use this toolkit

11. There is adequate documentation included with the toolkit 12. This toolkit allows me to create standardized files

13. This toolkit allows me to easily collaborate and develop mods within a team

Support

14. The game developers actively support the modding process 15. Community managers actively support the modding process

16. A wide selection of textures, sounds and prefabs is available on community repositories 17. The modding community actively supports the modding process

18. The general gaming community of users supports the modding process

(33)

Modders

21. The average size of the mod team(s) I was part of was...

22. How would you describe the average level of experience of the team? 23. How would you describe the average level of technical literacy of the team? 24. How would you describe the average level of creativity of the team? 25. How would you describe the average level of commitment of the team?

Mod tools personal experience6

(34)

APPENDIX B CORRELATION TABLE

1 2 3 4 5 6 7 8 9 10 11 12 1 Mod quality 1 2 Franchise .111 1 3 Modding contest .173* .311** 1 4 Steam workshop .113 .579** .332** 1 5 MC base game -.193* -.195* -.597** -.376** 1 6 Team size .274** .022 .265** .147 -.354** 1 7 Control .049 -.115 -.074 -.043 .103 -.096 1 8 Features -.072 -.311** -.301** -.169 .328** -.119 .452** 1 9 Official support -.051 .016 .252** .088 -.271** .022 .233** .131 1 10 Community support .043 -.116 -.185* .010 .087 -.209* .152 .424** .250** 1

11 Third party support .181* .235** .192* .153 -.193* .018 -.219** -.202* -.085 .011 1

12 Modder quality .327** -.046 .158 -.049 -.191* .400** -.109 -.055 .024 -.151 .038 1

(35)