High performing contests : the relationship between performance structures, juries and selection systems

(1)

Msc in Business Administration

Entrepreneurship and Management in the Creative Industries

High Performing Contests – The

Relationship Between Performance,

Structures, Juries and Selection Systems

Raul Edvard Axel Röhr, 10827161

Supervisor: dr. J.J. Ebbers

(2)

Statement of Originality

This document is written by Student Raul Edvard Axel Röhr who declares to take full responsibility for the contents of this document.

I declare that the text and the work presented in this document is original and that no sources other than those mentioned in the text and its references have been used in

creating it.

The Faculty of Economics and Business is responsible solely for the supervision of completion of the work, not for the contents.

(3)

This study approaches contest theory from a strategic perspective, reaching for deeper understanding on topic specific subjects and through combining contest theory with selections system theory. The study hypothesizes that contests based on innovation that have high market uncertainty but low technical uncertainty benefit from limited entry. Second hypothesis is that larger juries are better at identifying best possible entries. Finally, the third hypothesis states that when the jury has a high representation of one selector group, the resulting output is most valued by the corresponding selector group when evaluated. This study uses data from Finnish architectural contests in its analyses. The findings of this study are mixed, with very little statistically meaningful answers. Ti does question the accuracy of the hypothesis. Main implication of this study is that there are still many points in which both contest theory and selection system theory need to be developed in order to become truly functional tools for practitioners and academics.

(5)

Introduction

The interest for the interactions of seekers and solvers in contest settings has been growing lately. With the rise of more and more contests to solve various issues, whilst at the same time the slow developments in the literature around this subject, the research into these constructs has been seen as lagging. There is a distinct need for more integrative studies that link the existing contest research into research in strategy, organization theory, and innovation to further illustrate how the theories of contests work in practice. (Lampel, Jha, & Bhalla, 2012)

This study links the structures of a contest into with the performance of the outcome of said contest. Understanding the structures, which govern the outcomes of contests, is something that is not studied thoroughly at this point. Some studies, such as Lampel et al. (2012) do look at these from a theoretical point of view, but further empirical research is needed. On a larger scientific perspective, this study will be able to provide better understanding on how the choice of key constructs can affect the performance of the outcome of the contests. One motivator to start a competition is to find outcomes that cannot be achieved through traditional methods, such as internal R&D, or are costly to achieve with these methods. Better understanding of the implications that the choice of structures on outcome innovativeness can thus further our knowledge on the right and desirable ways of setting up contests. It may also open new and interesting settings for further studies in the creation of high performing solutions. Another interesting point of this study is its attempt at combining contest theory with another strategic theory, the selection system theory. Contest theory has reached a stage in which it is now possible to combine it with other theories, at least as an explorational effort. The work undertaken by for example Connelly et al. (2013) has set the ground, though yet shaky, from which to start the enrichement of contest theory with new viewpoints.

(6)

This study is deductive of nature, testing hypotheses generated from the available literature. By using a quantitative approach, the study aims to test the hypotheses using data from different sources.

This study will proceed with a literature review that outlines the current state of contest theory. After that the hypotheses are presented, followed by a presentation of the empirical setting as well as an overview of the data and methods. This is followed by the results and finally the study ends with conclusions, discussion and suggestions for future research as well as the limitations of the study.

Literature Review

Defining Contests

In order to understand what contests are, this study begins by outlining the origins of the term. Contests are often studied under the umbrella of tournament theory. They take place inter and intra organizations. A common denominator of contests is that the reward is handed out based on the rank order of entries. (Connelly, Tihanyi, Crook, & Gangloff, 2013). Contest differ from other forms of events or forms of tournaments, namely from knock off tournaments and round robin tournaments, in the sense that a contest is a simple and robust model of a principal agent game, in which all players perform together once, which results in the identification of a winner or winners. In round robin tournaments all entries go head to head with one another, in the same style as in qualifier rounds of some sports tournaments, and in knock out tournaments a binary elimination is used in which certain entries go head to head with only the winner proceeding (Yücesan, 2013).

Research on contests has not reached a point in which the terminology is standardized and commonly agreed upon. Some refer to the structures that rank agents as tournaments (Yücesan, 2013), others call these competitions (Lampel et al., 2012), whereas some refer to them as contests (Boudreau, Lacetera, & Lakhani, 2011; Terwiesch & Xu, 2008). In this study I will refer to these

(7)

structures in which there are two parties in an event in which the first party offers a challenge and the second is a body of solvers of that challenge, and in which the reward is based on rank order, as

contests, which as a word seems to be widely used and distinct. For example, calling these

structures competitions might lead to situations in which the discussion is mixed with general competition related topics.

Contests can be described as competitive constructs, in which reward is based on a specific rank (Connelly et al., 2013). Contests have two parties involved. This study uses similar terminology as Boudreau et al. (2011) and refers to the parties as seekers and solvers. However, some authors use other expressions such as sponsors and organizers, and contestants (Lampel et al., 2012). In essence, all these terms refer to a situation in which there is one party that has a problem and that forms a contest around this problem. The people or organizations entering the contest in order to provide a solution, and collect rewards from solving it, form the other party. The reward is usually financial in nature, and can be given to only one solver, or can be given to many solvers based on their rank order.

The contests can be singular events or take place on a longer time span. For example, Top Coder is a platform for seeking novel algorithms which has constantly ongoing contests (Boudreau et al., 2011). Examples from the history which are provided in the following chapter are one-off competitions that once having solved the problem were not revisited. To further understand the nature of contests, this study will continue with background information about the subject, then studies different ways to look at and define between different types of contests, and how to describe the differences between contest structures, and finally presents the Selection System Theory which is the organization theory which this study links with the research of contests.

History Of Contests

As Connelly et al. (2013) noted contests are often studied through the view of tournament theory, which was raised from labor economics more than thirty years ago. However,

(8)

the history of contests, and interests in using them especially as a means of generating novel solutions to problems reaches further than the history of tournament theory. Some examples of the

use of contests comes from as early as the 15th century, from Florence, when the 50-year-old

architectural problem to build the widest and tallest dome up to that point in time was opened up for entries by any members of the public who could provide a solution to the problem. The problem was indeed solved by a clockmaker by the name of Filippo Brunelleschi, and he became the winner out of twelve proposals (King, 2000, cited in Boudreau et al. 2011). In this contest, the reward was given to the entry ranked as number one. Another example of the similar type of contest comes

from the 19th_{century, when Napoleon Bonaparte set out to find a substitute for butter that could be}

suitable for his armed forces as well as the lower classes. The contest did eventually produce what we know as margarine, but the problem took so long to solve that the winner, who is either Hippolyte Mege-Mouris or Michel-Eugene Chevreul, was never awarded a prize since Napoleon had already passed by the time the solution was available (Bullinger & Möslein, 2010). Today, contests are used by governments, companies, foundations and NGO’s, to solve many types of problems (Lampel et al., 2012).

The use of contests has become more and more popular during recent times, especially since the rise of internet based solutions and the interest to open innovation (Lampel et al., 2012). Also, contest are seen as enabling the organizations to grasp a larger pool of solvers while at the same time being less costly than the so called traditional methods (Yücesan, 2013).

Contests can be found inside organizations as well as in settings outside of them. They have been studied in labor markets, franchising, sports and in the setting of income differences, to name a few strands of research (Connelly et al., 2013). However, the use of contests as problem solving and as a source for externally generated solutions is the aspect that this study is focused on.

(9)

Classification Of Contests

The theory on contests still lacks a commonly agreed upon typology. However, many researchers have proposed insightful ways to approach the problem of categorizing different types of contest. As this study is mainly interested in extra-organizational contests, i.e. contests that take place outside of the organization, and contests that seek to solve problems that an organization faces, the paper will not discuss for example those contests that labor economics are interested in, such as promotions (Connelly et al., 2013).

One way to differentiate between types of contests is to look at the types of projects they are based on (Terwiesch & Xu, 2008). This way, contests are formed around expertise-based,

ideation and Trial-and-error projects. This breakdown is based on two factors, the technical

uncertainty of the project, basically meaning how likely it is that the reached solution works, and the market uncertainty of the solution, which refers to how likely it is that the seeker likes the proposed solution. Expertise-based projects are low on both, namely because the noise factor of the contest is low due to the solution being derived from previous knowledge, in other words the project is a part of a continuum for which knowledge can be derived from past experiences. This knowledge helps to understand the technical context as well as the market conditions. Ideation projects are broader and less detailed problems, thus they score high on market uncertainty but low on technical uncertainty. For example, design contests for aesthetics of a building represent a project in which the created solutions most likely are technically viable, but there is high uncertainty on the preferences of the seekers from the perspective of the solver. Finally,

trial-an-error projects score high in technical uncertainty, but low on the market uncertainty. The solution

landscape for these projects is very rugged, usually meaning that the functionality of a solution cannot be known before testing it. An example of such a project would be a project with well-defined goal but not much of anything else, for example a pill to cure hangovers. Terwiesch & Xu (2008) do not define projects with high uncertainty in both aspects.

(10)

Another way to classify contests is based on the goals (Lampel et al., 2012). Very similar to the language used by Terwiesch & Xu (2008) the classification by Lampel et al. (2012) sees goals as either broad or narrow. Narrow agendas are seen to arise from lack of resources and the aim is mainly to generate innovations and solutions at lower cost. Broad goals on the other hand are aimed at creating non-existing markets or create evolution to a certain direction on a market. This means that contests with broad goals usually focus on industry bottlenecks or on technological and product trajectories that portray underinvestment. The authors provide an example of such a contest in the form of the Super Efficient Refrigerator Program undertaken by the International Institute for Energy Conservation in the 90’s, which aimed to create refrigerators that delivered significant energy savings with lowest possible cost. This contest aimed to impact the energy efficiency of this particular industry on the long term. The classification used by Lampel at al. (2012) in regard to goals in described by the authors as a gamut, meaning that the different goals that organizations have span on a continuum, from simple to broad, rather than fit distinct categories. The decision to set narrow or broad goals is not only conscious, but caused by the context as organizations with less resources and administrative flexibility usually are those that inhibit narrow goals. Also noteworthy is that, the authors see that there can be evolution within this gamut in contests that become more frequent in nature. As the seeker becomes more experienced in contest, the goals tend to become broader. This may be due to the fact that narrow goals are usually used by organizations with less resources and/or those that lack administrative flexibility, both of which may be suspect to increases as the seeker gains experience as stated before.

Lampel, Jha & Bhalla (2012) also provide another way of classifying contests by categorizing them as private-, public- or mixed-good benefits that they produce. This distinction is fairly straightforward, and based on the goods that the outcomes of a contests provide. Private-good benefits namely mean profits for the seekers and solvers. Public-good benefits are for example

(11)

wider diffusion of innovations produced by contests, which are often the benefits that contests run by foundations and governments aim to provide. Mixed-good benefits provide both.

Contest Design

A seeker has many decisions to make when forming a contest around a certain issue. First decision is made between using internal or external solvers. Internal contests are not in the focus of this study, thus attention is mainly given to the decisions that an external contest, i.e. a context happening outside of the organization, in which solvers are not part of the organization. After the decision to hold an external contest has been made, other aspects of the contest need to be set. For example, Terwiesch & Xu (2008) recognize the following decision to be the determination of the allocation method of the reward.

The decisions made regarding the constructs of the entry stage, as well as those made for the selection stage are the focal points of this study. To further illustrate the possible and necessary options a seeker has when forming a contest, we look at the different options previously studied by academics.

Entry Stage

Lampel et al. (2012) provide insights into the categories of contests based on their competitive orientations and structural composition. Competition orientation refers to the rivalry within the contest. In contests with low rivalry cooperation is supported and there usually is more than one prize to be handed out. On the other end, highly competitive contests usually have only one winner and there is no support for cooperation. The structural composition refers to how many stages a contest has and how many way to assess a solution exits.

The reward structure and awards are often discussed in literature (Boudreau et al., 2011; Lakhani et al., 2013; Lampel et al., 2012; Terwiesch & Xu, 2008). Based on the literature, contests can roughly be seen to either have one winner, in other words searching for an extreme outcome, or they can award many contestants based on rank. The first of the contests functions

(12)

namely when the main aim is to solve one problem, in the best possible way. The second option is often used when the aim is to create overall higher levels of performance, for example in sales competitions where the aim is to get higher performance out of the whole sales team rather than search for one extreme solution.

Much of the contest literature focuses on settling the right amount of entrants to be included in the contest in order to produce maximum quality of output. Many theories around the subject have to do with incentives (Boudreau et al., 2011; Taylor, 1995; Terwiesch & Xu, 2008), rivalry (Boudreau et al., 2011; Terwiesch & Xu, 2008) and uncertainty (Boudreau et al., 2011). According to game theoretical and purely economical models, the number of entries to a contest should be limited in order to reach the equilibrium state that produces the best possible outcome for a contest (Taylor, 1995). Open contests, especially ones without entry fees cause situations in which it is difficult to reach a valuable solution as an outcome, as solvers have little incentive to invest in the solution. Taylor (1995) discusses first best contests, and finds that bigger prizes increase the research investment equilibrium, in other words the bigger the prize the more effort is given to the solution. However, the more contestants a contests has the less effort is used, and in open contests the solvers underinvest heavily as the odds of receiving the reward, even a big one, is small. Another reason to limit entry is also to keep costs down (Fullerton & McAfee, 1999) but mostly the argument is focused on the effects of higher number of entrants to the motivation and incentives of the entrants overall, ie. how adding more solvers to the contest affect the incentives to develop and invest in the solution to be submitted. According to these theories, open innovation contests do not yield the best possible results, because the incentives of the solvers to use maximum effort is lowered by each new entrant into the contest. According to Terwiesch and Xu (2008), some authors have gone as far as to proposing that the entries should be limited to only two.

This theory is put under scrutiny by two viewpoints, the parallel path effect theory and the effect of uncertainty, as understood in this case as the amount of domains of knowledge that

(13)

need to be used in order to solve the problem (Boudreau et al., 2011). The parallel path effect describes the situation, in which the large amount of solvers also brings many ways, or paths, to the solution. Thus the probability of an innovative result, or an extreme output of the contests is higher. Parallel path effect is not however a sufficient theory to overcome the issues arising from underinvestment due to larger amounts of entries and the reduced probabilities of winning the prize because of more entries. What is seen as the moderating factor is the amount of domains that the solution needs to be derived from, which Boudreau et al. (2011) call uncertainty. What should be noted at this point is that Bodreau et al. use the word uncertainty in a different way than for example Terwiesh & Xu (2008), who use it to discuss technical and market uncertainties, which are analyzed by solvers. In the work by Boudreau et al., the more uncertain the solution is, i.e. the more domains of knowledge are needed to solve the problem, the more reasonable it is to assume that the parallel path effect will overcome the issue of underinvestment and thus having more entries increases the odds of finding an extreme solution to a problem, even despite the fact that all entrants are not incentivized to invest their maximum effort.

Some papers also bring forth the decision of having an entry fee into the contest (Taylor, 1995). The aim of the entry fees might be for example to gain financial benefits beyond only the economic value of the winning proposition. However, Taylor (1995) notes that rather than having an entry fee to cover for the risk of ending up with a winning solution that does not deliver the sought value, the seeker should opt to organizing a closed contest rather than an open one in which incentives to invest in for example research for the solution are undermined by the probability of winning. As Boudreau et al. (2011) notes, this phenomenon is also moderated by the uncertainty aspect.

Selection Stage

Lampel et al. (2012) categorize contests based on their governance model, namely by who the assessment is done. This is broken down into expert, peer and vox populi assessments.

(14)

Expert assessments are made by experts in the relevant areas to rank the contest proposals. In the peer assessment the same function is done by peers and is vox populi this assessment is made by the public.

Some scholars have also presented highly standardized, mathematical ways to undergo through the selection process. An example of this is the OCBA formula, which handles the selection as an optimization problem (Chen, 1995). These selection methods such as OCBA seem rather conceptual, and are outside the scope of interest of this study.

Selection system theory

To bring in a new perspective into existing contest theories, this study uses selection system theory as a way to study and discuss the competitive process. Selection system theory has been credited as an approach that focuses on value and how and by whom it is created. In essence, selection system theory is a framework in organization theory that focuses on the process of value creation – it focuses on the process of value creation and the interaction between different actors, and how they try to influence each other (Wijnberg, 2004). Competitive structures in selection system theory are structured around the determination of value. This determination is done by three ideal types of selectors, which are market, peer, and expert. The value of a product or a service is determined through evaluations of these selectors (Wijnberg, 2011). This study views contests as ad hoc organizations, and focuses on the way these ad hoc organizations create and detect value. The focus of both selection system theory and this study is to analyze certain competitive arenas through focusing on the characteristics that the actors, especially the selectors involved have (Wijnberg, 2004).

Each system consists of a distinct set of selectors. These classifications of selectors are ideals, that can sometimes be blurry in reality (Ebbers & Wijnberg, 2012). However, the classification has been used to study different types of value creation, for example in nascent ventures seeking for investment (Ebbers & Wijnberg, 2012), in the context of visual arts (Wijnberg

(15)

& Gemser, 2000) and in award settings (Gemser, Leenders, & Wijnberg, 2007). Selection system theory thus gives a good theoretical background for studying competitive structures, in this case how contests can be used to create ultimate value.

Market selection can be described as a competitive process in which consumers

decide the value for the object of evaluation. This could also be called a traditional type of competition (Wijnberg, 2004). On the other hand in peer selection the value is evaluated by peers of the producer. A common example of such as a system is that of the academic world, where the value of scientific research is mainly evaluated by peers. In expert selection the evaluation and determination of value is done by individuals that are neither consumers nor peers of producers. (Wijnberg, 2011) Expert selectors need to be ascribed particular knowledge or expertise on the subject (Wijnberg, 2004). This categorization on selectors represents the ideal by which selection system theory observes the competitive process. The classification of individuals needs to hold true during competitive process in a way that the selectors are in only one role during the process. (Ebbers & Wijnberg, 2012) The selectors can also be graded on their influence to the preferences of the consumers (Wijnberg, 2011).

Important notion on expert selectors is that for an individual to be considered an expert selectors, there needs to be a connection between the proposed selectors opinion and customer behavior, and with the outcome of the competitive process in which the producers are involved in (Ebbers & Wijnberg, 2012). Often these experts mainly determine value and are no co-producers involved in the actual creation of value (Wijnberg, 2004). Since audiences in the different stages of a competitive process can be very different (Hsu, Hannan, & Hsu, 2005), which leads to the notion that selector roles are not idle but dynamic, meaning that someone can be one type of selector in the first stage whilst having another role in the second stage of the process or even have no role in the second stage.

(16)

In some cases, selectors function as gatekeepers. Usually this refers to a situation in which a selector or a group of selectors functions as a decision maker that has power over which projects to fund. In this situation the role of the selector is to evaluate which proposition fulfill the set criteria and are most potential to compete in the next phase (Wijnberg, 2011). An example of such a situation was given by Ebbers & Wijnber (2012), in which they studied how investors make decisions on which film projects to fund. These investors thus evaluate the feasibility of projects, not only in the competitive process for funding but also at the next stage in which films compete for audiences. Interestingly, in this study even though a classic definition of market selection could not be used since consumers only enter the picture during the following stage, market selection could be identified as film distributors. Distributors base their investment decisions on the expected consumers preferences and thus act as intermediates, as a sort of quasi-consumers, forming a distinct selectors category that fits very well into the category of consumer selectors. In this study, peer experts were identified as selectors with similar education and/or professional background as the investment-seekers, whereas all other types were seen as experts.

Hypotheses

Based on the literature on contest size, the most high performing outputs should be found when the entries are limited, if the problem does not feature uncertainty as defined by the need to draw from multiple knowledge platforms. As the studied type of contest in this paper is an innovation

contest as defined my Terwiesch & Xu (2008), the contests do not feature the characteristic of

uncertainty, since the creation of an entry mainly requires understanding of building aesthetics. Without deviation in uncertainty and the parallel path theory not being moderated by uncertainty, the first hypothesis is as follows:

- H1: Contests with limited entry yield better performing outputs.

An important, yet often over-looked part of a contest is the jury. To explore the subject further, this study focuses on juries in the final two hypotheses. Firstly, jury size is studied. As the current

(17)

contest theory does not touch upon this subject in the most respected studies, we use theories from other branches of science as our basis for hypothesis. In management, the need for diversity in teams has been seen as a necessity for high performing groups (Govindarajan & Gupta, 2001). The capabilities of organizations to detect breakthrough ideas has been credited to be dependent on individual capacity and capabilities (O’Connor & Rice, 2001). Thus, as a simplification of this we assume that additional members add both diversity and capabilities to a jury. This leads to the second hypothesis:

- H2: Contests with larger juries provide better performing outputs.

Selection system theory has previously been applied to award settings, to see what type of awards have the biggest influence on consumer behavior (Gemser et al., 2007). However, at the moment very few studies have looked into the capabilities of different selection systems, when the decision and evaluation is made in the first stage of the competitive process and then repeated in the second one. This could be seen as the predictive power of selection systems in delivering end user value. This study will use the phrase jury composition match to refer to the extent to which the jury has a match between its composition and the evaluating selector group. Formulated into a hypothesis, H3 is presented below.

- H3: Evaluators perceive the high performing outputs to be delivered from contests that have an emphasis on the corresponding selector type in the jury.

This hypothesis is broken down into three sub-hypotheses

- H3a: Contests with relatively high amounts of market selectors in the jury, will provide outputs that fare well amongst market evaluators.

- H3b: Contests with relatively high amounts of expert selectors in the jury, will provide outputs that fare well amongst expert evaluators.

- H3c: Contests with relatively high amounts of peer selectors in the jury, will provide outputs that fare well amongst peer evaluators.

(18)

Following the presented hypotheses, the conceptual model of this study can be presented as portrayed by figure 1.

Image 1. The conceptual model of the study

Empirical setting

This study is conducted in a context in which contest are frequent, and arise from innovation based projects. The empirical setting is prototypical for innovation based contests, and is mentioned as an example in previous studies. Architectural design contest which seek to solve the problem of finding the suitable aesthetics of a planned building are an example of projects in which there is little technical uncertainty, yet high market uncertainty (Lampel et al., 2012). After searching for suitable sources for empirical setting, this study settled on the Finnish architectural industry. The Finnish Association of Architects collects a database and also publishes information on most architectural contests in Finland. Using the data found through the site, this study aims to

(19)

answer some key questions regarding contests today, which were formulated into hypotheses in the previous part of the study.

Data and Methods

In order to illustrate how contest structures affect the performance of outputs, this study uses secondary data and primary data, to create a dataset for the purpose. The secondary data from architectural contests, which were held in Finland between 1995 and 2014, is gathered from a databank owned by the Finnish Association of Architects, as well as from other external sources. The primary data used will be gathered through surveys addressed to the focal audiences of the study. Further explanations of both will follow next, after which the gathering procedures, methods and limitations are discussed.

Secondary data from the Finnish Association of Architects

The independent variables of the study will be measured using data gathered by the Finnish Association of Architects. The association gathers data on all the architectural contests held in Finland annually. Using this data will ensure an objective view of the subject matter, as the databank does not limit the sample in terms of for example size of the contest, who is organizing it, the type of the project or in any other way. Another option would be to for example use media sources to track architectural contests in the Nordic countries. Even though this sampling would be wider in terms of geographical locations of the contests, it would limit the scope to a more homogenous group of contests, namely big, usually public buildings. In other words, the focus of the study would be limited to mainly contests with a focus on generating public- or mixed goods benefits (Lampel et al., 2012). The data provided by the Finnish Association of Architects was complemented with data from contest holders and/or pr-releases when ever necessary and possible. Most of the data on jury compositions has been gathered from contest organizers jury reports that were publicly available.

(20)

To gather a sample significant in size, but still recent in nature, the dataset will be constructed from contests listed by the Finnish Association of Architects during 1994 – 2014. This provides with a significant enough sample (n>485) that key dependent variables can be tested. As contests have been evolving during the recent years (Connelly et al., 2013), limiting the sample to as recent data as possible is relevant. However, too strict limitation in terms of timespan would leave the study with a relatively small sample. The initial sample of 485 observations, or winning entries, was too large for a manageable survey, thus it was limited to 151 observations.

The secondary data will be used to construct a panel dataset. The main observed variables are:

-‐ Invitational or open contest -‐ Amount of entries

-‐ Size of the jury

-‐ Selection system composition of the jury -‐ Amount of contest phases

Some key variables are also gathered as controls. Such variables are as follows: -‐ Year of contest

-‐ Location of the contest

-‐ Amount of prize money to winning entry

Primary data gathered through surveys and secondary data on performance

To assess the outputs of the contests, the study focuses on their aesthetic quality as a singel of the performance of a contest. As the contest seek for aesthetically pleasing solutions, this study sees the performance to equal to how aesthetically pleasing these winning entries are evaluated to be by relevant selectors. Aesthetic quality perceptions will be surveyed through an online survey consisting of images of the winning entries in combination with a Likert-scale to indicate perceptions on the entries. The images are presented to three focus groups, that are similar

(21)

to the vox populi, expert and peer assessment systems (Lampel et al., 2012). This also corresponds to the selection system theory (Wijnberg & Gemser, 2000). This cross-sectional dataset is used as a dependent variable to study the relation between contest structures and the perceptions of aesthetic performance of the output of different selectors.

The survey consists of 151 images of winning contests. Evaluators were asked to rate the illustrations on three distinct scales, an excitement scale, a beautifulness scale and an overall architectural performance scale. These scales are based on scales used in other studies, that focused on architectural performance (eg. Gifford, Hine, Muller-Clemm, Reynolds, & Shaw, 2000; Marans & Spreckelmeyer, 1982; Stamps Iii & Nasar, 1997). Most studies featured more complex measurements, as their focus was on understanding architecture on a broader sense than just aesthetics of the building. Still, using these three scales was seen accurate enough for this study. However, only the last scale was utilized ultimately in this study, which is the most important of the three and directly measures architectural performance. An example of the survey view is presented in the following page of the paper. The survey was offered only in Finnish and all respondents were native Finns. The survey panel consisted of 12 people, of whom 1 was considered a peer, 2 were

(22)

industry experts and the remaining 9 were identified as market selectors.

Image 2. An example of the evaluator view of the survey

Analyses

The data was analyzed through statistical methods, using the software Statistical Program for Social Sciences also known as SPSS. The data was analyzed using various features of the software and statistical methods. Most importantly, the data was analyzed to check for normality and frequencies. To enable the use of crosstabs, some scale variables were also coded into new variables signifying high and low performers, which will be explained more in detail later on. Since H2 and H3 were about relationships between dependent and independent variables, regression analyses were seen as most suitable for this study.

(23)

Strengths and limitations of data and methods

Main limitations come from the geographical and time period related limitations of the sample of the independent variables, and the possible biases in the independent variables caused by human factors.

In the independent variables, there are always possibilities that the observed phenomenons are tied not only to the contest structures but cultural and regional differences. The control variables are set to place to eliminate for example to control for the effect of the size of the contest as bigger competitions usually involve more monumental buildings and aesthetics.

Second limitation, which is related to the dependent factors, is their interpretive nature. For example, winning proposals from 10 years ago might be seen as less appealing due to the change in aesthetics. Some control variables will help eliminate this, but the risk still exists. Also, categorizing entrants to single categories within the selection system theory might be difficult, as for example a journalist who writes about architecture, a classical example of an expert, also enjoys the benefits of the joys of aesthetics thus being also part of the vox populi, or can by formal education be an architect becoming a peer. To fully analyze all the categories, which the respondents belong, to be out of the resources of this study, thus respondents will be assigned to categories on based on few identifying variables.

The strength of the study lies on the fact that it brings together both theories of contests and of strategy. Thus it can help open new insights into the subject.

Results

Data Treatment and Descriptives

The total size of the database contained 485 observations, however there were representative images available of 151 of the observations. Thus the remaining 334 observations were disregarded

(24)

from the study. The remaining observations did not feature issues with skewness nor with kurtosis, thus no normalization or other treatments were necessary. To allow for crosstabulations, additional variables were created in which the median ratings given by the evaluators were used to create a low performance category and a high performance category. This way, the H2 could be analyzed using the separation of contests into invitationals, which have a limited entry, and open contests, which have less limitations regarding entry and the newly generated high and low performing categories.

An overview of the results is presented in the correlation matrix, which can be found from the Appendix section of this paper. The correlation matrix reveals several limitations to the data and other typicalities of the data. For example, the correlation matrix visualizes a significant negative correlation between the size of the jury and the peer ration. This is due to the fact that the contests had either 1 or 2 peers in their juries. This number is caused by the Finnish competition tradition in which either the invited firms or the organization of architects chooses a peer representative into the jury. In most cases one peer is chosen, in some the number is two. Considering that this number is almost constant, increase in the jury size affects the ratio negatively. Similarly, as this study treats decision making as a sort of zero sum game, which is measured as ratios that sum up to one, one meaning that the decision-making would be fully in the hands of a certain selector group, the ratios affect each other. An increase in for example expert ratio leads to a decrease in the other two. Contest type also affects peer ratio, because in general invitational contests tend to have smaller juries thus increasing the peer ratio. Year of the contest also seemed to affect the amount of entries to contests, which most likely is caused by the recent economic downfall, which has resulted in less construction and thus less architectural contests.

Most interestingly for this study we can observe a correlation between the amount of entries and the type of contest. This correlation demonstrates that open contests have more entries than invitationals, justifying the use of crosstabs to study H1. Also, H2 which is interested in the

(25)

relation between jury size and output quality is further supported by the correlations as there is a significant, p<0.05, moderate positive correlation between the size of the jury and the average grade. In terms of H3, there are no related significant correlations visible in the correlation matrix, which would suggest abandoning the hypothesis. This study did however analyze H3 further, if not for scientific purposes then as a curiosity factor and conversation starter, an exploration into the insights that selection system theory can bring to contest theory.

The influence of open vs. closed contests on overall performance

According to the hypotheses of this study, contests with limited entry should yield better performing outputs than open ones. However, this hypothesis was put under scrutiny by the correlation matrix. To dig deeper into this, further analyses were conducted.

H1 was analyzed utilizing the data from the survey and the information on the contest type, namely if the contest was an open one, meaning it had higher number of entries, or if it was a more limited, invitational one. The analysis was undertaken using the crosstabs analysis, results of which are presented below. Sample size was n=151.

Table 1. Crosstabulations between open and invitational contests and high vs. low performance.

The distinctions between high and low performance categories are based on the average grade, where the median is the separating value. Observed counts are presented without parentheses.

Low performance High performance χ2 Φ

Open 15 (19.3) 26 (21.7) 2.460 -.128

Invitational 56 (51.7) 54 (58.3)

In the category of open contests, the observed counts surpassed the expected ones in high performance, whereas the observed counts for low performance were lower than expected. This effected was inversed in the invitationals. However, these results did not fulfill the p<0.05 criteria

(26)

and are not significant. They do suggest that the initial hypothesis may be wrong, if we regard this notion as a non-significant finding.

Jury size and performance

H2 expects that the larger the jury is, the better at recognizing the best entries it is due to higher diversity for example. Indeed, the correlation matrix revealed a positive correlation between the size of the jury and the average grade. As correlations do not necessarily mean causality, the H2 hypothesis was studied further with more sophisticated measures. Also, since correlation matrixes are measures of simple correlation, they do not take into consideration other possible factors, which in the more refined analyses are used as controls.

The relationship between the size of the jury and the performance of the output was studied through two hierarchical linear regression analyses, with a sample size n=151. The data was split into invitationals and open contests, due to the difference in measuring the control variable for the size of the contest. As open contest size was measured in terms of the first prize amount but the size of invitationals was studied using the participation fee as an indicator, these two categories did not fit into the same model. Summaries of the analyses are presented below in tables 2 and 3.

Table 2. Hierarchical regression model for jury size’s effect on the performance of invitational

contests.

R R2 R2

Change

B SE β T

Step 1 .435 .189 .189

Year of the contest .075 .050 .308 1.512

Participation fee 2.080E.5 .000 .355 1.742

Step 2 .435 .189 .000

(27)

Participation fee 2.144E-5 .000 .365 1.282

Size of Jury -.002 .043 -.016 -.056

Table 3. Hierarchical regression model for jury size’s affect on the performance of open contests.

R R2 R2

Change

B SE β T

Step 1 .282 .079 .079

Year of the contest -.025 .027 -.161 -.937

First prize amount 7.210E-6 .000 .270 1.575

Step 2 .287 .083 -.003

Size of Jury .009 .025 .065 .338

H2 was studied with two hierarchical regression analyses, first of the two focused on invitational contests, whereas the second one focused on open contests. The analyses investigated the ability of jury size to predict the performance of the output of a contest, as signified by the average grade given to the winning entry by the panel. Visual interpretations of grades given by different types of selectors as presented by scatterplots revealed that only the average grades followed a linear relationship, most likely due to the small size of the panel. The year of the contest, as well as the size of the contest as indicated by the amount of either participation fee in invitationals or winning entry prize in open contests were used as control variables. Due to the different indicator used for the size of the contest, invitationals and open contests could not be analyzed simultaneously. Interestingly however, the model showed no difference in the proportion of the variance between the two stages, with the variance staying at 18.9% in the second stage,

(28)

when analyzing invitational contest data. This would suggest that there may not be any affect on the size of the jury and the performance of the output.

First of the two analyses did not provide statistically significant results. In the first step, which included the control variables resulted in p>0.05, F (2,20)=2.329. The second stage in which jury size was entered, the significance was very far from optimal, p=0.956. Thus, no conclusions regarding H2 could be derived based on the invitational contests.

Second analysis yielded little insight. With a p-value of 0.256, F(2,33)=1.420 already on the first stage, and a p-value of 0.737, F(1,32)=0.114 in the second stage, no further information could be derived from open contests. The proportion of variance grows very little between the stages, from 7.9% to 8.3% when moving from stage one to two, but as noted these results are very likely to be random.

The relationship between jury composition and performance

The final set of hypotheses involved selection system theories. These hypotheses basically claim that matching jury compositions should yield outputs that are regarded as performing better in the eyes of the corresponding selector group. For example the hypotheses suggest that he more peers there are in a jury, and the more decisive power they have over the selection of the winning entry, the better the grades given by second stage peer evaluators should give to the winning entry, as the viewpoints and tastes are similar.

As mentioned earlier in this study, the final of the three hypotheses was studied though it showed no significant correlations in the correlation matrix. This was undertaken as a curiosity measure, so to speak, rather than as a scientifically sound escapade. Thus, the results of the final set of hierarchical regression analyses are at best conversation starters.

The sample size for these analyses was the same as in the previous ones, n=151. Similar to H2, also H3a, H3b and H3c were analyzed using year and size of contest as controls, as described in the analyses regarding H2. Each sub-hypothesis was studied twice, first the

(29)

invitationals and after that the open contests. This lead to a total of 6 analyses. Below, each sub-hypothesis is analyzed and the results provided in the tables.

Table 4. Hierarchical regression model for market selector ratio’s effect on the performance of

invitational contests as evaluated by market selectors.

R R2 R2

Change

B SE β T

Step 1 .388 .151 .151

Year of the contest .027 .055 .101 .408

Step 2 .404 .163 .012

(30)

Table 5. Hierarchical regression model for market selector ratio’s effect on the performance of

open contests as evaluated by market selectors.

R R2 R2

Change

B SE β T

Step 1 .162 .026 .026

First prize amount 4.578E-6 .000 .163 .926

Step 2 .164 .027 .001

Market selector ratio -.060 .440 -.024 -.136

First of the two analyses for H3a did not provide statistically significant results. In the first step, which included only the control variables resulted in p>0.05, F (2,20)=2.329. The second stage in which the ratio of market selectors in the jury, the significance was very far from optimal, p=0.607. Thus, no conclusions regarding H2 could be derived based on the invitational contests, but the analysis did point to the addition of jury ratio to diminish the proportion of variance by 35%. Similarly, the second analysis for H3a, which used data from open contests had an unacceptable p-value in both stages, in the first it was 0.645, F(2,33)=0.444, and in the second p-p-value was 0.892, F(1,32)=0.019. In this analyses, the addition of the ratio for market selections only further explained the variation by 1%.

(31)

Table 6. Hierarchical regression model for expert selector ratio’s effect on the performance of

invitational contests as evaluated by expert selectors.

R R2 R2

Change

B SE β T

Step 1 .256 .064 .064

Participation fee 3.367E-6 .000 .034 .158

Step 2 .254 .064 .000

Expert selector ratio .037 .987 .008 .037

Table 7. Hierarchical regression model for expert selector ratios on the performance of open

contests as evaluated by expert selectors.

R R2 R2

Change

B SE β T

Step 1 .305 .093 .093

Year of the contest -.076 .051 -.252 -1.479

Step 2 .344 .118 .025

Year of the contest -.084 .052 -.280 -1.621

(32)

The analyses for H3b yielded little to no further insights. First, the analyses for invitational contests had first stage p-value of 0.515, F(2,20)=0.686, far above the p<0.05 criteria. Second stage p-value was 0.971, F(1,19)=0.001, signifying with almost full certainty that the reached results were random. There was no change in the proportion of variance explained by the model between the stages. In the same vain, the analyses of the data gathered from open contests had no significant results, with first stage p-value of 0.2, F(2,33)=1.692, and second stage p-value 0.344, F(1,32)=0.924. The proportion of variance grew from 9,3% to 11,8% between the changes showing very little relationship between the expert ratio and evaluations.

(33)

Table 8. Hierarchical regression model for peer selector ratios on the performance of invitational

contests as evaluated by peer selectors.

R R2 R2

Change

B SE β T

Step 1 .546 .298 .298

Step 2 .565 .320 .021

Peer selector ratio 4.059 5.253 .197 .773

Table 9. Hierarchical regression model for peer selector ratios on the performance of open contests

as evaluated by peer selectors.

R R2 R2

Change

B SE β T

Step 1 .077 .006 .006

Step 2 .119 .014 .008

(34)

The analyses regarding H3c suffered from similar issues as the analyses for H3a and H3b. First analysis, the one using data from invitational had a first stage p-value of 0.029, F(2,20)=4.250, which is within the p<0.05 criteria. However, the p-value in the second stage did not fit the criteria, p=0.449, F(1,19)=0.597. In this model the proportion of variance grew by 2.2% between stages, from 29.8% to 32%. Similar issues with the second analysis were evident. The p-value in the first stage was already at 0.903, F(2,34)=0.102, and grew to 0.923, F(1,33)=0.276. The proportion of variance explained by the model grew by 4,2% between the models.

Conclusions and discussion

General discussion

This study was undertaken in order to test and scrutinize existing contest theories and to bring together other strategic theories with existing contest theory. Especially the linking of selection system theory with contest theory was a key goal for the study. As these two scientific branches are relatively new, both could benefit from news ways of looking at the issues at hand. Using data from a creative industry with long traditions in contests, the study found some interesting conversational insights, but lacked in statistically significant results. The discussion and revision of these results is presented with noteworthy limitations and suggestions for future research.

Hypotheses, conclusions, and theoretical implications

Two of the hypotheses, H1 and H2, approached contest theory in its pure form. These did not bring in the additional views of selection system theory, which were the focus of the third hypotheses. Though different in their nature, all three were seen as very suitable for both the available data as well as interesting in their own terms.

H1 continued on from the previous contest theories, which, deal to a large extent with the discussion on the right amount of entries for contests. The most sophisticated of studies, and the

(35)

most recent ones have found conditions under which open entry is to be courage. One example is high uncertainty in terms of the necessary amount and strains on knowledge required to find the optimal solution for a given contest. This study identified our empirical setting to be an innovation-based project, in which the uncertainty does not fulfill the criteria set for cases in which open contests yield optimal results. However, our results, though not significant, point to a different direction. In this study, open contests seemed to produce outputs that were perceived as higher performing than the outputs of contests with limited entry. This could mean that either the classification of architectural contests as innovation-based undermines the true uncertainties involved in the search of building aesthetics, or that the existing theories do not represent the reality. In the latter case, the theories would need to be adjusted into stating that even slight uncertainties are significant enough to justify holding an open contests rather than one with limited entry. This would mean that we reject H1, but as stated the poor p-values of the analyses leave room for doubt.

H2 was a hypothesis that focused on contest theories only, but brought in a focus on the juries. Even though an important factor in any contest, this aspect has not received as much attention as for example entry amounts or rewards. The simplistic assumption of H1 was that more equals better, though in reality the relation probably is an inverted U-model. As the data pointed to the direction that in reality the jury sizes are confined to a certain practical interval, the assumption that more is better seemed realistic enough. The analyses conducted on the data suffered from issues of significance when analyzed further than mere correlations. Like in H1, also in H2 the study is left with a lot of room for doubt. However, the insignificant results point to very little of the variance being explained by the addition of jury members. This would mean that H2 is also rejected. More jury members does not mean that better entries are identified.

H3 of the study was the most experimental of the hypotheses. It combined theories on selection systems with contest theory. Already the correlation matrix showed that the data to study

(36)

this ambitious set of hypotheses was not sufficient, a notion which was further highlighted by more sophisticated analyses. None of the analyses conducted on the data fulfilled any of the significance criteria of a reliable study, and some revealed p-values over 0.9. The results did suggest that jury composition has very little affect on the performance of the output as evaluated by the corresponding selector groups. This would suggest rejecting H3, or at best approving it with the notion of the relationship being in practice almost meaningless.

The theoretical contributions of this study are limited at best. The original motivation of acquiring deeper understanding regarding existing theories as well as combining two very interesting facets of business research was namely reached only on the level of studying new empirical settings. Very few - if any - contest theory related studies have used material from architectural contests. However, this setting can provide many future insights into contest theory. This study shows that there is data available, and that this data can be a source for insights or at least food for thought. Also, this study provided one way to bring selection system theory together with contest theory, by creating a conceptual model that utilized selection system theory as a starting point. Interestingly, already the literature review pointed to similarities in the existing ways of categorizing contests and the way selection systems are categorized. Even as the hypotheses remain unsolved, this study adds to the existing theory through novel use of data and theories. Practical implications

Despite the fact that the study did not yield as significant results as was hoped, it does open new consideration points to practitioners in contests. The study for example suggests that contest practitioners may be better off if setting up an open contest, even if the limited entry one is cheaper. The study points to a direction in which not only trial-and-error based contests inhibit enough uncertainty to be suitable for open contests but also innovation-based ones fulfill the requirements. For anyone looking for an extreme output, choosing the right form of contest when designing it is a crucial decision, as the money lost due to an inferior solution may far outweigh the

(37)

extra expenses caused by organizing an open contest. Thus, any additional information and knowledge to support in this decision is crucial.

The second hypotheses pointed the focus into the jury and it’s size. The study did not yield many insights into the correct size of the study. Rather it showed that the range used in architectural contest - that is from 4 to 20 - may be the optimal range. Though the results are not conclusive, they do point to the old saying don’t fix what is not broken. For a practitioner in contests benchmarking successful ones in terms of jury size is an action worth taking.

Last of our hypotheses was theoretical in nature, and at this point is more of an exercise in business research than an important factor to consider for practitioners. Until further studies bring more light into this topic, the advice is not to overthink the jury composition. Future research may bring more light into this subject.

Suggestions for future research

This study shows that there are still ways in which contest theory can be developed. As noted in the literature review, contests are still lacking the definitive classification. This leads to situations in which identifying any given contest into a distinct category and analyzing it as a member of such is difficult. The mixed notions of H1 are an example of this. Even if the results would have been significant, it would have been hard to know whether the existing theory on limited vs. open contests needs rethinking or if our classification is the issue. As some meta-analysis into contest theory suggest, forming a conclusive classification is an important next step for contest theory (Connelly et al., 2013).

Future research into contest theory should also look into juries. Not all contests use juries, after all in some contests the winning solution is one that functions where as the losing ones do not. However, for example in contests seeking solutions that are aesthetical in nature, what is considered good or high performing is a matter of judgment, and in contests this judgment is made by the jury. As selection system academics have noted, the competition often happens in two stages,

(38)

first in one where the gatekeepers choose an entry that they believe is best for the market and secondly in the actual market. Thus, understanding who are best suited to review and decide upon the winners in the first stage can be crucial for the success in the second stage. Both selection system and contests theories can benefit from each other. In the future, combining the two could be undertaken for example in other empirical settings than architecture, or from another point of view.

Also, a deeper understanding of what diversity means in a jury should be studied in more detail. This study only looked at the subject in terms of number of members, assuming that more members mean more diversity, but other factors could be added. For example looking at the age composition, educational background, work background, experience etc. and connecting these with the performance of the output might yield new insights.

The growing population of contests, both online and offline, require more understanding. For contest theory to find its way into the contest practice, useful, comprehendible and reliable theories are needed. Worst-case scenario is that contests become popular only to fall out of fame due to the lack of understanding and useful theoretical tools. Thus experimental and risk-taking studies such as this one are needed to provide the general public with food for thought and a richer knowledge base when creating contests that find the best solutions and even change the world.

Limitations

This study, like most other studies, suffered from several limitations. Some were already known before data collection and analysis, some became evident when moving through the research process.

There were limitations in the used survey tools that may have affected the results. For example, there was no way to randomize the order of the presented pictures for respondents. Thus, the information may have suffered from what could be called a fatigue bias, as the survey took an estimated 45 to 60 minutes to complete. Another issue with the survey was the fact that making sure

(39)

that all images presented are equally representative is a difficult task. The criteria for images in this study was that whenever possible they would portray the proposed building from the point of view of a pedestrian. This was not possible for every entry and may have caused difficulties in evaluations. In this case, prepping the respondents in person could have helped, but due to the limited resources and the location of the respondents being Finland but the researchers location being the Netherlands was not possible.

The question whenever forming a survey is which scales to use. Often this boils down to accuracy versus simplicity. Especially in an online survey such as the one used in this study, simplicity is key. As the respondents had to already give a significant amount of time to the evaluation task, the scale was kept as simple as possible. In the end, only one out of three dimensions was used, as the first two were used in order to test an older, more architecture specific theory. In the end, this theory was abandoned due to refocusing in pure contest theory. This was an error made by the researcher that could have been avoided. However, the remaining scale was one that had the best theoretical background of the three and is often included in evaluations of architectural performance. There are also other aspects to consider when talking about architectural performance, which could be used in the future. For an online survey, they might be out of scope, but with more resources and for example through face to face sessions with evaluators, more aspects could be taken into consideration.

The composition of the evaluator panel was small and homogenous. Due to low response count, the panel did not correspond to the aim for this study. The original goal was to get three experts and three peers, as well as ten market evaluators. After dozens of emails, the final panel was formed from what was possible. This is also something that could have been avoided if more personal approaches than emails would have been possible. This issue leaves the door open for biased results due to small panel size. Ideally, a future study could repeat what has been

High performing contests : the relationship between performance structures, juries and selection systems