Constraining Factors for Decision Making in Serious Games - An embedded Case Study

(1)

______________________________________________________________________________

Constraining Factors for Decision Making in Serious Games -

An embedded Case Study

Master Thesis

Eric Schmitz – s2280019 MSc. Supply Chain Management

University of Groningen Faculty of Economics and Business

Supervisor: Nick Ziengs Co-assessor: Jasper Veldman Word count: 12,958

Abstract: This paper investigates the relationship between motivation, ability and opportunity

(MOA) on decision making (DM) behaviour in the context of educational supply chain management games. Within the scope of an embedded case study, based on the Universal Exports game, competing MOA models have been compared with respect to their explanatory power. It has been found that the MOA framework is indeed suitable to explain DM behaviour and that the constraining factor model proposed by Siemsen et al. (2008) provides a superior approach over traditional models in this respect. Thereby it could be demonstrated that a bottleneck approach not only explains behaviour better than multiplicative or additive models, but also offers advanced possibilities in improving the game environment for increased learning outcomes.

(2)

Over the last decades a transition of educational processes could be witnessed, where traditional forms of teaching are increasingly competing with methods that are mediated by digital technologies (Vlieghe, 2015). Serious games represent one example of this development, as they extend or replace classical methods such as face-to-face classroom teaching by interactive and immersive ways of learning in a more life-like context (Gunter, Kenny, & Vick, 2008). Serious games have long been used for training purposes and their origins can even be traced back to war games in ancient China (Wolfe, 1993; Pasin & Giroux, 2011). Also in the field of supply chain management (SCM) education they have a long tradition, with the Beer game being a prominent example existing since the early 1960s (Sterman, 1989; De Leeuw, Schippers, & Hoogervorst, 2015). Since SCM is receiving increasing attention in today’s globalized world its nature of considering complex business environments as a whole also puts growing demands on tools used for teaching in this context (Mangan & Christopher, 2005). One particular advantage of games over traditional methods is their ability to put learners in the role of decision makers (Kim, Park, & Baek, 2009). In educational games students are required to actively apply theoretical knowledge throughout the game while being responsible for their actions, ideally resulting in a creation of knowledge through self-experience (Zantow, Knowlton, Illinois, & Sharp, 2005). Therefore, it has been suggested that games are particularly suited for improving student’s abilities in business decision-making and strategy formation (Faria, Hutchinson, Wellington, & Gold, 2009), which are especially valuable in a supply chain environment.

(6)

be successful in it, resulting in games that might support the development of social interaction skills but fail in creating academic knowledge among its participants (Gunter et al., 2008). Even in games which theoretically require players to apply previously acquired knowledge there is little empirical evidence on if and how this indeed happens in practice. However, deviating from the intended way of playing a game increases the risk of missing its intended learning objective. The challenge in game design therefore resides in ensuring that participants are incentivized to indeed adopt the desired patterns of behavior. In the case of SCM games as addressed in this research this is represented by subsequently improving their decision making, which requires moving from intuitive to knowledge based, rational approaches over time. Thus, for evaluating a game’s success in reaching its educative goal it is necessary to gain a thorough understanding of players’ behavior and their corresponding decisions made throughout the game to explore what factors influence their decision making process.

(7)

behavior in the context of a supply chain management game? Second, how do the MOA variables interact?

For this purpose, the universal exports game of the University of Groningen, embedded in the course “Management Science”, is examined within the scope of an embedded in-depth case study, resulting in several theoretical and practical contributions. First, existing theory on decision making in SCM games is expanded by an in-depth exploration of factors influencing players’ behavior. The use of the MOA framework provides a novel approach in this respect and enables to categorize and analyze influences on the decision making process. Thereby, also competing MOA models can be compared and evaluated on their explanatory power in this context. Finally, practical guidance is given on how to adjust elements of the game and its learning environment to enable an increased learning outcome for students in the future.

(8)

II. THEORETICAL BACKGROUND

2.1 Supply Chain Management and Supply Chain Management Complexity

Before it is possible to elaborate on the effect of educational frameworks surrounding SCM games, it is necessary to clarify what SCM exactly is and what makes games an appropriate means of teaching in this specific context. With many definitions on the term itself existing, this paper builds on Mentzer's (2001:18) frequently cited attempt to unify the manifold aspects surrounding this theme, defining SCM as “the systemic, strategic coordination of the traditional business functions and tactics across these business functions within a particular company and across businesses within the supply chain […]”.

(9)

decision process, in which one decision largely affects other decisions, is known as supply chain decision-making complexity (Manuj & Sahin, 2011).

2.2 Supply Chain Decision Making

Decision making generally is a process that, consciously or unconsciously, occurs in everything we do, not only on an individual but also collective level (Bazerman & Moore, 2008). In the field of SCM it plays a crucial role on both operative and strategic levels, ranging from the determination of lot sizes in production to the strategic selection of suppliers. Decision making involves criteria and alternatives to choose from, each of them dependent on preferences and goals (Saaty, 2004). A distinction between different types of this process has evolved, resulting in a spectrum whose ends represent two opposite forms: rational and intuitive decision making (Simon, 1987; Wolkoff, 1992).

Rational decisions are described as choosing deliberately among alternatives in a way that accords to well defined preferences and goals of an individual or group (Doyle, 1999). In other words, under circumstances of certainty, where a set of alternatives and outcomes is known, individuals will select the option that satisfies their preferences the most (Samuelson & Zeckhauser, 1988). On the other side, intuitive decision making can broadly be characterized as a perceptual process that does not require rational analysis and might be based on inadequate or non-objective information such as the decision maker’s experience and emotions (Shirley & Langan-fox, 1996; Burke & Miller, 1999). Thus, the decision making takes place as a subconscious act of linking disparate elements of information (Raidl & Lubart, 2001).

(10)

operations research and management science, resulting in various models and frameworks that support the decision making process such as network- or forecasting-models (Simon, 1987; Cachon & Lariviere, 2001). Their intent is to reduce uncertainty and to allow making more rational than intuitive decisions. However, while mostly aiming at optimizing decisions under conditions of relative certainty, traditional literature often still fails to recognize the human factor in decision making and the problems it implies (Mantel, Tatikonda, & Liao, 2006).

One stream of research addressing these issues is represented by behavioral decision making literature (cf. Wright, 2013). As a part of behavioral operations management theory this literature focuses largely on human level aspects of decision making and how this process is influenced by internal and external effects, such as changes in the environment, the decision maker’s personal perspective, or even the task itself (Payne, Bettman, & Johnson, 1993). One of its key arguments is that humans naturally have a “bounded rationality”, causing them to unwillingly make non-rational decisions due to a lack of important information or cognitive capacity (Bazerman & Moore, 2008). In order to deal with complex problems as faced by SCM, humans tend to use simplifying heuristics, especially when there is no clearly dominant best alternative (Simon, 1997). Thereby, decision makers may ignore particular pieces of information and fail to adjust their initial intention (Sanbonmatsu, Posavac, Kardes, & Mantel, 1998). Thus, in the context of supply chain education in general and decision making in particular it is desired to reverse such behavior by making deciders aware of their own biases and encourage them to use rational over intuitive approaches.

(11)

2.3 Understanding the effectiveness of Serious Games

From its beginnings in the 1950s to the present day SC education has evolved significantly over time and is increasingly replacing or complementing traditional teaching with methods such as serious games (Sweeney, Campbell, & Mundy, 2010). Within the scope of this research, and in line with Wouters et al. (2013), serious games will be defined based on their general characteristics, which is being interactive (Prensky, 2001; Vogel et al., 2006), based on a set of agreed rules and constraints (Garris & Ahlers, 2002), and directed toward a clear goal that is often set by a challenge (Malone, 1981). Further, games continuously provide feedback, either as a score or through adjustments in the game scenario, to enable players to keep track of their progress toward the goal (Prensky, 2001). This typically results in an easier, somewhat intuitive access to learning matter, as it is adapted in a more visual and problem-oriented way than by traditional teaching methods (Pasin & Giroux, 2011). As a side-effect, a number of studies suggested that this leads to a more engaging and motivating learning experience (Garris & Ahlers, 2002; Salas et al., 2009; Myers, 2010).

(12)

importantly, students have to be active throughout the whole learning process, while experiencing a complex decision process that leaves them responsible for their actions (Zantow et al., 2005). Thus, this study proposes that games inhibit a number of advantages over traditional instruction methods for teaching concepts of decision making a SC context.

One point of criticism that has been brought up against the application of serious games is that although some consensus on the positive learning effects of games under certain circumstances seems to exist, little is known about what actually influences their educational outcome. In the context of SCM education it appears trivial to state that, given the high degree of complexity involved, teaching decision making may yield better results in an interactive game environment than through traditional classroom lessons. However, the question which factors define the success in reaching the learning objectives of games in general, and SCM games centered around DM in particular, remains insufficiently explored (Lewis & Maylor, 2007). Certainly, no game can be designed in such a way that it guarantees a positive learning outcome for every single participant. However, whenever students miss to reach these proposed goals it is important to discover why the game failed to deliver the intended experience and how this can be improved in the future. In the case to come an insufficient learning outcome is represented by students who could not reach the level of structured and rational decision making that is needed to perform well in the game. Since the course surrounding the game is particularly built to equip students with tools and techniques that should help them achieving this goal it is necessary to explore why some of them were unable to use them properly, and how it can be facilitated that fewer students will encounter the same difficulties in the future.

(13)

2.4 The MOA-Framework

First mentioned by Vroom (1964) and refined by Blumberg & Pringle (1982) the MOA framework has become an established construct for explaining human behavior over time. In a variety of modifications it is based on explaining individual performance as a function of motivation (M), ability (A) and opportunity (O) (Tuuli, 2012). The framework has been employed to explain a wide array of behaviors such as consumer choice (MacInnis, Moorman, & Jaworski, 1991), firm-level decision making (Wu et al., 2004) or knowledge management practices (Argote, Mcevily, & Reagans, 2003).

Motivation has been defined as the individual willingness to engage in a task and is commonly classified into the constructs of intrinsic and extrinsic motivation (Rothschild, 1999; Garris & Ahlers, 2002). Intrinsic motivation stems from the fact that a task is interesting or enjoyable itself, whereas extrinsic motivation results from the desire to accomplish a certain outcome and is controlled by external influences (Ryan & Deci, 2000). In the context of embedded educational games intrinsic motivation among students may result from enjoyment of the actual gameplay or the perceived usefulness of its contents, whereas extrinsic motivation could be created by the desire to achieve a certain rank or grade (Garris & Ahlers, 2002).

Opportunity is defined as the extent to which external factors limit an individual’s desire to act and to which degree environmental mechanisms enable action (MacInnis et al., 1991; Rothschild, 1999). In a game context this is represented by the conditions of the game and course environment, such as preparation courses, the game interface or student support systems.

(14)

Despite the popularity of the framework itself ambiguity regarding the interrelatedness of its variables exists to the current day. Over time, various attempts have been made to explore their complementarity, being the degree to which one variable depends on the presence of others, yet it remains hard to justify causal relationships theoretically (Siemsen et al., 2008). Little empirical evidence has been brought up to support the complementarity as proposed in existing models, requiring further refinement and application in other fields to overcome these deficiencies (Terborg, 1977; Tuuli, 2012). To enable a comparison between different approaches of complementarity the competing MOA models will briefly be reviewed in the following.

From a traditional work-performance perspective, motivation, opportunity and ability are of moderate complementarity, which implies action to be a multiplicative function of these variables (Maier, 1955; Vroom, 1964; Blumberg & Pringle, 1982). Thus, all variables need to be present to some degree for an action to occur, while lower values of each factor are assumed to strongly reduce action (Blumberg & Pringle, 1982; Siemsen et al., 2008). Similarly, an increase in any of the variables is suggested to have a positive effect on behavior directly but also indirectly through an increase in the other variables. This construct is referred to as the “multiplicative model” which is still a common application of the MOA framework, despite the fact it has never been empirically validated (Tuuli, 2012).

(15)

Although a common understanding on such moderate complementarity can be found in literature, doubts have been expressed on the explanatory power of the multiplicative construct. Cummings & Schwab (1973) agreed that at extreme levels of ability or motivation some interdependency between these variables must exist, since no action can occur if either of them is completely absent. However, especially in those cases where some minimal amount of both motivation and ability among actors can be expected, they propose an additive approach that simply sums up the levels of all factors to work equally well. Thus, they assume the complementarity between the MOA variables to play a subordinate role in assessing individual performance, which in turn gives each construct a bigger impact on its own. This is referred to as the “linear model” (Siemsen et al., 2008).

Figure 2: The Linear Model

(16)

performance if it represents the bottleneck in this case. One advantage of this perspective is that by identifying a constraining factor it becomes possible to set priorities and specifically target potential deficiencies in the attempt to improve performance. In their study Siemsen et al., (2008) demonstrate their model to be superior over both linear and multiplicative models from prior literature and call for further application of the MOA framework as a meta-model in general, and the CFM in particular in a different research context.

Figure 3: The Constraining Factor Model (CFM)

(17)

III. METHODOLOGY

3.1 Overall Research Design

To answer the underlying research questions and to investigate the alternative models an embedded empirical case study has been conducted as the main method of this research. This research design has been suggested to be particularly useful when the goal is to describe the features, context and process of a phenomenon and thereby allows identifying key components of human or environmental systems (Scholz, 2011). Thus, it is a suitable approach to investigate complex real life phenomena such as the underlying mechanisms for rational decision making in a virtual training environment as present in this study (Runeson & Höst, 2009). Due to its explorative nature it further allows to develop relevant theory from an empirical investigation (Eisenhardt, 1989; Eisenhardt & Graebner, 2007). Through the integration of multiple sources of both qualitative and quantitative data a more detailed level of research can be facilitated (Yin, 2009). Thereby, it allows not only to identify factors influencing the students’ decision making process, but also to determine what relations exist between them.

3.2 Case Description

(18)

3.2.1. Gameplay

The UE game is played in self-assigned groups of 3 participants, counting up to a total of 128 teams. Before the start of the actual game, teams are asked to choose a weighting among three performance dimensions: People, Planet and Profit (3Ps). Since the performance in each of these dimensions determines the success in relation to other teams and thus the game-rank, they represent their overall strategy throughout the game. The game runs for a total of 318 periods, each of which represents one day in game time or one hour in real life during the game phase and which are calculated by the computer throughout the other phases. An overview of the game timeline can be found in figure 4.

Figure 4: Game Timeline

(19)

throughout the game phase. An overview of all potential options can be found in Appendix A. Finally, after 218 periods the game proceeds to the end-game phase, where the computer calculates the proceedings of another 100 virtual periods. Thus, teams cannot change their settings anymore and are presented with their final score in the following.

Within the game interface the management dashboard provides the gateway for accessing information such as sales, cash position, demand, stock at distribution centers or finished goods and raw material at plants and allows making decisions in the game based on these parameters. Since the game and the given SC-structure mirror the characteristics of a real, globally operating company, decisions are complex and highly interconnected. Different conditions on demand, prices, capacity or lead times at each of the respective market areas, distribution centers, plants and suppliers result in a multitude of potential configurations. Further, each decision to adjust any of these settings involves a trade-off between the pre-selected performance dimensions. For example, although poor working conditions in one factory lead to lower production costs and an advantage in the profit dimension, their negative humanitarian costs will have a negative impact on the People dimension.

In its core, the game therefore tries to not only make students aware of the complexities of a global SC, but also the need to use mathematical models to deal with them in a structured and rational manner. Whereas at the beginning of the game students might be incentivized to make decisions intuitively due to the multitude of options to choose from, they ultimately should realize that using the tools they were given upfront in the preparation classes help them to perform better in the long term.

3.2.2. Learning Environment

(20)

making progress. An overview of the timing and arrangement of course related activities can be found in figure 5.

Figure 5: Course Timeline

Whereas the lectures, held by the course coordinator, aim to give the theoretical background for these concepts, both the tutorials and practicals focus on how to apply and practice them and require active participation from students. There, students first create mathematical models in the tutorials and later, in the practicals, transfer them to excel to create digital models they can use in the game afterwards. All of these sessions are held by one of ten different teaching assistants (TA), being ten students with a different didactic background ranging from zero to four years of experience. These TAs are also responsible for the design, execution and grading of weekly ungraded assignments which are an integral part of the learning experience of students. As the controlling element of the application-focused classes assignments are supposed to act as milestones to keep all students on a satisfactory level of knowledge by letting them apply the models on practical examples.

(21)

immediately afterwards and is organized in a three hour session, attended by all students, TAs and lecturers. Thereby it should not only be ensured that all students start the game under the same conditions, but that they also could be provided with immediate support and clarifications in the traditionally difficult periods in the beginning.

Throughout the game daily consultation hours are held to provide students with personal advice on a regular basis. These sessions take place in rooms equipped with computers and were guided by two changing TAs each. Students could come by during these sessions without appointments and ask an available TA for any kind of help, ranging from general process related questions to more specific problems such as malfunctioning excel models. Different from last year, having the rooms equipped with computers allowed students to just login to their accounts and show the TAs their specific problem if needed.

Besides the personal support the course offers students a multitude of self-study options in case they would fall behind on skills and knowledge. For all theoretical issues the mandatory course-textbook provides a rich source of information, since the course itself is based on the concepts introduced in it. Students could also rehearse theory through additional video tutorials by the lecturer which were introduced this year. Further, a manual is provided for all game-related issues such as how to access and use the game interface and to provide information about potential decisions and related parameters. Finally, for questions outside of the regular class hours, students could either reach out to one of their teachers via email or post questions in the support forum.

(22)

and the raw performance score for only 20% of their final game grade. However, also underperforming students with little use of models throughout the game are given the opportunity to write a successful report, as long as they could build the models for the report itself and explain how an earlier use could have had improved their final performance.

3.3 Data Collection

Interviews were selected as the main source of information in order to be able to gather the rich data needed for this explorative research. 15 students, covering the whole spectrum of possible performance in both the final game score and the report, set the foundation of the data collection and represent the unit of analysis of this research. Further, 5 TAs, 2 lecturers and 2 game designers, out of which 1 represented both categories, completed the set of interviewees and ensured a well-balanced mix between learners and teachers.

To ensure data reliability an interview protocol for each of the interviewee-categories was formulated along the MOA-variables as presented in the theoretical background of this paper and was adapted after an initial interview to account for missing or unnecessary items (see Appendix B). Questions encompassed aspects of game play and the learning environment and aimed at gaining comprehensive insight into factors that influence students’ decision making throughout the game. To leave enough room for new questions or discussions to emerge, only open-ended, semi-structured questions were formulated. Further, reliability of the interview process was ensured by recording and transcribing the interviews (Yin, 2009). Since some interviewees lacked skills in the English language, interviews were not always transcribed verbatim and thus, adapted for matters of comprehensibility where needed. The interviews were conducted in offices of the University of Groningen, together with a fellow researcher investigating the case from a slightly different theoretical perspective, and with the exception of two students that were contacted via telephone.

(23)

source of information. These additional data not only helped in establishing a starting point for the interviews and the actual field investigations, but also enabled the verification of statements from the interviews at a later stage in the analysis process. Thus, by using these multiple sources of data triangulation could be achieved to ensure data reliability (Yin, 2009). An overview of all sources of data and how they were collected can be found in table 1.

Table 1: Data Collection

3.4 Data Analysis and Quality

(24)

Table 2: MOA Definitions

Based on the definitions as shown in table 2, each factor was further divided into sub-categories. These sub-categories were then operationalized through items that were inductively derived from the interviews, representing their most relevant parts. Accordingly, patterns of student statements or opinions have been grouped through representative phrases and factors that indicate the students’ agreement or rejection with a particular topic or problem (see Appendix C). These interpretive or third order codes can be seen as the link between the factors of the MOA-framework and the actual decision making process of students to explore how the first influences the latter (Voss, 2009). Examples of this process are depicted in table 3 and 4.

Table 3: Coding Excerpt (I)

(25)

report. Combined with the remaining statements this sums up to a level of extrinsic motivation that lies above average in this specific case.

Table 4: Coding Excerpt (II)

This process was applied to all students to explore each of their individual levels of the MOA variables. It was used to analyse how the different levels of the variables interact and, most importantly, benefit or impair the performance of students. Since this method also accounts for inconsistent or even contradictory characteristics also students without a clear tendency towards a certain MOA level could be covered in this respect. This flexibility helped to identify and contextualize the complementarity between factors in each case.

(26)

interviews such as teaching, support, course setup or game design. In a last step, observations, describing the current conditions and acting as the desired control mechanism in contrast to student statements, were summarized and separated from suggestive statements aiming for improvements in the future. The latter were used at a later point in time to formulate an outlook and potential changes to the game in the years to come.

The actual data analysis centred on students’ individual levels of Motivation, Ability and Opportunity and how they explain their respective behaviour in the game and the report. Therefore, data gathered was first analyzed from a student perspective and then compared to data from teachers and concrete performance indicators such as the login frequency, game rank or report grade. By detecting repeating patterns among students a representative picture of the underlying mechanisms of the students’ decision making process could be drawn to aid the evaluation of the different models.

(27)

IV. FINDINGS

Following the data analysis structure as described in section 3.4, the corresponding findings of the analysis are presented in the following section. It is shown how motivation, ability and opportunity influence students’ behaviour and thus, their application of mathematical models for making rational decisions in the case of the UE Management game. First, the general impact of each factor was evaluated based on student data. In a next step, data gathered from teachers and game designers was used to contrast their statements and combined to gain a comprehensive understanding of underlying mechanics for learning throughout the game and the course. Finally, these findings were evaluated in the context of the MOA frameworks as introduced in the theoretical background. Thereby, an individual profile for each student has been created, analysing how the factors work together and determining whether or not a constraining factor exists.

(28)

4.1 Students

4.1.1. Motivation

Table 5: Student Motivation

In line with general observations during the interview process the analysis of students’ motivation shows notable differences. Based on all students’ statements, it could be concluded that the level of motivation was high for 8 students (combined: + or ++), low for 3 students (- or --) and around average for 4 students. Students with a motivation level above average in general were not only willing to invest a high amount of time and effort into the game and the report, but also maintained or increased this devotion over time. Their reasons to do so were similarly extrinsic and intrinsic in nature: Students who were, for example, interested in a good grade also perceived the game to be useful and enjoyable. Further, team dynamics, encouragement through support or the challenge to compete against others could have an impact on these factors. A selection of representative statements of motivated students can be found below.

“We accessed the game for about 15 times a day, also just for quick checks of 5 minutes”

“When I got stuck I tried to figure out a solution by looking in my book or asking the teachers, because I actually wanted to make things good”

(29)

On the opposite, students with low motivation were neither investing enough effort nor inclined to change their behaviour over time. They were rarely interested in their performance and achieving a certain grade, or simply got frustrated due to other circumstances. Similar to the positive case the level of extrinsic and intrinsic motivation showed no notable difference.

“We accessed the game one time per day to see what happened over night”

“We started too late and felt back too much in the beginning, so we got lazy and didn't care about the game anymore at some point”

“I didn’t really care about the grade”

Overall, the students’ level of motivation aligns only moderately with their performance1. Examples for mismatches can be found in cases, where high motivation leads to opposing results in the game rank and the final report (see student #1 or #8) or where high motivation did not result in a good performance at all (see #4 or #10). In a unique, potentially outlying case (#5) a student could also achieve a high grade in the report without any noticeable understanding of the game or motivation to learn about it. Such examples can be attributed to both the game design and the quality of data gathered from students; an issue which will be addressed in more detail in later sections of this paper. Although it is weakened by the small sample size in this case, a more consistent conclusion can be drawn for under-motivated students. Although a high level of motivation did not necessarily result in a good performance, it can be concluded that low motivation mostly lead to a bad performance. This is not only observable through a low rank or grade, but also in a general lack of understanding about mechanics and the actual purpose of the game, leading to an inevitable miss of the learning goals.

1_{Performance in this respect is understood as an intermediate evaluation of students’ actual learning}

(30)

4.1.2. Ability

Table 6: Student Ability

Similar to motivation, also the analysis of students’ abilities yielded mixed results, with a more negative outcome overall. The level of ability was found to be high for 5 students, low for 8 students and neutral for 2 students. Students with high abilities were able to understand the underlying mechanics of the game and the course early on and therefore typically managed to acquire the necessary knowledge for building models and making rational decisions in the game.

“We knew what was coming, so we never felt really overwhelmed”

“The models were not really difficult to make because we already practiced them in class”

“We always used models to change things in the game, I think that was the whole purpose of the game and the report”

(31)

“There was too much information in the beginning, and we were not told what to do with all the data” “We tried to use the models from the tutorials but we didn’t understand them”

“I attended all the lectures, tutorials and so on, but somehow I couldn’t remember the information anymore during the game”

Overall, the students’ level of ability aligns relatively well with their performance. Deviations mostly occur when players with high abilities end the game at a rank that is still sufficient in the context of the game, but could have been higher given their assumed potential. Another notable exception can be found at student #4, which seemingly performed very well with a low level of ability. The explanation can be found in a strict distribution of tasks within his team, which was also present in many other teams. Since this particular student indeed had a very low level understanding of building models and dealing with excel, he was simply not responsible for tasks covering these specific aspect within the team. On the other hand, this also implies that in a well functioning team it is possible to “fly under the radar” and achieve good grades and rankings without the ability to make and explain rational decisions in the game. However, in many other cases students’ level of ability explained their performance relatively well, indicating a high correlation with achieving the intended learning outcome.

4.1.2. Opportunity

(32)

Also the analysis of opportunity shows rather mixed results, with a negative outcome comparable to ability. The perceived level of opportunity was found to be high for only 4 students, low for 9 students and neutral for 2 students. Students with a high score on this factor felt generally well prepared for the game through classes or support and considered the game to be well designed and a useful addition to the course.

“We practiced enough during the tutorials to know the steps for building the models” “The game was a really good ending of the course”

However, many students perceived the learning environment to be insufficient. The most common criticism has been expressed regarding both the transfer process and the applicability of knowledge gained in the classes prior to the game. Students were mentioning assignments and exercises to be too little related to what happens in the actual game and therefore were confused once they encountered the actual tasks when it started. Further, the opportunities at which these examples have been practiced were criticized to be too rare and too guided, leaving students with little room for reflecting on the underlying concepts for applying the knowledge in different contexts.

“I just went to the practicals, went home and forgot everything. We would need more practice with the models” “We had practice, but it was very guided. So we actually didn’t understand what we were doing” “The examples that we worked with were very mathematical and too less focused on how to make decisions”

(33)

“I always felt under time pressure and overwhelmed with information and tasks”

“Sometimes we changed something and weren’t really sure if it had an impact, and then we were already about to make a new decision”

“The environment penalties felt unrealistic. If you did nothing, you just scored high”

The teaching during classes and support surrounding the game received positive acclaim overall, but students still faced various issues. A major point of critique concerned the organization of the consultation hours, which were reported to be overcrowded and thus only of limited help during peak periods. Further, the technical and didactic skills of teachers were mentioned to vary to a notable extent, which in extreme cases even resulted in teachers giving contradictory advises to students.

“There were too many students for too few student assistants”

“Sometimes there were not giving clear answers. But sometimes they just solved the whole problem for us” “So once I made a forecast and asked why my results weren’t as expected. The teacher told me something, but it

didn’t work. So I asked another teacher the next day and she told me my approach was still wrong”

Overall, the students’ level of opportunity aligns relatively weak with their performance. Although in a few cases lacking opportunity can be related to performance (see student #7, #9 or #13), these students were facing even more severe struggles in the dimension of motivation or ability. Further, some students with a low level of perceived opportunity were still able to do well in terms of ranking and grade (see #2 or #12) or vice versa (see #5). Thus, although a multitude of problems was mentioned by students they could not consistently be traced back to their performance, implying that this dimension has smaller direct impact on the learning outcome than others.

4.2 Teachers

(34)

the arrangement of this section deviates from the previous one. Thus, sorted by importance, the teachers’ perception of students’ opportunity, ability and motivation will be analysed in this order.

Generally, teachers were unified in their depiction of the learning environment and an overall satisfaction on its status. Both preparation classes and the gaming environment itself were described as well designed for students, especially since various changes have been made compared to the previous year. Besides an improvement of the accessibility of classes by adapting their content and difficulty, other features have been introduced, such as the kick-off lecture, a guest-lecture and small test after regular lectures to let students reflect on what they learned. The game itself did not change significantly but has also been adapted in its difficulty to make intuitive decisions less successful. Additionally it has been extended by a minor storyline.

“We did not change many things from the game itself, but rather more on the organizational side”

“This year we were trying to improve a lot. We used scrap-sheets to give them instant feedback in the lecture, where they could answer a few questions just after class”

“One way to push students not to go for this intuitive approach was to create a story line for the game, so that the “company” is dealing with lost sales and the task of the students is to find a way to increase the sales ratio”

(35)

without a strict review of results through the teachers, making it possible to finish the preparation classes without gaining a real understanding of the crucial contents. Also, building models was too rarely practiced in the actual game environment or on a similar level of difficulty, leading to a feeling of confusion for the majority of students when all tasks were combined at the start of the game.

“We specifically told them what steps they need to take, so first forecasting demand in order to assess the needed capacity and so on. The problem is that all the information they are given is somehow lost by the time they start

playing the game.”

“We tried to reduce autonomy by telling them what to do. But telling them what to do seems to be the thing that went completely wrong. Maybe they were not ready to hear it.”

“I believe that if all tutorials and practicals would be in a more game-like environment, where they could download data and build models, students would understand the necessity and the importance of using the models in the

game.”

Another recognized issue considered the amount of feedback given by the game. Not only were most data only accessible via the log-file, which needed to be processed by students to look for the desired information, but also by design the game provided feedback rather slow. Since events in the game occurred in a converted real time, long lead times in the game, such as a shipment from a distant supplier, also caused a significant waiting time for students when they made a decision influencing that parameter. Sometimes also technical errors in the game engine caused problems, resulting in disappearing ships or the inability to change configurations. Further, through the scoring mechanism students could – purposefully or by accident – manipulate their game ranking, which partly led to frustration among those who could not “benefit” from it.

“It's sometimes hard to get feedback for students, for example from the excel solver. It produces some vague error message but doesn't tell exactly where the mistake was made in the first place.”

“Overall, more feedback in the actual game itself would be great, but we cannot really implement that in our interface at the moment.”

(36)

Lastly, the support and teaching given by the teaching assistants was subject of controversies among teachers and students. It was commonly mentioned that especially during peak periods the consultation hours during the game were overcrowded. More importantly, teachers acknowledged that some among them were not always giving consistent help during these sessions and especially their regular classes. This might stem from the fact that some TAs are rather inexperienced and are lacking intrinsic motivation and skills to assist students in the best possible way. This was mentioned to be especially problematic during the regular classes, since they account for a huge part of student training.

“The capacity for the consultation hours was not right for peak moment: For the first hour there was a room for 30 people available and around 120 actually showed up.”

“Some TAs should perhaps not be teaching here. They don’t have enough knowledge to see the bigger picture and really teach students in all different aspects.”

However, also well prepared TAs could struggle with assisting students as good as possible due to the fast pacing of classes and limited time during the consultation hours. Some teachers lamented that it was not possible to effectively provide help on an individual level although they would want to explain things on a much deeper level to help students reflect on what they learned. Instead, they would only ensure that at least one person in a group knows how to proceed when a problem occurs and trusted in the group dynamic and self-initiative of students to help each other out to get the necessary knowledge.

“You can only teach “average”, so not helping the ones that are behind but also not giving more for those who want it.”

“I encountered a lot that only one in the group actually understood it. There wasn’t time to let really everybody understand it, and often enough the group was also satisfied with that.”

(37)

4.2.2. Ability

The teachers’ evaluation of ability is directly linked to the perceived quality of the classes and the game environment, since these were the mediums through which ability could be generated for students. In contrast to the statements of especially underperforming students it was commonly noted that students were thoroughly prepared throughout the course and theoretically equipped with all necessary knowledge. Also, the importance of model building and the need to apply these concepts later in the game was explicitly pointed out.

“When you look at the course outline, we provide every possible kind of information to them, every element of the game.”

“Students were told about the importance of models throughout the entire introductory lecture of both the course and the game and also during the lecture about supply chain management.”

Still, it was recognized that students often were unaware of the need to use this theory in practice. Although students were even provided with the templates for all models they had to create for both game and report, some were unable to recognize the multitude of information they were provided with and utilize it for achieving better results. To some extent, teachers were clueless about how this could happen and would not see their teaching and support as the reason for this mismatch. Some of them therefore suggested an even extended practice session prior to the game to be suitable to address these issues.

“I don’t understand why they can’t relate to the tutorials anymore. They worked during the tutorials with those models, so it should be clear for them what they are meant for.”

“We even provided them with the templates for each model. They only need to change the data.” “Having a practice round might really help them; they already get the data anyway.”

(38)

education and the increased demands on being responsible for their success themselves. Also students’ mathematical skills resulting from different school backgrounds were seen as a cause for unequal chances to learn among them.

“I even wouldn’t say that I am teaching university students right now, because it more feels that they still are high-school students, and we actually have to form them to real students over time.”

“Some students wouldn't even know how to log in to Nestor or how to use the coffee vending machine.” “Every year some students lack mathematical skills completely, so we thought about a weekly lecture for really

basic material.”

4.2.3. Motivation

In contrast to students’ opinion teachers see lacking motivation as a major cause for potential under-performance. Although they recognize other flaws as discussed earlier the overall conditions to learn the concepts and achieve a good ranking and grade are considered to be in place. Thus, teachers strongly relate student performance to their level of motivation, and especially see the reason for bad performing students in their unwillingness to spend enough effort and time to work on their problems. Also, failing to achieve a high rank in the game did not necessarily result in bad grade overall, since students could still build the requested models for the report and explain their mistakes and potential improvements resulting from their use.

“I don't always have the feeling students actually gave their best. We provide a lot of information and I think it is highly underused.”

“If they were trying to they could do well overall, you also see that in the reports afterwards. We sometimes knew that they were struggling hard but then they delivered a very good report.”

“I think at least half of the students would get better results if they would try harder.”

(39)

However, also “good” TAs were reported to historically have low performing classes and vice versa, indicating a still limited impact of this issue. Further, the atmosphere inside teams was confirmed to be a major source of motivation. Especially during the consultation hours TAs could recognize whether students tended to dig deeper into a problem together or rather de-motivated each other. Also, most students were de-motivated to end up at a high ranking but weren’t particularly interested in the theoretical work in form of the report that followed afterwards. Teachers noticed this phenomenon for example when talking to students that did relatively well with decisions based on intuition.

“I think overall the role of the TA has a big influence on students’ performance. You have 26 hours of teaching by those TA, so if they are not motivated, how can you be motivated yourself?”

“Team dynamics are also very important for this. Students that were overall motivated pushed each other to do better, while others de-motivated themselves.”

“Most of the students just see the score as the goal and the models as some annoying thing that needs to be done for the report.”

Despite potential motivational flaws of students it has been pointed out by the game designers that ultimately the game itself should still provide enough incentive to engage with the theory in a better way than classical classroom teaching could. Thereby, students would not only spend time and effort for the game to achieve a good grade, but because they genuinely enjoy playing and learning itself.

4.3 Data summary

(40)

mostly limiting themselves. This is in line with the teachers’ perspective, which pointed out motivation among students to be the main indicator for students’ success. Both groups of interviewees acknowledged various problems in the process and design of the game, but were unified in perceiving it to be a useful and overall enjoyable addition to the course. Table 8 summarizes the findings of both students and teachers on which aspects potentially enabled or prevented students to reach the desired learning outcome.

Table 8: Enablers and Barriers of Student Performance

4.4 MOA-Model Analysis

(41)

bottlenecks. Throughout this process a short profile of each student has been created to draw a complete picture for each student and summarize the findings (see Appendix D).

Table 9: Combined MOA Factors

When looking at the course in general, it can be concluded that motivation, ability and opportunity influence each other over a longer period of time and thereby vary in their levels as present among students. For example, a highly motivated student might find it difficult to grasp theoretical concepts for building models due to a lacking mathematical background after leaving high school. Whereas other students might be able to follow the course content by only attending the regular classes, this student might need additional support through the help-forum or video-tutorials. In this case, providing additional opportunities for studying the content might increase the student’s ability to keep up with the content step-by-step. This in turn might result in a further increased motivation to self-study, even higher skills and thus, a student who is more likely to reach the desired learning outcome of the course.

(42)

first place, meaning that their decision making skills and overall performance did not benefit from it. Although in every case it was found that all MOA-factors are present to at least some extent, increasing one factor does guarantee an effect on another.

It is notable that this directly contradicts the linear model as introduced earlier in this paper. Since this model proposes that changing one of the factors will directly result in a change of performance if all factors in general are present to some extent, it can be concluded that it is not appropriate to explain student behaviour in the context of this study. Throughout the data collection and analysis it became clear that levels of single factors could change without noticeably influencing the performance of the student, with the development described earlier as a typical example of this process. Thus, a simple additive approach lacks explanatory power in cases where adding up levels of MOA variables fail to explain the actual characteristics of students.

Similarly, also moderate complementarity as proposed by the multiplicative model was found to not fully capture the characteristics of the data. In theory the model indeed captures the relationships between the variables better than the additive approach, since it accounts for the influence of one factor on another. For example, the absence of any of the factors was shown to strongly reduce performance, which supports the assumption that at extremely negative levels performance is expected to suffer respectively. However, the multiplicative character of the relationships also implies that a change in one factor will always influence the others and thereby the performance or general outcome to some extent. It was found that such continuous adaption not always exists, which is further exemplified in the following section. Thus, also the second model does not account for cases in which changing one variable has no measurable impact on the others or the learning outcome itself.

(43)

power in cases where neither an additive nor multiplicative approach yields sufficient informative value on the learning outcome of students. Thus, students who are limited by a certain dimension, and who cannot improve their performance without improving this particular aspect, can be accounted for in this view.

With respect to the analysis as presented earlier it is most notable that based on student data a constraining factor indeed seems to exist for the majority of them, although not all students share the same factor. Ability has been identified as the most common bottleneck among students and was also frequently mentioned by those that were struggling even harder with other issues. Thus, in many cases missing skill acquisition and therefore lacking knowledge on how to start and proceed with the game were mentioned as the main reason for under-performance. The antecedents and implications of each potential constraining factor are discussed in the following.

4.2.1. Ability

As mentioned earlier, ability was found to be the most common constraint for students to follow the intended gameplay and make rational decisions based on self-created models. In cases where it represents the bottleneck, especially missing skill acquisition and therefore lacking knowledge on how to start and proceed with the game were mentioned as reasons for underperformance. Even if students were motivated to build good models for the game and had more time to play it, they would have been unable to do so without higher skills.

“I was quite motivated but I had no clue what to do in the beginning, so I just did something. That didn’t really work out, but we could only create some models at the very end.”

(44)

Opportunity was found to be the second most common constraint. In cases where it represents the bottleneck, students were mainly unsatisfied with the learning environment prior to the game. Different than for ability, all students with opportunity as their limiting factor ended up with a generally good performance. Thus, it can be concluded that all factors were present to a relatively high degree and that missing opportunities mostly limited good students to get even better. For example, students were frequently mentioning missing opportunities to practice the game upfront. On the one hand, those with only little understanding of the game mechanics (ability) or little willingness to spend extra time (motivation) would perhaps barely benefit from such practice sessions, yet they could take up capacity in the place of those who might benefit. On the other hand, students who are both motivated and able to use a practice session to its potential would certainly increase their performance even further.

“I think we knew quite well what was coming, but we still got overwhelmed by the game at the beginning. [...] A practice round would have been really nice.”

Again, this means that opportunity as a constraining does rather indicate overall good than bad performances but also that opportunity alone does not explain performance in the game sufficiently well.

4.2.3. Motivation

(45)

their point of view mostly motivation limits students throughout both the course and the game. The statements from student #15 support this view, as she, although achieving a good rank and grade, clearly identified her lacking motivation as her personal limitation through self-reflection on her performance.

“I think our motivation was our main problem. If we would have started earlier we could really do more.”

It is therefore assumed that also other students could fall into that category, which were either unable to reflect on their behaviour sufficiently or did not contribute accurate data due to flaws in the interview process.

4.2.4. Summary

(46)

V. DISCUSSION

5.1 General Game Insights

From a general perspective the UE Management game was found to indeed provide a superior learning environment compared to traditional means of classroom teaching. The majority of students explicitly pointed out the beneficial application of theory in a practical environment and the different learning experience compared to other courses, which is in line with the prevalent scholar opinion on the merits of serious games (e.g. Sterman, 1989; Garris & Ahlers, 2002; Sweeney, Campbell, & Mundy, 2010). These students gained motivation from the engaging and competitive experience the game provided, with few of them even stating that the game itself was more beneficial for their future career ambitions than the course overall (cf. Faria, Hutchinson, Wellington, & Gold, 2009). In contrast to critics such as Clark, Yates, Early, & Moulton (2010) the game was generally not only perceived as providing value for entertainment or for increasing social skills, but as a realistic simulation of a SC environment. In fact, both students and teachers pointed out the complex and thus, realistic game setting which was created by the structure of suppliers, factories, distribution centres and the vastness of decision parameters surrounding them (cf. Manuj & Sahin, 2011).

(47)

In various cases it was clear that students could not additionally benefit from this concept of playing a game and reflecting upon their actions afterwards. These cases are in line with the criticism brought up by Gunter, Kenny, & Vick (2008), who argued that games lack the creation of academic knowledge among their participants. However, since the previous analysis revealed that these students typically lack the necessary abilities or motivation to keep up with the course overall, the reason for their struggle can rather be found in the preparation for the game or their own mindset than the game itself.

On the other hand, many students were indeed following the underlying concepts throughout the course and the game. Although similarly struggling in the beginning, they improved their decision making abilities over time. As described in Behavioural Operations Management literature they could thereby reduce individual biases and overcome their bounded rationality (cf. Bazerman & Moore, 2008) by using models instead of simplifying heuristics (Herbert A. Simon, 1997). Thus, playing the game explicitly added value to their learning experience and enhanced their academic knowledge.

It remains a main challenge for designing and balancing the game to also pick up those students who perform below average. One main conclusion from this research is that not necessarily the game itself needs to be the subject of adjustments, but that the course or environment it is embedded in can be even more important for this purpose (cf. Lau, 2015). The difficulty in determining the exact reasons for under-performance in each individual case makes the CFM based on the MOA variables valuable in this context as will be discussed in the following.

5.2 MOA Evaluation

(48)

framework than traditional approaches such as advocated by Terborg (1977) or Blumberg & Pringle (1982). By identifying bottlenecks in student performance it helps not only to determine patterns on what supports or impedes decision making for students but also makes it easier to focus on these areas for potential improvements. The main reason for this can be seen in the restrictiveness of the multiplicative and additive model or the flexibility of the CFM respectively. By design the traditional models define that once a change in one of the factors takes place a change in either the other factors or the performance has too occur automatically. This might explain performance sufficiently in a variety of contexts, but certainly implies flaws in evaluating the behaviour of first-year bachelor students playing a serious educative game. Although it can be summarized that all factors are present to at least a minimal degree, the circumstances under which students play the game lead to an increased unpredictability of behaviour. Apart from struggles with the game and course itself, students encounter them in their very beginning of their university career, while still trying to get along in this new learning environment. Thus, their behaviour can be seen as unreasonable to a certain extent, which is not accounted for in the traditional models at all. In fact, it is even widely recognized that both models lack empirical validation but have not been subject to notable change ever since (Bell & Kozlowski, 2002; Tuuli, 2012). The CFM approach not only also captures such potentially odd behaviour, but also explains expected student behaviour better than the multiplicative and additive approach.

5.3 Practical implications

(49)

target for which group to increase the efficiency of potential adjustments. Thereby, a set of measures that suit the available resources can be evaluated against the background of a bottleneck analysis to ensure a more specific focus on the issues or group of participants they aim to address. Examples of adjustments to the UE management game and the management science course that could be detected throughout the analysis are discussed in the following.

Students performing at an average level or above were overall found to be sufficiently prepared to deal with the challenges that the game brought to them. Thus, the learning outcome of these students is indeed likely to be increased the most by improving the gameplay itself, so that participants are either more challenged or can overcome their problems easier. A suggestion that serves both of these purposes and that has been supported by many teachers is to increase game complexity over time. This could be done by gradually adding parameters to the supply chain such as suppliers, factories or distribution centres. Students would feel less overwhelmed in the beginning and get more involved over time as their skills are challenged further. With an increasing difficulty they could even be confronted with situations that the current version of the game cannot account for, such as unforeseen external events that disrupt their virtual supply chain.

(50)

(51)

VI. CONCLUSION

This paper applied an embedded case study to investigate how motivation, ability and opportunity (MOA) influence decision making in the context of an educationally embedded supply chain management game. The UE management game, played by first-year students in the management science course of the University of Groningen, provided the ground for this research. The findings support the prevalent opinion that all MOA factors influence each other and can thereby explain the decision making behaviour of individuals. Further, this study empirically tested the validity of competing models based on this MOA framework. Due to its flexibility the constraining factor model (CFM) as proposed by Siemsen et al. (2008) was found to be superior in explaining student behaviour than traditional models.

This study makes several contributions to existing theory. First, to the best of the author’s knowledge, the MOA framework has never been successfully applied to explain student behaviour in serious supply chain management games. Thereby it could be empirically shown that by determining students’ motivation, ability and opportunity it can indeed be determined how these factors interact and how this results in certain decision making behaviour. Further, by comparing competing MOA models it could be argued that the CFM yields superior results over traditional models when evaluating student behaviour. This study proposes that this will also be the case in similar contexts such as other games, disciplines or students. However, future research of a qualitative nature might be necessary to substantiate these claims.

(52)

limited resources and design space to make actual improvements from an administrative point of view. As presented in this paper, bottleneck approaches such as the CFM can help in detecting suitable areas for this purpose.

During the research process several limitations of this study have been encountered. Due to the research design as a single embedded case study this paper only covers one particular game of a specific context. Further, given the scope of this research only a certain number of students could be interviewed. Some of them were also lacking English skills, which could have lead to miscommunication in some cases. However, given the depth of the interviews and the setup of conducting them as a team, the potential loss of information could be reduced to a minimum. Also, the single case design enabled a very detailed and in-depth analysis of the chosen case. Due to the systematic interview structure and the transparency regarding both conducting and analysing the data the results of this study are generalizable to other games to a certain extent. Besides factors that are specific to the supply chain context such as the gameplay itself, the analysis of students’ behaviour can also be seen as representative for other games.

Constraining Factors for Decision Making in Serious Games - An embedded Case Study