Goal-Design Alignment Heuristics in Serious Games for
Strategic Level Complex Decision Making
SUBMITTED IN PARTIAL FULLFILLMENT FOR THE DEGREE OF MASTER OF SCIENCE
Alasdair MacLeod
10865500
M
ASTER
I
NFORMATION
S
TUDIES
G
AME
S
TUDIES
F
ACULTY OF
S
CIENCE
U
NIVERSITY OF
A
MSTERDAM
August 14, 2015
1st Supervisor 2nd Reader
Goal-Design Alignment Heuristics in Serious Games for
Strategic Level Complex Decision Making
Alasdair MacLeod
University of Amsterdam Graduate School of Informatics
Science Park 904, Amsterdam alasdair.macleod@student.uva.nl
ABSTRACT
In this paper, 20 heuristics are outlined that describe game design elements that are aligned with the educational objectives of a serious game training for complex decision making at a strategic level.
The serious games market is lucrative, but a high proportion of gamified solutions are predicted to fail in meeting their objectives. Despite this, there are a lack of tools available that assist games designers in aligning the design choices they make with the educational goals of their project.
The Value Centre for Applied Games is a collaboration between TNO, HKU and Dutch Game Garden to develop and promote best practice in serious games design. Using the Goal Design Alignment (GDA) model as a starting point, the VCAG group have prototyped an alignment game that allows games designers to evaluate the goal-design alignment of their game concept. Alignment heuristics sourced from literature and from the experiences of those working within the 3 partner organisations are discussed. 20 final heuristics relating to goal-design alignment in games training for complex decision making are presented. Validation of these heuristics was conducted during playtesting, with players discussing the heuristic statements as part of play and in their evaluation of the VCAG alignment game. This feedback is used to evaluate the ultimate suitability of the heuristics presented. The success of these heuristics is mixed and some are rejected by participants. The success of using the alignment game as a validation tool is also considered; it ultimately provides a useful means of soliciting and capturing discussion regarding the concept of goal-design alignment in general and of the outlined heuristics specifically, but not all heuristics are given equal consideration by players.
KEYWORDS
Games, Design, Alignment, Heuristic, GDA.
1. INTRODUCTION
The serious games market was worth an estimated 1,739.55 million US dollars in 2013 and is anticipated to rise to $2,404.12 million by 2018 [13]. In this increasingly lucrative and competitive market, assessing the extent to which a serious game meets its stated learning objectives in both the short and long term is essential for its ultimate success and for the future development of the serious games industry.
Gartner predicted in 2012 that, by 2014, 80% of (then currently existing) gamified solutions would fail to meet their objectives[5] and identified poor design as being the major factor. Brian Burke, research Vice President at Gartner, states “Poor game design is one
of the key failings of many gamified applications today…the focus
is on the obvious game mechanics, such as points, badges and leader boards, rather than the more subtle and more important game design elements, such as balancing competition and collaboration, or defining a meaningful game economy,…creating gamified applications that are simply not engaging for the target audience”[11]
From this, it is clear that Burke believes the failings of many serious games or gamified applications are down to poor design choices, made without any significant consideration of the effect they will have on players or the contribution they will ultimately make in achieving the learning goals. Successful serious games need to be designed so that their constituent elements are aligned with their defined educational goals in a meaningful and impactful way. Despite this, there are few tools currently available that provide serious game developers with a standardised means of ensuring their product successfully achieves its goals, a problem highlighted by e.g. Mitgusch and Alvarado [19]. Part of the problem faced by the games development community is that games design is a creative discipline that relies heavily on intuition and experience to reach a successful outcomes. Collating and sharing this tacit knowledge of what constitutes best practice is, therefore, difficult to do in a quantitative or standardised sense.
The Value Centre for Applied Games (VCAG) is an initiative jointly supported by TNO, HKU University for the Arts and Dutch Game Garden to promote excellence and best-practice in serious game design, implementation and validation. Part of this project has revolved around the creation of a serious game or game-like tool that could be used to help evaluate how well the ongoing design of a game concept fits its intended learning objectives or goals. The Goal Design Alignment (GDA) model has been used to inform the discussion that has led to the creation of this game. Gameplay uses heuristic statements that players can employ to test the alignment of their game concept. In the original document outlining the GDA model, heuristic statements provided examples and insight into how alignment between goals and design may be achieved. It was agreed at an early stage by members of the VCAG group that this collection of heuristics should be expanded upon and incorporated into the project as a means of demonstrating how design choices can be aligned with a serious games learning goals to increase the likelihood that the finished product will be successful.
Game design is a huge and diverse practice and compiling the heuristics that encompass all types of games, goals and audiences is an equally mammoth task. For this reason, the heuristics used for the purposes of this project primarily concerned educational goals derived from teaching strategic-level complex decision making skills, a subject in which many of those involved in the VCAG project already had extensive expertise.
Additionally, the decision was made to omit heuristics that related specifically to user interfacing. Although user interface design is a vital part of game design, it is an area in which a significant amount of research has already been conducted and incorporating the many well established user interface heuristics available risked eclipsing those relating to other aspects of games design.
2. RELATED WORK
2.1 The Goal Design Alignment (GDA) Model
The GDA (Goal Design Alignment) model [16] was specifically designed to form a framework for the design of serious games. Broadly speaking, this model consists of two elements; the Goal framework and the Design framework. The Goal framework allows for the desired learning goals of a serious game to be defined in terms of the emergent behaviour provoked in participating players and the competencies and experiences that they acquire as a result. The Design framework describes the elements that need to be considered if the game is to ultimately reach its defined goals and is further split into the Big game and the Small game.
The Big game is defined by the GDA model as the context within which the game takes place. In particular, emphasis is placed on didactic embedding (creating context, targets, victory conditions, etc., relevant to the game’s goals) and organisational embedding (effectively the logistics of the game; player numbers, location, etc.).
The Small game is comprised of the structural elements required to make the game itself work; Narrative, the World model, World representation, the rules and mechanics that determine the process and outcome of play and the interface through which players ultimately experience the game.
The model describes the elements that need to be considered when defining the goals of a serious game and the types of design choices that need to be made. What the model currently lacks is the inclusion of any tools or framework that provide guidance to a game’s creators on how to evaluate the success with which a serious games design is aligned to its intended goals, during or after its creation.
In spite of this ‘alignment gap’, the GDA model has proven to be an effective tool when used to deconstruct existing serious games, its constituent frameworks allowing for the isolation and consideration of separate game components in a structured and logical manner. This use of the GDA model for deconstructive purposes has in turn been effective in educating students in serious game design. An example of this deconstruction can be seen in “Example Goal Design Alignment Analysis: DEMOCRACY 3”[18], a document compiled from the findings of Masters students analysing the game ‘Democracy 3’
2.2 Strategic-level Complex Decision Making
in Games
In order to define a suitable set of heuristics for games concerning strategic level complex decision making, it is important to be able to understand what this entails and therefore what the educational goals of a game that provides training in this area should be. Strategic level games Van der Hulst et al describe Naturalistic Decision Making as the main area of research in complex decision making [14]. Within this context, the features of a typical complex decision making setting are a dynamic and continually changing environment, inadequate information (information that is missing, ambiguous, or erroneous), ill-defined tasks and goals, multiple stakeholders, with errors having a heightened impact at a personal
and organisational level. They then identify four core competencies required for effective complex decision making, namely shared situational awareness, situational decision making, communication and coordination.[6]
Heuristics relevant for aligning educational goals to design in this domain need to be useful in recreating an appropriate environment that can be used to train players in essential competencies, using the given definition of complex decision making and the four identified core competencies as a basis.
3. RESEARCH QUESTION
The GDA model lacks any integral component that can be applied to the task of aligning game design to intended educational goals. The GDA model has already proven to be a useful tool for analysing and creating serious games but (in order to become fully effective) heuristic statements need to be used in combination with the model; these heuristics can guide the games creator(s) in ensuring that the design of the final game is aligned to its intended purpose. The main intention of this thesis is therefore to identify and evaluate those heuristics most relevant to serious game design. As stated earlier, the project timescale and expertise of project partners made it advantageous to focus on heuristics for games that train for complex decision making at a strategic level and heuristics relating to user interfacing were omitted
Based on the context of this project, the following research question was posed:
What heuristics can be used in conjunction with the GDA model to help it to fulfil its intended application in allowing the creator(s) of a serious game (training for complex decision making at a strategic level) to better align game design with their intended goals?
Due to the objectives of the VCAG group, it was further intended that these heuristics would be a playable component within a concept for a serious game that could itself be used to evaluate a concept or completed game design’s alignment with its objectives. The playtesting of this ‘alignment game’ was also intended to harvest expert evaluation of these heuristics and would be conducted with games design and serious game experts. It was intended that this game should be complementary to the GDA model but not exclusively so; ultimately it should be relevant and usable to games designers regardless of the methodology they otherwise employ in games design. This posed the following sub question:
Does the inclusion of the identified heuristics in such a game provide an effective means of capturing expert opinion on the validity of these heuristics in relation to serious game design for complex decision making?
4. RESEARCH METHODOLOGY
4.1 Overview
The research questions were addressed using predominantly qualitative methods. The first phase of this consisted of a search through relevant literature to source heuristics that were useful in aligning serious game design with education goals relating to the provision of training in the core competencies and skills required in complex decision making situations at a predominantly strategic level. Again, heuristics regarding user interfacing were mostly omitted, as this is already a significant area of research within games design.
This was combined with anecdotal evidence and feedback from members of the VCAG group and from partner organisations. More
in-depth insight was obtained in the form of an open interview with Rudy Boonekamp, senior researcher at TNO and one of the creators of the GDA model; the intention of the interview was to provide more contextual information on the model and also to gain his insights into the use of heuristics in serious games.
Also interviewed was Ronald Jongen, a researcher at TNO investigating the use of games in promoting creative thinking in planning at a strategic level. His experiences in designing and testing his game concept provided an additional source of validation for heuristics already obtained from literature.
The majority of validation was to be obtained from the playtesting of the serious game being designed by VCAG, as discussion of the alignment heuristics was a central component of play.
4.2 Validation
As stated, partial validation of each heuristic was sought within literature by looking for corroborating examples and evidence for each one
Parallel to this process of literature review and consultation, the VCAG group designed a game concept that would attempt to provide players with a means of evaluating a game-concept in terms of how well its design was aligned to its goals. Members of the design team gave feedback on identified alignment heuristics during this design process, contributing to the selection of heuristics that were ultimately to be used in playtesting.
Playtesting of this game provided the final source of validation for the selected heuristics. Six playtests took place, involving games designers and researchers at TNO, HKU and Dutch Game Garden. Each playtest session lasted approximately an hour. The first session took place at TNO in Soesterberg with 3 TNO staff participating. The second playtest took place at HKU in Hilversum and again with 3 participants. The third and fourth playtests took place simultaneously at Dutch Game Garden in Utrecht and involved staff from TNO, HKU, and independent designers of serious games. The fifth playtest was conducted at TNO again with 2 TNO staff participating. The sixth playtest was conducted by other members of the VCAG group and took place at HKU in Hilversum.
For each session, players were asked to use the game to evaluate a game of their own choosing, with the condition that the game being evaluated had to train for strategic-level complex decision making in some capacity. Comments and discussion on individual heuristics were to be captured as play took place. Players were invited to annotate the heuristic cards and game board used and observations made during play were discussed with the players at the end of each session. It was intended that during playtesting participants would be able to encounter and discuss most heuristics but it was anticipated it some heuristics may not be encountered by players in the time available.
Each heuristic were considered to be accepted if the players unanimously agreed it was valid and useful within the context of strategic level complex decision making, or if they started applying it to their game concept without discussing the heuristics validity. Where there was some disagreement if the heuristic was valid or applicable in the context of complex decision making but it was accepted by the majority of players then it was considered to be partially accepted heuristic.
If a heuristic was rejected by all players, then it would be considered invalid. Ultimately this did not occur, but in some
instances the (well-reasoned) rejection by individual players was so emphatic that the heuristic was considered to be partially rejected.
4.3 The Alignment Game
4.3.1 Design
The game was designed iteratively by the VCAG group over several brainstorming and games design sessions. Several concepts were evaluated by the group before a final concept was agreed upon. This concept was then prototyped in readiness for playtesting.
4.3.2 Game outline.
The goal of the alignment game is to get players (typically game developers) to evaluate the goal-design alignment of a specified game. This game could be either an existing game or a game concept currently in development.
Players have to collectively move from one side of a game board to the other, trying to follow an optimal ‘line of alignment’ as closely as possible. At the start of each turn, a heuristic card is drawn that describes one alignment heuristic. Players discuss the heuristic, focussing on whether it is present (or even necessary) in their game and whether or not it could be applied more effectively.
The players move forward at the end of each turn and the outcome of their discussion further alters the route they take. Players can additionally elect to discard heuristic cards if they are not required or if no consensus can be reached, or create their own heuristics using wildcards.
5. GAME HEURISTIC SOURCES
The term heuristic has several related meanings. The Mirriam-Webster dictionary defines heuristic simply as ‘using experience to learn and improve’[18]. Marsh, Todd and Gigerenzer describe the use of the term heuristic in the title of a paper by Albert Einstein as indicating a point of view that was ‘incomplete and unconfirmed, but nonetheless useful’ and then summarise the work of Duncker and Polya in defining heuristics as ‘useful mental shortcuts, approximations, or rules of thumb used for …making decisions”[12]. The definition frequently used in games design appears to be an amalgamation of these three – knowledge acquired and validated to some extent through experience (but otherwise untested) that is used to facilitate decision making.
Much work has been done to collate heuristics that relate to game design. The emphasis has typically been on entertainment games; Jessie Schell’s ‘Book of lenses’ is structured around a collection of insights and experiences many of which could be described as heuristic in nature[20].
A focus of heuristics relating to games has been interface design and broad principles of game design. The work on heuristics relating to interface design takes methods and principles created initially for web and software design pioneered by individuals such as Nielsen in the ‘90s [4]and extrapolates these to the games domain.
Some studies have been done of heuristics pertaining specifically to serious games. Carmody [6]conducted a Delphi study involving a panel of 12 experts, interviewing them on their experiences of serious game design and then extracting heuristics which were subsequently presented to all participants and evaluated over three rounds. Carmody’s study is effective in soliciting and validating heuristic information acquired by these experts. However, most of the heuristics that emerge are to do with project management and say little about serious game design. That is not to say that they are
without merit, as project management is in itself a complex discipline and the knowledge Carmody captures is no doubt of great use to project leaders in serious game design. What it does not do to any great extent is capture those heuristics and rules of thumb unique to games design and specifically to serious game design. The ‘400 project’ was started by Hal Barwood and Noah Falstein [9]to collate and organise general rules of thumb that could be used to inform game design, based on the assumption that there were at least 400 of these rules in existence. It is unclear how they arrived at this figure, especially as the project seems to have stalled at 112 such rules. There has been also been little activity since 2006, with the two creators now pursuing other commitments, but a spreadsheet is available on Hal Barwood’s personal website that details the progress that was made. These rules have been submitted by the creators and other games designers and heuristic in nature and some attempt has been made to classify these rules based on their relevance to specific areas of game design
Within the 400 project, some consideration given to basic rules relating to serious game design. For example, rule 35 in their list is “Address Needs of Instructors, Teachers and Trainers… Make sure a serious game is easily usable by teachers (and) provide ways to assess learning and make the game customizable to specific curriculums.”[10] This rule is very broad in scope and consequentially provides little guidance and there are four such rules relating to serious games in total.
As it currently stands, the 400 project is not a consistently useful resource. Other (non-serious game) rules include “Emphasise Micromanagement for German Speakers”[10]; this particular heuristic lacks a description but is accompanied by the comment that this should be part of a more general rule to ‘consider national sensibilities’, itself absent from the list. It can be inferred from this and other similar notes that many of the rules have not been researched or developed in much detail and the project as a whole seems to be inactive at this time.
Björk and Holopainen’s work on patterns is a stronger attempt to collect, describe and organise techniques and structures within games design[1]. Although not portrayed or discussed as being heuristic in nature, the nature of many of these patterns and the manner in which they have been collected and validated makes each one a form of heuristic. In their book, they explain that the game design patterns they have collated “rely on general descriptions of particular areas of gameplay without using quantitative measures”[1], matching the definition of heuristic as knowledge gained through experience.
What is impressive about their online collection of 490 (and growing, as observed during the writing of this thesis) collection of patterns is how they are cross-referenced, indicating how one pattern instantiates others, or what other patterns it is instantiated by and what the impact on play and players is. From this collection, it is possible to derive several useful heuristics and to add further weight to those identified from other sources.
Unlike the 400 project, serious games do not appear to be a consideration and the patterns observed and described all appear to have been derived from entertainment games. However, as both entertainment games and serious games share many design principles, there is a great deal of relevant material that can be obtained from this source. These patterns are collected together and freely available within a large online wiki[3] that invites external comments and contributions. Using this source to identify alignment heuristics is not a simple process, as what is described
are the observable patterns within games; the effects on players and on gameplay (and therefore on what players ultimate learn from the game) are discussed in the text accompanying each heuristic. Extracting potential alignment heuristics therefore requires the reader to work backwards and judge the suitability of each pattern as a tool for aligning goals and design.
The advantage that Patterns project has over similar endeavours such as the 400 project is that it is an ongoing project with clearly stated criteria and assumptions applied to potential patterns that are then validated through further observation, testing and discussion. Despite the wide use of heuristics as a means of sharing experience within the games development community, there has been surprisingly little work done on heuristics that provide guidance on what different aspects of game design (mechanics, narrative, interface, etc.) are best used in achieving particular educational objectives and how they can be employed most effectively.
6. ALIGNMENT HEURISTICS
Alignment heuristics were obtained from a variety of sources; from literature, anecdotally and via interview from members of the VCAG group and other members of the serious gaming design staff present at TNO, HKU and Dutch Game Garden and from this authors personal experiences.
Heuristics were considered based on their relevance to designing games that successfully train players in the competencies required for complex decision making discussed in the literature review. Many examples of complex decision making in games can be found in tactics and strategy gaming, so this research is limited to the design of strategic level games.
Wherever possible, when one heuristic was obtained, additional supporting sources were sought.
The following heuristic statements were used within the alignment game.
6.1 Big Game Alignment Heuristics
The following heuristics relate specifically the big game as outlined by the GDA model, referring to those elements involved in the didactic and organisational embedding of the game concept.
6.1.1 Multiplayer
A multiplayer environment is vital in training for CDM.
One of the essential characteristics of complex decision making is the multi-actor environment. It involves collaboration between several stakeholders, or between team members with different levels of expertise and authority. The simplest way to model this environment is to make the game multi-player, with individuals or teams representing the different stakeholders involved.
Incorporating a multiplayer element into a serious game training for complex decision making was one of the first heuristics encountered; throughout literature and in discussion it is considered to be vital in such a game and was provided as an example of an alignment heuristic at the very beginning of the VCAG project. Support for this heuristic can be found in multiple areas. An analysis of the entertainment game ‘Democracy 3’ using the GDA model drew the conclusion that the decision making element of the game would have been more dynamic if the game included a multiplayer element [3]. The multiplayer pattern, as outlined in the patterns wiki, is described as an important source of (for example) social interaction, stimulation for cooperation, coordination and planning and a source of unpredictability. Van Dijen also describes
the inclusion of multiplayer as the ‘best option’ when modelling negotiation and collaboration at a strategic level[8].
6.1.2 Replayability
The game needs to be replayable, with players experiencing new scenarios or outcomes each time.
Experiencing a variety of different situations is important in training for complex decision making. Scenarios, in-game events, outcomes, consequences, and so on, should therefore show some variety (consistent with the education goals and structure of the game) with every replay.
Serious games for complex decision making cannot take place in a static environment:
“For the training of situation assessment, it is essential to confront trainees with as many different settings as possible within a given amount of time…to train for competencies that must adaptively be applied to entirely new situations” [17]
Training for this adaptivity requires that the game environment alters itself or can be altered for high replayability. Replayability also allows for additional knowledge and insight gained from a period of reflection to be applied and tested in subsequent games. Carmody identifies ‘replayability’ as a heuristic that is broadly applicable to serious games in general[6], although the scope of his study does not establish why that might be.
6.1.3 Games master
Game play needs to be managed to ensure that it remains on track and relevant to the educational goals of the game
Given that games involving complex decision making frequently need to let players and teams have a lot of freedom to act and react, this can difficult to express through rules or underlying game mechanics.
A games master can guide narrative, events, play and action. In addition, the games master can coach players in how to proceed and inject an element of randomness and surprise into play. It is possible to implement this with software, or with detailed rules, although a human being brings more intuition, flexibility, adaptability and empathy at a fraction of the cost.
The games master in this context can also be seen to take on the role of an instructor, observing the actions of players and providing expert feedback, essential during or after play [17]. Van Dijen describes the presence of an instructor to lead play for games that train at the strategic level as ‘a must’[8]. Bjork describes the Games master as someone who helps players interact with game worlds ‘so complex that not all can be presented at once’[3]
6.1.4 Briefing and backstory
A briefing can help to provide motivation and context for players and help them relate play to organisational and training objectives.
A briefing is needed that describes the environment in which the game takes place and helps players to understand the relevance to their own training and experience and how it contributes to organisational goals. Briefing also helps to ease players into the game environment and familiarise themselves with basic rules and concepts early on.
This heuristic comes from conversations with TNO staff and the VCAG group. As interaction with (and exploration of) the narrative is a core element of strategic level complex decision making [8] it is important that players are familiar with much of it from the
beginning. Briefing also defines the confines within which players are supposed to operate; as a game on complex decision making will frequently offer its players great freedom of choice, it is important that they understand their mandate as it exists within the game.
6.1.5 Reflection
The game should include time devoted to reflection on game play and outcomes.
There should be an element of reflection on actions and outcomes that occurred during play, so that players can evaluate their success (or lack thereof) and internalise knowledge and understanding of complex decision making situations so that it can be applied in a real-world context.
Reflection is identified as an essential component of training for complex decision making in the NDM literature [17]. A reflective phase of discussion between players participating in e.g. the ‘mayor game’ has been described in conversation as being as important as play itself, the players getting useful and relevant insight into their own decision making skills by comparing their experiences with those of other players.
6.1.6 Player goals
Player objectives and goals should be broad and loosely defined.
The game should provide objectives and targets sufficient to motivate players without providing a clear indication of how to obtain them. These objectives should be broadly defined and there should not be an optimal route to realising the objectives embedded within the game.
This heuristic emerged in early discussions within the VCAG group. It was felt that where there is a clear route to achieving objectives, players are more likely to play in a way that increases their chances of completing or winning the game. The opportunity to learn complex decision making skills could therefore be lost, as players cease applying (and reapplying) decision making skills in a dynamic environment.
In interview, Jongen describes scenario objectives used in a game currently being designed to promote creativity in strategic level planning and decision making. The scenarios presented a logistical bottleneck in the form of a civilian settlement under hostile control. In one version, the possibility of a night attack on the settlement was mentioned and most participants incorporated that attack into their plans. In versions where the night attack was omitted, participant’s suggestions for course of action were more varied, involving negotiation with locals, opening up new logistical lines and so on.
6.1.7 Asymmetric roles
Players and/or teams have asymmetrical roles, targets, actions, etc.
The resulting disparity can be used to encourage negotiation and ultimately cooperation and compromise between players. This can also be used to emulate the differing levels of specialism, influence, expertise, etc., found in real-world settings.
The GDA document [16] describes asymmetry as being usually necessary when training for real-world situations, as well as describing its role in instigating negotiation and cooperation through competition. Schell corroborates this and also describes asymmetric play as being more interesting and thought provoking for players [20].
6.2 Small game heuristics – game economy
The following alignment heuristics all relate to the small game as defined by the GDA model and concern those elements that make up the structure of the game itself. Several heuristics were sourced that covered aspects of the games economy as it relates to complex decision making games.
6.2.1 Economy
Real or abstract resources that influence complex decision making can be represented with an in-game economy
An in-game economy can represent finite physical resources, such as money or material. It can also act as a metaphor for abstract resources that are otherwise difficult to represent, such as time, actions and influence and can encourage prioritisation, collaboration, negotiation and other emergent behaviours.
This heuristic was sourced anecdotally from members of the VCAG and staff at TNO. It is also included in the GDA document. Examples of in-game economies can be seen in Democracy 3[7] (with 3 currencies representing money, political capital and popularity) and Go4It[15], with an in-game economy that simultaneously represents money and actions.
6.2.2 Scarcity
The in-game economy is restricted, so that players have to consider how to use resources most effectively.
A limited economy encourages planning and strategizing as players seek the most effective way to make use of what resources they have. These scarce resources can also increase collaboration and negotiation between players and teams.
Scarcity is described in the GDA document [16] and was mentioned frequently during development of the alignment game. Bjork does not use the term ‘Scarcity’ but refers to ‘Action caps’ as a means of preventing players from being able to do whatever they wish and therefore, encouraging them to plan and similarly refers to ‘resource caps’ as a means of balancing and controlling player behaviour[3]. Personal observations of players taking part in the game Go4it also provides evidence that scarcity motivates collaboration and negotiation.
6.2.3 Savings
The game economy allows players to save resources for use at a later point.
Players should be able to accumulate resources over time or from one turn to the next. This can encourage players to spend more time on planning and observation so that they can implement their plan at a later point.
Saving is also outlined in the GDA document [16] as a means to give a little more freedom in spending. This could be seen to counteract Scarcity but additional limits could be imposed to prevent excessive hoarding or accumulation of resources. Saving is used within the Go4it game partially as a means of encouraging more considered, long term planning to emerge after the initial rush of early rounds and to increase the ability of teams to negotiate and collaborate with each other.
6.2.4 Regenerating resources
Economy resources need to be able to regenerate.
Resources that form the foundation of an in-game economy need to be able to regenerate so that play does not artificially grind to a halt.
Running out of these resources may seem realistic to a designer, but may give play a sense of futility and reduce engagement in players. This heuristic is derived primarily from Bjork [3]who describes it as a means of ensuring pay continues at the desired pace without players completely depleting resources and how it can be used with resource caps to ensure that conditions of scarcity are still maintained.
6.3 Small game heuristics
The remaining alignment heuristics all relate to the small game.
6.3.1 Fidelity
Priority should be given to making the game feel realistic
It is important to focus on what makes the game feel realistic, as opposed to look or sound realistic. At a strategic level, the sense of realism experienced by players is more likely to come from game play and interactions with the scenario or other players and not from (for example) detailed graphics and sound effects.
The concept of functional fidelity, as it relates to training for cognitive skills, is described in the GDA document [16]. Graphical realism is described as expensive and irrelevant in achieving educational objectives at this level and emphasis should be placed on ‘getting the message across’. In the context of serious games for decision making, high fidelity simulation is not necessary:
When (serious games) could provide a so called ‘relevant reality’, i.e. provide the tactical cueing necessary, they were found suitable for decision making in a multi actor setting, at far lower costs compared to the simulators in use up until that moment. [14]
This is further reinforced by Van Dijen, who argues that in a serious game aimed at strategic level cognitive tasks, a feeling of reality is better embodied in a strong narrative, the movement and availability of information and in the actions that players are able to take; detailed graphical representations are unnecessary and non-representative of what would be experienced by strategic-level decision makers in the real world[8].
6.3.2 Fog of war
Players should experience limits to what they can know about the game world.
Even at a strategic level, players should be limited in what information they can have. Information about the scenario, resources, etc. should be limited to what one individual or team could realistically know in a real-world setting. Players should not have access to all information from the start of the game, but should have sufficient information to confidently plan and act on. These limits to knowledge can be expressed visually by obscuring areas of play, or textually through omission and misinformation. ‘Fog of war’ is a concept derived from military games and is a commonly used term in games to describe an absence or concealment of information about competitors and the general game environment. It is therefore analogous to the ‘missing, ambiguous or erroneous’ information that characterises a complex decision making environment. It is described by Bjork [3] and Van Dijen [8] refers to its usefulness in forcing players to make considered decisions.
6.3.3 Consistent scenario
The game scenario and game play should present a consistent and believable environment to players and its relevance must be apparent.
The introduction of fantastical elements during the game should be avoided; if included, they must be present throughout the game and make sense in that setting (e.g. as a metaphor or proxy). Introducing a new element to play that players perceive as nonsensical reduces the feeling of realism (fidelity) and can break immersion.
Van Dijen states that narrative is a primary element in strategic level complex decision making games [8]. Narrative is, in effect, the main source of information about the game environment, objectives and events at a strategic level. For this reason, maintaining a consistent narrative is hugely important. This is echoed by Schell, who suggests that“…the player will be
frustrated, will stop taking your story world seriously and will stop projecting his imagination into it…your world will have changed from a real, live place to a sad, broken toy.”[20]
6.3.4 Control
Players should feel that their actions have consequence and that they are free to determine and implement their own strategy.
This increases immersion and personal investment in complex decision making and its outcomes.
Players have the ability to affect the structure of the game, for example by deciding which actions are possible within game and who can perform those actions. If the game has to define which actions are possible (e.g. via a card deck, or via explicit rules), there should be a significantly high number of different actions available to maintain the illusion of control.
Giving players control (or the illusion on control) over what occurs within the game is a frequently discussed method of increasing immersion and engagement in players. This heuristic is described in the GDA document [16] and it stands to reason that in a game training for decision making, players need to feel that their decisions within the game have an appropriate level of impact.
6.3.5 Excluding goals
Some of the player goals of the game may be arranged so that completion of one makes completion of the other(s) more difficult or impossible.
Incorporating mutually exclusive or inhibitive objectives into the game introduces and element of complexity and the need for prioritisation of goals. If these excluding goals are spread asynchronously between players or teams, it can lead to increased negotiation and cooperation.
This heuristic is described by Bjork [3] and has been observed by the author in Go4it (where players start to collaborate when they realise that no goals can be achieved in isolation, but some can through compromise and negotiation) and in Democracy 3 (when working to appease one part of the electorate frequently alienates others)
6.3.6 Ambiguous information
Event outcomes and narrative information should be open to interpretation by players and should not unrealistically bias players towards a particular course of action.
Players should be free to make their own decisions within the game and apply their own experience, views and morals. They should not be guided towards success via hints or suggestions, as this will increase the likelihood that they learn simply to beat the game, rather than acquire the soft skills required for complex decision making through trial and error.
According to Van der Hulst et al, the competencies required for complex decision, such as shared situational awareness, must be trained for in changing and unpredictable situations [14].
6.3.7 Unpredictable behaviour
Event outcomes and non-player character behaviour should not be too easy to predict by players
Within the confines of the scenario or training situation, events and non-player characters should not be completely predictable. If behaviour and outcomes are predictable, players are more likely to learn the best way to beat the game, instead of learning the soft skills required in a complex decision making environment. This heuristic is derived from characteristics of complex decision making situations such as uncertainty and ambiguity [17]. Again, anecdotal evidence from the VCAG team and staff at TNO and HKU supported the use of a sufficient level of unpredictability (i.e. unpredictable within the confines of the game and scenario, but not completely random) to ensure that players were training real-world skills and not simply learning the patterns of action required to beat the game. Bjork also describes unpredictable behaviour as an instigator of unpredictable outcomes and player performance and also as a factor in supporting the consistency of a game narrative by given players the impression that non-player characters (for example) have their own agenda [3].
6.3.8 Timing
Depending on the scenario and training environment, time must be represented at an accelerated rate.
Strategic level decision making often takes place over long periods of time. Therefore, the passage of time within the game world will need to be accelerated, especially if it is intended that players have multiple scenarios or replays. Play could, for example, be turned based, with each turn representing a set period of time.
This heuristic is referred to in the GDA document [16] and is supported from the observation that games training for complex decision making, the passage of time is nearly always hugely accelerated. In Democracy 3, for example, one turn represents many months. The need to accelerate time in games might be self-evident as it is rare that any game emulating an activity or scenario does so in real-time; even in ultra-realistic big-budget entertainment games, game time usually proceeds at a hugely accelerated rate compared to real-time. However, it is included here as it is important to remain aware that the game environment created will nearly always have to differ from the real world equivalent in this area.
6.3.9 Time pressure
Forcing players to attempt goals within a restricted time frames can heighten tension and frustration and can motivate players to prioritise and delegate tasks.
Including an element of time pressure within the game can help to eliminate so-called ‘analysis paralysis’ where game-play is stalled by player’s taking large periods of time to consider their actions. The tension created by time pressure can also ensure players become more emotionally invested in the game and scenario. This heuristic came from conversations with VCAG members and TNO staff. Bjork refers explicitly to the emotional investment of players experiencing time-pressure and of its use in counteracting analysis paralysis [3].
6.4 Heuristics Rejected During Development
The following alignment heuristics were identified but were dropped before playtesting of the alignment game. In most cases, many of these heuristics had potential but were felt to be too broad in scope or too similar to other heuristics. Some were identified by project members as unusable in the current context. As the game was to be limited to approximately 20 alignment heuristics, these were therefore omitted. A brief description of each and the reasoning behind their removal follows, but the sourcing of these is not discussed for the sake of brevity.
6.4.1 Avatars
Can be used to limit, localise and focus player perception/interaction. Removes omnipotence/omniscience, can be applied to increase functional fidelity.
This was dropped as it confuses the issue of maintaining an appropriate level of fidelity in strategic level games. Although creating a player avatar is not necessarily counterproductive in such a context, it is certainly not necessary.
6.4.2 Balancing Effects
The use of mechanics within a game to steer play, which can be used to hamper success and increase frustration, give struggling players a boost to lift morale and engagement.
This was felt to be too vague and is not a heuristic relating specifically to games training for complex decision making. To some extent, this is covered by other included heuristics, such as ‘Games master’
6.4.3 Entitled players
Players have the ability to affect the structure of the game, for example by deciding which actions are possible within game, or which actions other players can perform.
This was dropped as it is too broad in scope and implication and not necessary in this context. Players should feel free in choosing what course of action they can take within a game, but they should not need to alter the structure of the game.
6.4.4 Game element insertions
The game should not introduce all new elements, scenarios or courses of action at the beginning and these should be revealed as the game progresses. This can be used to maintain an appropriate learning curve for new players, to increase frustration and fidelity and to provide greater challenge as players become more proficient.
Again, this was dropped for being too general and not a necessity in games training for complex decision making. It was also felt it required too much explanation to make it useable within the context of the alignment game.
6.4.5 Illusion of open space
Illusion of open space allows the game space to appear much larger than it actually is and can reinforce a player’s sense of freedom of choice and improve elements of fidelity by allowing focus on small play area while representing broader world.
This was a dropped as it was too specific and it overlapped with ‘Control’ and ‘Fidelity’
7. RESULTS
7.1 Heuristics
7.1.1 Accepted heuristics
During the six playtests, the following heuristics were passed by agreement; fidelity, fog of war, games master.
Realistic and consistent storytelling, reflection, and Briefing and Backstory were accepted on the basis that there was no disagreement. These heuristics were applied to the game concepts used in playtesting without discussion by participants regarding their validity.
7.1.2 Partially accepted heuristics
The multiplayer heuristic was almost unanimously accepted although one play-tester disagreed strongly that it was a heuristic, stating “where is the heuristic? This is an order”. Another playtester pointed out that within a digital context, multiplayer can be difficult and costly to implement, although it was agreed that multiplayer could encompass a broad range of behaviours, such as the sharing of experiences between players during reflection and that the multiplayer element in a digital game did not itself have to be digital.
Player objectives and targets were broadly accepted but in two separate playtests led to discussions regarding the difference between aims, goals, targets and objectives. Rewording of the heuristic name and accompanying text as encountered in the game were proposed to increase clarity.
Economy was broadly accepted by playtesters although it was pointed out that it was not an essential element in serious games modelling complex decision making situations. Scarcity was accepted, but one player questioned if this should be limited to game economies and if its relevance to other aspects of game play should be made more explicit.
Ambiguous information was accepted, but described by one group of play-testers as covering several heuristics; the card referred to reducing bias and player’s freedom to impose their own morals and values.
Timing was accepted, although it was felt that the wording was confusing and that it should be renamed to ‘time compression’ or similar.
Time pressure was also accepted, although again some players debated whether it was actually a heuristic or not. The debate regarding what did and did not constitute a heuristic arose frequently and reflects the different backgrounds and specialism of the playtesters and also of the sources used.
Excluding goals as a heuristic only came up once in playtesting and was accepted but some players questioned if this was a broadly useful heuristic in the context of complex decision making. In the 6th playtest a long discussion emerged with regard to mechanisms supporting the generation of tension between players goals and this aspect was deemed fairly essential.
Replayability was generally accepted, with one playtester stating that a game that has no replayability just isn’t a very good game, in any area. However, others pointed out that replayability may not always need to be considered as the time available to those using serious games to train may be limited, making subsequent replay (and therefore the need for replayability) unnecessary.
Asymmetric roles was accepted by most playtesters. During one playtest, a player questioned its necessity, stating that Chess (a game that he saw as being about complex decision making at a
strategic level) is very much symmetric, with players having equal pieces, moves and options. However, chess does not correspond to complex decision making in a training sense; it certainly presents players with a complex array of decisions, but this alone does not mean it accurately represents a complex decision making environment.
7.1.3 Partially rejected heuristics
No heuristics were unanimously rejected, but some players raised strong objections to their inclusion, making these heuristics candidates for later rejection after further evaluation.
Savings was accepted in several playtests, but in the fifth playtest it was described as a ‘game-breaker’. It was pointed out that designers using this heuristic could create a situation where players can hoard resources until they can act independently, making collaboration and negotiation (central elements of complex decision making scenarios) less likely in some situations, especially if a game economy played a central role in the game concept. Similar reservations were expressed about regenerating resources although it was considered to only really be a significant problem if used in conjunction with the savings heuristic.
Unpredictable behaviour was also rejected by some play-testers, as they were concerned that insufficient distinction was made between behaviour that was completely random and behaviour that displayed variety but that could be anticipated to an extent by players. Behaviour that was too unpredictable would make player choice and actions meaningless, as they would not be able to relate outcomes to the decisions they had made. In the 6th playtest, this heuristic was challenge as well, is was concluded that there should not be a randomness in the effects on the players’ actions, but that there should be an unpredictability in the events that occur during the course of play.
7.1.4 Additional observations and comments
Comments were made that overall some of the heuristics had significant overlap. This was anticipated, especially with regards to those relating to a game economy.
In the sixth playtest, players felt that the identified heuristics should have included one relating to tension. They felt that an appropriate level of tension was essential within a game training for complex decision making to maintain an appropriate level of motivation and pressure on players. They acknowledged that tension would be the by-product of many of the heuristics already described (such as time pressure or excluding goals) but felt that it was sufficiently important to warrant inclusion.
7.2 The Alignment Game as Validation Tool
The alignment game has proven to be an interesting and useful tool for the evaluation of the heuristics described here. Of particular interest is how the game manages to allow the evaluation of the level of goal-design alignment in an existing game concept, but also can be used to harvest the knowledge of games designers who may possess significant expertise in this area. A more traditional approach would have been to approach games designers and researchers individually and ask them to provide feedback on each heuristic; a process which can require significant time and effort and which may gradually lose the interest of some participants, as experienced by Carmody during his Delphi study. The alignment game made it possible to consult the opinion of many experts (averaging three to four with each playtest) at once.
However, the evaluation of heuristics conducted by these experts was frequently lacking in detail. This was partially due to the
nature of heuristic knowledge itself; the experiential knowledge employed by playtesters could readily identify when a heuristic was or was not valid, but often struggled to articulate why this might be the case. The nature of the playtesting, limited usually to one or two hours of game-play, also restricted the amount of time available to players to provide more detailed feedback.
One difficulty prevalent throughout this project was that the professionalism and experience of the games designers meant that their first instinct was to apply their critical skills to the alignment game design itself, especially as it was obviously a work in progress. Whilst their feedback on the alignment game was incredibly useful and will inform the development of future iterations, it was often a struggle to lead them away from discussion regarding the games’ design and onto a more detailed discussion of alignment heuristics.
Another shortcoming of playtesting was that many of those involved had varying interpretations of the term ‘heuristic’. Although the relevant context within which the term was being used was described to participants before each playtest, the variety of backgrounds (education, computer science and games design) caused some focus on the classification of some of the heuristics as such, as opposed to evaluating their usefulness in aligning a games learning objectives with design.
8. CONCLUSIONS
Of the 20 heuristics outlined in this document, those that were accepted or accepted with limitation should be seen as useful heuristics for in goal-design alignment for games training for complex decision making at a strategic level. However, due to the limitations discussed in using the alignment game to solicit expert opinion on these heuristics, further validation of these heuristics should be sought through their continued inclusion in playtests of the alignment game, additional observation and analysis of games and game literature and expert consultation.
Those heuristics rejected emphatically by some playtesters should not necessarily be discarded immediately, as many other playtesters accepted them with no concerns. However, some reassessment of these heuristics should take place through continued validation processes. Also the Alignment Game should allow players to discard heuristics that they feel would not contribute to their specific strategy game design.
The alignment game itself will hopefully continue to be developed, allowing for continued evaluation of these and other heuristics as a well as providing an increasingly effective tool for games designers. As observed during playtesting, players were often more interested in the mechanics of the alignment game than the alignment heuristics that they encountered.
However, this will hopefully diminish as the game continues to be developed and becomes more polished, both in terms of play and appearance; as the game begins to look and feel more complete, the discussion surrounding it should begin to focus more on the game concept being evaluated and the usefulness (or lack of) the alignment heuristics themselves.
As previously stated, players would additionally disagree amongst themselves as to what constituted a heuristic. For this reason, it is recommended that in future studies the title ‘heuristic’ is abandoned in favour of a more specific term that would help guide the conversation back to goal-design alignment. For example, the term ‘Alignments’ could be used; this would have the effect of making clearer to users what their function is and would hopefully serve to frame any surrounding discussion in these terms. It would
of course be necessary to make clear to users that these ‘Alignments’ are heuristic in nature, but it would not otherwise be necessary to label them as heuristics.
Re-naming these heuristics as ‘Alignments’ may also help with their inclusion and application in the context of the Goal Design Alignment Model, helping games designers better understand the relationship between Goals, Design and the role these alignments can play in marrying the two together.
9. REFERENCES
1. Björk, S. and Holopainen, J Patterns in Game Design. Hingham, Charles River Media, 2005
2. Björk, S., Lundgren, S., and Holopainen, J. Game design patterns. Proceedings of Level Up: Digital Games
Research Conference (Utrecht, 2003) 180–193.
3. Björk, S. 2015. Games design pattens wiki. Retrieved 16th June 2015 from http://gdp2.tii.se/
4. Brown, M. Evaluating Computer Game Usability : Developing Heuristics Based on User Experience.
Proceedings of IHCI conference (Cork, 2008)
5. Burke, B. Gamification 2020: what is the future of
gamification? Gartner, 2012. Retrieved May 18 2015,
from
https://www.gartner.com/doc/2226015/gamification--future-gamification
6. Carmody, K. Exploring serious game design heuristics : a delphi study. Educational Doctoral Theses Boston, Northeastern University, 2012
7. MSc INFORMATION STUDIES (GAMES STUDIES)
Example Goal, Design Alignment Analysis:Democracy 3,
Amsterdam, University of Amsterdam, 2014
8. Van Dijen, F.S.G. Serious Game Design Guidelines for
the Military Training. Amsterdam, University of
Amsterdam, 2014.
9. Falstein, N., and Barwood, H. The 400 Project. FiniteArts, 2006. Retrieved on 24June 2015, from http://www.finitearts.com/Pages/400page.html
10. Falstein, N., and Barwood, H 2006. The 400 Project Rule
List. FiniteArts, 2006. Retrieved on 24 June 2015, from
http://www.finitearts.com/Pages/400page.html
11. GARTNER. Gartner Says by 2014, 80 Percent of
Current Gamified Applications Will Fail to Meet Business Objectives Primarily Due to Poor Design.
Gartner 2012. Retrieved August 14, 2015 from http://www.gartner.com/newsroom/id/2251015 12. Gigerenzer, G., Todd, P.M., and Marsh, B. Cognitive
heuristics; reasoning the fast and frugal way. In Leighton, J., and Sternberg, R.. ed. The Nature of reasoning. Cambridge, Cambridge University Press, 2004. 13. Greer, T. The 2013-2018 Worldwide Game-based
Learning and Simulation-based Markets. Serious Play
Conference, Ambient Insight, 2014. Retrieved September
14 2014 from
http://www.ambientinsight.com/Resources/Documents/A mbientInsight_SeriousPlay2014_WW_2013_2018_Game BasedLearning_Market.pdf
14. Van der Hulst. A., and Ruijsendaal, M. Serious Gaming for Complex Decision Making. Proceedings of the 1st
International Workshop on Pedagogically-driven Serious Games (Saarbruken, 2012) 51–60. Retrieved on 28 May
2015 from
http://ceur-ws.org/Vol-898/pdsg8.pdf\nhttp://ceur-ws.org/Vol-898/ 15. Van der Hulst, A., Boonekamp, R., and Van den
Homberg, M. Field-testing a Comprehensive Approach simulation model. International Confernence on
Information Systems forCrisis Response and Management (PennsylVania State University, 2014)
16. Van der Hulst, A., Boonekamp R., Oprins, E., and Visschedijk, G. Goal-design alignment. Soesterberg, TNO, 2014.
17. Van der Hulst, A., Ruijsendaal, M., Mullet, T.J., Buiel, E., and Van Gelooven, D. Serious gaming for complex decision making: training approaches. International
Journal of Technology Enhanced Learning, 2014. 6, 3,
249–264.
18. MIRRIAM-WEBSTER. 2015. Heuristic.
Mirriam-Webster Dictionary (online). 2015. Retrieved June 20,
2015 from http://www.merriam-webster.com/dictionary/heuristic
19. Mitgutsch, K., and Alvarado, N. Purposeful by Design: A Serious Game Design Assessment Framework.
Proceedings of the International Conference on the Foundations of Digital Games, (2012) 121–128. h
20. Jesse Schell. 2008. The art of game design: a book of
lenses. Burilington, Morgan Kaufmann, 2008,
