Does the use of civilian gamer based solutions have added value for
tactical military decision making?
SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF
MASTER OF SCIENCE
Ronald Jongen
10873112
M
ASTER
I
NFORMATION
S
TUDIES
G
AME
S
TUDIES
F
ACULTY OF
S
CIENCE
U
NIVERSITY OF
A
MSTERDAM
August 12
th, 2015
1st Supervisor 2nd Supervisor 3rd Supervisor
MSC. Rudy Boonekamp Dr. Anja van der Hulst Dr. Frank Nack
Does the use of civilian gamer based solutions have added value for
tactical military decision making?
Ronald Jongen
University of Amsterdam Science Park 402, 1098 XH Amsterdam
+31 20 820 8060
ronald.jongen@student.uva.nl
ABSTRACT
Creativity is an important aspect of military decision making. The Dutch department of Defence and TNO are investigating ways to stimulate the creative decision making process among the commanders by means of a game. In this paper, the possibility of using civilian gamers for crowd driven tactical decision making was explored. Including the main research question ‘Does the use
of civilian gamer based solutions have added value for tactical military decision making?’ This was explored in a two phased
experiment: First the civilians (N=14) and military soldiers (N=3) played the crowd driven tactical decision game and filled in a self-assessment questionnaire. In the second phase, three military experts evaluated the solutions. Based on the qualitative results of the evaluations and the self-assessment questionnaire it is concluded that about 42% of civilian based solutions have direct added value for tactical military decision making. However, indirectly, partly rejected plans might have added value as a toolbox for new solutions increasing this amount.
GENERAL TERMS
Pilot, Performance, Experimentation, Human Factors, Theory.
Keywords
Military, creativity, novel combinations, gamers, decision making
1. INTRODUCTION
Strategic decision making is a crucial aspect in military warfare. This is clearly displayed by the many historical battles where armies against great odds were victorious.
The first example is the battle of Vienna, in 1529. Vienna was besieged by 150.000 Turks and had 1.700 defenders [14]. During the siege the Turks tried to tunnel their way in. However, Vienna countered this with by putting basins of water in the cellars, of houses near the wall, which displayed ripples when someone was digging nearby [14, 21, 24]. They used this to find the Turks and stop them.
A more recent example is by General Mohamed Farah Aidid, from Somalia in 1993. He used radio stations to turn the local populace against the UN-forces [23]. During a mission militia shouted to the populace to defend their homes against a raid by UN troops [23]. The result of the battle was 83 wounded and 20 casualties on UN side. On Aidid's side at least 500 militia/civilians died and over a thousand wounded. This was a tactical loss for Aided but a strategic victory because after this UN-Forces departed from the country and Aided proclaimed himself president [23].
Besides the two creative ideas mentioned here, many more historical battles exist, where creative minds saved the day1. It is now time to look at what happens when creativity is not used at all.
In 2006, the Israeli military made use of the adapted strategy known as Effects based operations, which was adopted from the USA. This strategy combines precision bombing of crucial targets to support ground forces. However, the Chief of the General Staff used it as a substitute for land warfare. The ground forces, if they had received training, were trained for raids instead of combat [23]. Due to lack of training 24 tanks were nearly defeated in an ambush when they were crossing a bridge near a jungle. The first tank to cross the bridge was blown up trapping the column on the road. Additionally, none of the tanks used their smoke grenades to protect themselves. This war is an example were creativity was not used at all [23].
The two examples mentioned in the paragraphs above have in common that the commander had a surprising new creative idea, which gave the commanders the advantage that was necessary to achieve victory. By being creative with the tools, units and landscape they had, they managed to overcome great odds. Without creative ideas, these victories might not have been possible.
From the examples given it is clear that creativity is a necessary aspect of strategic decision making. A commander can overcome great odds and achieve victory by thinking of a new creative strategy, but has to tread carefully not to make the same mistakes the commander above made. The Dutch department of Defence recognizes the importance of creative strategic decision making and requested TNO to develop educational tools in order to train battalion and brigade commanders in creative decision making [13]. Currently, the commanders are only trained during large scale exercises and it is possible that commanders have only recently been appointed and lack experience [13]. Additionally, the main focus during these exercises and certification missions lies on the functioning of the entire force, not the individual [13], meaning that a commander can effectively free ride on the experience of their staff. Finally, in these exercises the 'soft' skills, e.g. involving negotiating with the local governments, and civilian story lines are often not adequately represented [13]. Therefore, soft skills are not being trained properly and furthermore it
1
Cambyses II of Persia: The Cat Army [28], Julius Caesar: Battle of Alesia [10], Shah Timur with flaming camels [1] and the Oda-Nobunaga [30] are some of these creative commanders
reduces creativity as one type of unit, e.g. the group of civilians, is missing, limiting the courses of actions to be taken. After investigating and weighing the options to tackle this issue, TNO decided to develop a serious game which is used as an intensive form of experience training that supports skill based training [6] and focuses on the potential of crowd sourcing and peer evaluation.
In the game, called crowd-driven tactical decision game (CDTDG), scenarios are sketched that have a high degree of freedom built in. This means that players can come up with almost an endless amount of solutions. The only limitation is their imagination. In these scenarios, the player will have to think of a tactic to deal with the situation. In the final stage of the game, an expert, who has field experience, will analyze the tactical decisions, evaluate its creativity and viability where after they provide feedback. A detailed explanation of the game is provided in section 4.2
In this paper, it is investigated whether the use of civilian gamer based solutions can add value for tactical military decision making, not the process of creativity itself. In section 2 the related work is presented which forms the basis for the conceptual framework (appendix A). In section 3 the main research question with three underlying questions is introduced and the game CDTDG is explained in more detail. Initial results of a pre-pilot study are presented in section 4. This is followed by the methods used; the encountered challenges and actions taken to resolve them in section 5. The results of the experiment are presented in section 6. This includes the testing of the prototype game with fourteen gamers, four military experts and three military assessors. The results are discussed in Section 7 and finally the conclusion and future work are presented in section 8.
2. RELATED WORK
2.1 Definition of Creativity
Evaluating creativity is possible when a definition is found that fits the problem space. To be able to understand what creativity is, the following work of Boden “The creative mind: Myths and
mechanisms (2004) has been taken into consideration. Creativity
is defined by Boden as: "the ability to come up with ideas or artefacts that are new, surprising and valuable" [4]. This ability is available to everyone because it is an aspect of human intelligence and grounded in the ability such as conceptual thinking, perception, memory, and reflective self-criticism. Furthermore, it is not a question of creative- versus non creative- ideas but rather a question of how ideas are creative [4]. In the book a distinction is made between psychological (P)-creativity and historical (H)-creativity.
An idea is classified as P-creativity when it is a new surprising
valuable idea to the person who thought of it. This is regardless of
how many people thought of it before. However, a new idea is identified as H-creativity when no one, in the known history, has thought of the idea before. A creative idea is therefore made up of the concepts new, surprising and valuable [4]. An idea can be
surprising in three ways; namely, 1) by going against statistics
and therefore being very unlikely, 2) the idea fits into a known style of thinking but the creator did not know the idea was part of it, and 3) the idea is impossible as no one could have thought of it and the creator feels astonishment for thinking of it [4]. Finally, the definition of valuable is unclear as it is subjective to time, people and change of opinions [4].
Boden identified three forms of creativity, namely, exploring
conceptual spaces, transformation and novel combinations [4]. In
exploring conceptual creativity the possibilities within the conceptual space are explored. In this conceptual space there is a limited amount of actions that can be taken. However, this can still be an “astronomically” large amount. These ideas are usually group or culture based. If Israeli general mentioned in the introduction had adapted the strategy correctly he could have explored its usefulness in the war and using it when applicable. When an idea is made by transforming the problem space it is transformation creativity. This means that the creator bends the rules after realizing the limitations of their default methods in order to achieve a goal. An example of transformation creativity is general Aidid who bend the rule of not using civilians in war by turning them against the UN-forces. The last form of creativity described by Boden is novel combination. This form of creativity is about making unfamiliar combinations of familiar ideas. An example of this is the battle of Vienna. Here the defenders combined the idea of tremors causing water ripples with the tremors caused by digging and effectively found the tunnellers. Novel combination creativity matches closely to what is done in military tactics. In military tactics various weapons, military units and physical and human landscapes are combined to achieve a goal. To combine these in a unique way requires out of the box thinking which will be the goal of the game.
2.2 Gamers & Creativity
It is important for officers to learn to think outside of the box [13]. The paradigm is the very strict protocol they normally have to work with. Civilians, however, do not follow military protocol simply because they do not know it and their backgrounds and expertise are different. Therefore, they can think and act differently from military staff in a situation. Now the interesting question arises if this also applies to tactics? This leads to the question, can civilians come up with creative valuable solutions? Military personal follow procedures [13], use the intrinsic context and military hierarchy2 which limits their creative capabilities. Research of bias against creativity even indicates that pursuing creative ideas often involves (risk of) failure and social rejections[18]. These are factors that a soldier does not wish to get involved with and ,therefore, they might have uncertainty-bias against their own creative ideas [18]. To counterbalance this, the proposition is made to add a group that does not follow military procedures, namely civilian gamers. Besides not following military protocol, there are six additional reasons to choose them as the second group, namely: 1) there is a correlation between gaming and creativity. Research has proven that there is a correlation between playing video games and all measures of creativity [15]. The results show that the greater the amount of time spent on playing videogames, the greater the creativity score is, regardless of gender, race or game type. However, it must be mentioned no causality between gaming and creativity was identified and that the test subjects in this research were children. 2) Risk taking has been proven to correlate with specific aspects of creativity. In Glover and Sautter found that risk taking is an
2
"Wolfowitz noted: one infantry officer wrote about the future of armoured warfare, only to have his commander tell him that if he published anything contrary to 'solid infantry doctrine,' he would be court- martialed.It took the intervention of (Gen. John) Pershing's chief of staff to put the soldier's career on a new path. That officer was Dwight Eisenhower." [8]
apparent function of some aspects of creativity [12]. To be more specific, subjects scored significantly higher on the attributes flexibility and originality when part of the high risk taking group and compared to the low risk-taking group. Furthermore, research has also shown that risk taking tendencies of gamers in racing games can transfer to their real world driving [3, 9]. There is also a theory which is linked to the previous one because it has been proven that (online) gamers are more impulsive than non-gamers and have higher risk-taking tendency than non-internet users [31]. This, by using the theory mentioned at point 2, means that gamers could score higher on certain aspects of creativity. 3) Research with 121 adults has found that strategy gamers have a greater propensity for actively open-minded thinking than non-strategy gamers [11].4) Gamers tend to be direct and straight forward. If they disagree with you, they will let you know. Soldiers may not agree but still tend to follow orders. This is important as Van Uhm, former Commander of the Royal Netherlands Army (2005-2008) points out:"Daring to say NO three times to your commander is real loyalty. It is not about who is right but that the right decision is made" [19]. Therefore it can be argued that gamers, who tend to optimize their play, are more likely to say no compared to a soldier who has a different plan in mind. A soldier might be ordered to give up on his idea by his superior, whereas the civilian gamer does not have to worry about the hierarchy and can go on with saying no. 5) "[Strategy gamers] have far more in common with their professional counterparts than either group may imagine or admit [as] no serious gamer plays merely to pass the time [nor does the professional] [20:12]. Most hobby gamers can in fact rival professional wargames in the time invested to [analyse and improve their strategies]. [When] the professional gamers overcome their self-consciousness and prejudice, the professional wargaming community can learn much from wargamers. [The gamers and professionals can then] integrate the lessons learned by both of them and wargaming can continue to expand its capability to educate and entertain the players. [This is believed because:] 'Cross-fertilization of hobby playability and professional realism can help both types of games improve their ability to educate and enlighten their users through the powerful medium of active and absorbing involvement in the challenge of making “life and death” decisions'” [20:7-8]. 6) Finally, there has always been an overlap between the two groups. The game GO, is for example the first wargame ever identified and is played by both groups [20:15]. "What makes wargaming unique is its ability to teach its’ players about war and themselves" [20:12]. During these games one must "analyse the strengths and weaknesses of various dispositions, devise strategies and tactics to overcome the opponent’s strength and compensate for one's own weaknesses while achieving the objective"[20:16-17]. This strengthens the belief mentioned in the 5th item.
There are of course reasons that advise against the use of gamers. The most important reason not to is the Rommel Syndrome. It states that 1) "A person who plays war games is essentially undergoing all of the rigors of a real commander, less the horror of war, and 2) a person adept at war games will be adept at real war. These two forms are the Rommel syndrome: the delusion that each wargamer can become his own Rommel3 and therefore lead his troops to victory" [20]. Gamers are believed to identify too much with the game character, however, this delusion has a
3
Marshal Erwin Rommel famous for the battles won in North Africa during WW2 and the attempted assassination on Hitler.
flaw and it is common sense [20]. This means that if gamers know the implications of their actions and shortcomings there should not be a problem. Furthermore, gamers are not meant to replace the military commanders but to support them with novel ideas. It is up to military commanders to decide what inspires them.
2.3 Measuring Creativity
How to measure creativity is still a considerable point of dispute. There are test like the Torrance test to measure what they define as creativity. However, the Torrance test is built on Guildford's divergent thinking. Divergent thinking is part of creativity but is not the sole attribute [27]. Another problem with these tests is that it is uncertain if these tests even measure divergent thinking [5, 25]. Self-reporting on the other hand is an accepted form of measurement [5, 26]. For this research the self-assessment questionnaire from TNO will be used. This questionnaire has been tested on 1042 students and the constructs have a Crohnbach's Alfa ranging from 0.76 to 0.84 making it very reliable. This questionnaire include six constructs, e.g. curiosity, persistence, resourcefulness, feedback, output orientated, being different and pride[27]. Additionally, it measures space, directions and support felt from the organisation. These could potentially explain the results found in the other constructs depending on the organisations.
3. RESEARCH QUESTIONS
Based on the findings of the literature study the main research question is as follows:
Does the use of civilian gamer based solutions have added value for tactical military decision making?
Before it is possible to answer the main question three sub questions are identified. These need to be answered before a conclusion can be made on the main research question.
3.1 Can gamers create usable creative
solutions for defence training purposes?
When experimenting in an educational setting it is important to not negatively affect trainees. Therefore, it is important to analyse the quality of the solutions made by the gamers. If these solutions turn out to be invalid, unrealistic or unsafe they could negatively influence the trainees. To prevent this from happing it needs to be investigated if the solutions, from gamers, are usable for defence training purposes. To determine this, the solutions were evaluated by three military experts in terms of safety, feasibility, acceptability and legitimacy. Additionally the solutions needed to be based on different approaches and perspectives used from the military without being less safe than military solutions.
3.2 What are differences between the
creative decisions of the gamers and military?
With this research the goal is to find out if gaming can support novel combination creativity by making use of two groups with different backgrounds. In these questions lie the assumption that the different groups come up with different strategies/tactics to resolve a situation. However, it is important to first know if they even differentiate. It is, therefore, necessary to investigate what the attributes of the military and civilian solutions are, where after it becomes possible to compare them and identify the differences. This comparison is made by making use of the self-assessment creativity questionnaire and analysing the evaluations of solutions by the experts of both the gamers and soldiers.
3.3 What is added value of civilian solutions
for military situations?
Before military solutions made by civilians can be used it is important to know if they have added value. The added value is determined by the results of the research questions 3.1 and 3.2 in combination with the creative ideas used in the generated solutions.
4. DEVELOPMENT OF CDTDG
In order to answer the research questions, a game prototype was designed & developed. TNO had already designed a game design prior to the start of this research. This research did provide some of the necessary scientific arguments to convince the stakeholder of this concept and aided in the development of the game.
4.1 Requirements
Based on interviews and workshops with military experts4, six requirements were defined for CDTDG:
1. Individual game play to prevent commanders from free riding [13] and preventing groupthink;
2. Reflection and discussion of actions taken should be a key element of the game;
3. Analysing of other solutions and adapting once own solution should be an element of the gameplay; 4. The game has the option to crowd source challenges to
increase diversity in the solutions.
5. The game has two roles namely player and an evaluator; 6. It has to be a digital game.
4.2 Game Concept
The concept of CDTDG is most effectively described as crowd centred inspiration source for creative tactical thinking. The game concept consists of four modules and five game phases that together provide enough degrees of freedom and support for creative plans to be made. The first module is the scenario description, which includes information about the setting, task to be achieved and the equipment available; the area described in the scenario is visually represented by a map which is the second module (see appendix B). On the map symbols can be drawn to visually represent the actions taken. The symbols available consists of the NATO APP-6C symbolic, which includes all military symbols for warfare, this set was extended with TNO V1227 Symbolic which supports more social, development and diplomatic symbols; The last module are the stakeholders; they were added to further expand the options on the diplomatic front as well as increase soldier civilian interactivity. Players are free to interact with the stakeholders as they see fit, e.g. demanding resources or assigning tasks. This leaves the explanations of the game phases. In phase 1 the players create their solution for the given scenario with the use of the map, stakeholders and symbols whilst writing out their plan; after completing the first phase players analyse the solution of a peer and divide 35 points on a radar chart which is phase 2. This phase is included to provide new insights and provided arguments for the discussion which is phase 3. Admiral Watkins stressed that wargaming needs to be closely tied to operations research and tactical analysis [20]. It is believed that this will stimulate the chance of novel combination creativity. Therefore, it is only logical to include it in the game.
4
retired Lieutenant colonel Henk Oerelemans, Lieutenant colonel Wijnhoud & Rudi Gouweleeuw (TNO).
Once the discussion is completed, phase 4 starts in which the players get a chance to revise their solutions with new insights attained from the discussion. In phase 5, the final phase of the game, an instructor or expert provides their feedback on the solutions created based on field experience.
4.3 CDTDG Prototype
When James F. Dunnigan5 in 1977 was asked what type of wargame he would produce his answer was simple. "It had to be a game the decision makers could easily use by themselves, with minimal training, and obtain believable results immediately. Or as he put it then, “a game the user could play on a computer terminal at home” [20:XVII]. The plan was to build what Dunnigan envisioned and even take it one step further by making it playable anywhere any time. A digital game like this would support participants as it would automatically enforce mechanics, e.g. searching for and creating symbols. However, due to time constraints it was decided to develop a paper prototype first. In this prototype the phases two to five of the game concept were excluded as they were non essential in the answering of the research questions.
The paper prototype provided some additional advantages. It was cheaper, easier and fast to adjust [2], additionally users are more critical of lower fidelity prototypes [2] and one only needs a low amount of testers to test early stages [2]. This was an excellent way to test the prototype and it gave key insights, (mentioned in 4.4), for the actual experiment which also was paper based. The prototype consisted of a General scenario description, the situation description, a map of the city and a set stakeholders (for visuals see Appendix B).Three scenarios were developed6 of which two were used. One was used in the pre-pilot study and one in the experiments. The reason the third scenario was never used was because it was a 100% kinetic scenario, which resulted in a too limited set of actions that could be taken. The map was initially created by hand drawing. However, literature pointed out that accurate maps are crucial for a game made for the military [20]. The final map was therefore created with the aid of Google maps and Photoshop. The stakeholders were added to the game concept during a development session with the project team. The idea is that participants have to organise the stakeholders in groups and assign action to them. This should aid the understanding of the scenario and stimulate a broader course of action than just kinetics, e.g. by influencing relevant stakeholders. This is achieved by giving a face to the "players" involved.
4.4 Pre-Pilot
To determine if the game functions according to plan, a play test with five participants was held. The time limit for this was one hour which turned out to be just doable. From this experiment three noteworthy findings were made.
1) The phrasing used in the scenario has a large impact on the solution generated. Just because of the line "tonight you plan to attack" every single participant decided to attack even after being instructed that they could choose not to. Asking why they chose to
5
James F. Dunnigan, developer of over 100 wargames, military-political analyst, inducted into Academy of Adventure Gaming Arts & Design Hall of Fame
6
This was done by Suzanne van Trijp, captain in the Army Reserve corpse of the Netherlands. Scenario writer for TNO
do so anyway resulted in the response:"it is said so in the scenario".
2) The idea of letting the players create the actor diagram did not turn out the way it was expected. All participants made one and used the actors to increase their understanding of the situation. However, none made extensive use of it for their solution neither did they understand it straight away.
3) The NATO APP-6C Charlie symbols were not used by all but one participant and the V1227 symbols are also only used by one participant (see Appendix B.2D). Despite this three out of five participants said that the V1227 symbols would be easy to use. When asked why they did not use the symbols they said that their simply was not enough time to read them all.
Using the information gained from these participants three changes were made for the remaining 15 participants. The scenario used for further testing was changed to the second scenario. The reason for this was that this scenario offered more degrees of freedom in how to approach the problem. The time limit was removed as all participants felt limited because of it and the use of the symbols suffered due to it. The third and final change was the further filtering of the NATO APP6-C set. The set caused an information overload, by limiting it to actions symbols this was ought to be resolved.
5. METHODS
The foundation of this research is built on qualitative research methods7. These methods included an experiment, two questionnaires and an analysis of evaluated solutions. In the following paragraphs these will be explained in detail.
5.1.1 Questionnaires
Two questionnaires were used in order to get a in depth insight how gamers are different from soldiers and amongst each other. The first questionnaire covers the constructs of creativity. It uses a 5-point Likert-scale ranging from “doesn't fit me at all” to “fits me absolutely”. The second questionnaire (see appendix G) consisted of personal questions on education and work, a few open questions on the game played and questions about gaming experience. This information was gathered in order to get a deeper insight why some gamer solution might be accepted whilst other might not.
5.1.2 Participants
The participants were divided into three groups, e.g. gamers, military experts and military evaluators. The gamers were recruited by means of purposive sampling [22] with the requirements that they play games with a strategic interest. This caused the sample to include more males than females. However, this is not considered to be problematic since the military is primarily made up of males as well. The same technique could not be used military groups due to a scarcity of military with the required ranks. This means that the military experts group are from the required rank. However, the judge group consist of participants who generally are not in direct command on the operational level sketched in the scenario. They are the second in command which means that they support the first in command and have equal or more years of experiences than the first in command. Furthermore, they are all trained instructors which
7
This was decided as the quantities of military personal with the required rank are far too low for Quantitative analysis.
means that they are familiar with the material and they are, therefore, fit to evaluate the solutions. The groups are: gamers, (N=14)8 10 male and, four females, age between 22 and 34 (M=25.87; SD = 3.31), military experts, (N=3)9 including two lieutenant colonels and one major; military evaluators, (N=3)9 including, one captain, one warrant officer, one sergeant-major instruction. Now for grounded theory a maximum impact is achieved with a total sample size between 15 and 25 added together (N=17) so that is within these bounds even after excluding the one participant [17].
5.1.3 The experiment
The experiment was divided into three stages. Firstly the gamers tested the game which took approximately 1.5 hours per player. The first stage was conducted in a semi-controlled environment at a place familiar and time convenient to the gamer. This was mostly at home or at work in a room where they were not disturbed. This choice was based on two facts. Participants did not live anywhere close to the lab and secondly, which is the most important reason, a familiar environment. (Video) Games are mostly played at home and not in a controlled environment. During sessions with multiple gamers at the same time and space, talking was prohibited as well as looking at each others' solution until they were done. Prior to playing the game participants filled in the informed consent and the self-assessment creativity questionnaire after which they played the game and finally they filled in the second questionnaire. After completing the game players were allowed to elaborate their solution and discuss it. This caused many players to be enthusiastic about the game even though at first they were less happy with the amount of reading material. For the experiment protocol see Appendix C.
During the second stage the military experts played the game and completed the creativity self-assessment questionnaire. Again talking was prohibited. The experts were very enthusiastic about the game and provided feedback for improvement including a toolbox idea which is explained further in the discussion
Evaluating the solution usability of both groups is the final stage. To explore if there is consistency among the evaluators and to prevent them from being biased, an evaluation criteria form was developed for the final stage. This was done with the aid of retired Lieutenant colonel Oerlemans and on the bases of the military fundamental principles [16] (For the list of evaluation criteria see Appendix D). Furthermore, in order to prevent compensatory rivalry validity [22] the evaluators were not told which solution was generated by which group. Eliminating potential biases of evaluators is important as J.F. Dunnigan states:"You trifle with wargame validity at your own risk." [20:XX] This quote is given by Dunnigan after given the example of Japanese admirals deeming results of a wargame analysis invalid. Causing them to lose four carriers and naval superiority in the Pacific in 1942 [20]. The evaluation of the 18 solutions was done in two sessions of two hours. During these sessions each judge received the scenario description and the solutions. Again no discussion was allowed
8
Originally (N=15) however one had to be excluded as he was an ex- Lance Corporal making him unqualified for either group. 9
Initially (N=4). However, the fourth evaluator was unavailable and the fourth military solution was unusable, both were rank Major.
during the evaluation of the solution. The evaluators were rewarded for their efforts with 10L of German beer.
5.2 Data Analysis
The results of the self-assessment creativity questionnaire were coded according to questionnaire constructs. The mean per construct was calculated for each participant which was then used to calculate the mean per group per construct (see figure 2 for the results per group). A Shapiro-Wilk analysis showed that the assumption of normality was not violated. After conducting an Independent T-test on the differences between the groups only Pride and Persistence were found to be significant (see appendix F.4). However, with small differences found and as quantitative analysis is not the main goal of this research this is not further explored.
Before analysing the evaluations of the evaluators it was explored if their evaluations matched. To do this the interobserver agreement was determined with the aid of a weighted Kappa analysis [29]. In order to do such an analysis the amount evaluation categories needs to be the same across all questions [7]. The evaluation categories were therefore recoded as described in Appendix E Table 1& 2. The results of the Kappa analysis showed very low scores for the evaluator pairs WO.-SMI, SMI-Capt and WO.-SMI-Capt for all. The WO.-SMI. Agreed fairly (0.227) and WO.-Capt. Slight (0.121) SMI-Capt. both slightly (0.121) (see appendixes F.1). This shows that there is a large difference in opinion on how to evaluate the solutions and implicates a high level of disagreement. Therefore, the remarks made are explored in more detail later. For all quantitative tests a confidence level of 95% was used.
6. RESULTS
The results are split up per research question as each question is one of the stages of the experiment. In section 6.1 the results on the aspects of safety, feasibility, legitimacy and acceptability of the judged solutions are explored in combination with the gamer profile gained from the post questionnaire. The following criteria apply for the aspects. A solution is acceptable when the median is at least 50 and the mean is at least 55, legitimacy is only achieved with a score higher than 3 unless arguments can be found that 2 is also a pass; lastly feasibility is considered achieved with a median value of >50. In paragraph 6.2 the differences between gamers and soldiers are explored using the creativity questionnaire and the generated solutions. In these paragraphs only the results of the acceptable solutions will be explored as the primary criteria of usability is acceptability.
6.1 Usability of Creative Civilian Based
Solutions
The three experts were asked to evaluate all 18 solutions, including both gamer and soldier generated solutions, on safety, feasibility, acceptability and legitimacy. The median was used to explore this as statistical analysis, and it showed a right sided tail in the data10. In Table 1 the results of the Median analysis are
10
Using SPSS the data was tested for skewness and Kurtosis. The skewness values (N=360) are .460 .982 and .561 for the individual evaluators. The Kurtosis values for the evaluators are -1.425 .637 and -1.010.
presented. This shows that two out of the three11 military solutions are acceptable. Of these two both need further explanation on safety, and only one needs more explanation on legitimacy. Lastly in regards to Legitimacy Solution number 3 is not applicable as there is no problem with the plan and for number 1 it is explained that it could use a little bit more detail. From this it is concluded that two out of three military solutions are usable in this sample.
Table 1 Median values of Safety, Legitimacy, Feasibility, Acceptability and Creators acceptability
Safety & Legitimacy Criteria values are 1 = not present but should be or is wrongly used. 2 = needs more explanation, 3 = is present and Adequate, 4 = Present and Excellent, 5 =N/A due to solution
presented or scenario. For Feasibility the values are 0 = Not feasible, <25% = limited, 50% = Possible, >75%= adequately and
100% = completely. Lastly the scale for acceptability is 0 = not acceptable 25 = limitedly, 50 = (NOT) possible, 75= adequately and 100 = completely. In the table the acceptable solutions, based
on median/ mean are coloured green
Solution nr Acceptable Mdn vs. M Safe Mdn Legit Mdn Feasible Mdn Creators acceptability 1 100 / 91.67 2 2 >50% N/A 2 25 / 25 2 2 >50% N/A 3 100 / 83.33 2 5 >75% N/A 6 50 / 58.33 1 2 >75% Unsure 7 0 / 8.33 2 1 <25% Unsure 8 50 / 66.67 2 2 >75% Yes 9 50 / 50 2 2 >75% Yes 10 75 / 83.33 3 3 >75% Yes 11 100 / 91.67 2 3 >75% Yes 12 0 / 33.33 1 1 <25% No 13 75 / 83.33 2 1 >75% No 14 50 / 33.33 1 1 0% Yes 15 50 / 58.33 1 1 >50% Yes 16 0 / 16.67 1 1 0% No 17 50 /50 1 1 <25% Yes 19 75 / 50 1 2 >75% No 20 75 /75 3 1 >75% Partly In perspective seven out of fourteen gamer solutions are acceptable with a mean greater than 55 and a median of at least 50 (Solution 6-8-10-11-13-15-20). It is important to mention however that solution 15 barely makes the cut when looking at the mean and remarks. The solutions 9-17-19 have 50% acceptability when compared to the Mean.
On safety, of the seven acceptable gamer solutions all but two scores worse than the experts' soldier group, which is described in 3.1 as the criteria. One did not work out the details of safety while another brought soldiers willingly into danger with a clear target. The Legitimacy is an important criterion, as going against the law is not acceptable. Out of seven acceptable solutions initially only
11
Solution number 4 of the soldiers was not evaluated due to the fact that the solution only consisted of military symbols.
two scored passable scores on this aspect. However, when the remarks of the judge are taken into account an additional three become acceptable, namely, solutions six, eight and twenty bringing the total up to five. The reason these three were found to be acceptable even though initially scoring bad is that the evaluators mentioned arguments in favour of these solutions, such as that there is a conflict in scenario assignment and description, while none of these were made for solutions nine and thirteen. Finally feasibility is only a problem for one of the seven that pass the acceptability, namely, for solution 15 with one evaluator mentioning that it is wrongly used, one finds info lacking and the last one believing it to be adequate it is not possible to say feasibility can be achieved. The other seven that scored low on acceptability have varying issues such as being inconclusive, too movie like or just plain unrealistic even though some score high on feasibility (dropping pontoon bridges from the sky). Therefore they are excluded.
The results of the evaluation therefore are seven of the gamers' solutions are acceptable, of those seven, five are Safe, seven are Legitimate and five are Feasible. Now to explore these results further the results post game questionnaire will be presented.
6.1.1 Post Game Questionnaire
The experiment was new to the gamers as they generally do not have to solve strategic problems of this kind. To provide additional insight in the created solutions gamers were asked several questions related to the task. The results of the questionnaire are that all found the scenario description clear and that they could emphasize with the given role. However, three out of fourteen did not understand the mission statement fully. Remarks were made that they could have understood it better in a digital game which would automatically enforce the mechanics. The same holds true for the symbolic used, even though, eleven, of which eight partly and three fully, said to have understood the NATO and TNO symbols meaning, only seven used either of them. When asked about the usability eight mentioned that training is necessary because otherwise it take too much time to use the NATO symbols, whereas only one said so about the TNO symbols. It is, therefore, believed that in a more structured manner in a digital game the (correct) use of symbolic could potentially be higher.
Besides asking about the game experience participants were asked to judge their solutions on acceptability. This was done in order to explore if the Rommel Syndrome had occurred. The results of this are shown in Table 1 in the Creators acceptability column. Two participants did not make a verdict as they were not sure about their solutions. However, the other twelve participants did so. When these twelve are compared to the evaluation of the evaluators there are three potential outcomes identified, namely 1) they both agree on the acceptability or limitations of the solution, 2) the evaluators partly agree and 3) disagree with the participant. The results of this comparison are that six both fully agreed, on three they partly agreed and on three the participant made the wrong call. The total completely invalid self assessments are, therefore, three out of twelve or respectively 25%. 25% is a quite large percentage that might suffer of Rommel syndrome. However, what is crucial to keep in mind is that commanders are free to ignore those solutions affected.
6.2 Creative Differences between Soldier &
Gamers
6.2.1 Creativity Questionnaire
The results of the creativity questionnaire show that the average soldier in this sample has a higher score on everything except on curiosity and being different. The difference being -0.07 for the former, and -0.37 (see Figure 1). After conducting a T-test they significantly differ on Pride (α=0.05 P=0,021) and Persistence (α=0.05 P=0,035). It must be noted that a power analysis was not conducted and is most likely insufficient. The implications of this will be discussed in chapter 7.2.
Figure 1 Creativity results
6.2.2 Solutions Creativity
In the solution creativity two major differences between soldiers and gamers were found. First the way they approached the experiment and second the way they resolved the given task. All the gamers asked questions, which ranged from questions about comprehensive reading, to the available equipment, to what actions were allowed. The answers to these questions were not described in the scenario. The soldiers on the other hand did not ask any of these questions. They started immediately on the assignment without asking anything.
The way the groups resolved the task can be divided into three categories, e.g. the level of detail, expertise, and the way they resolved the problems. The solutions created by the soldiers were tight and compact and easy to interpret by the evaluators. The soldier's level of expertise, in the use of symbolic and military terms, certainly contributed to this. Furthermore, soldiers know the framework for creating military solutions by heart which further increases the understand ability. On the other hand, the solutions created by the civilians were often large and shallow on how they wanted to accomplish their goals, making the solutions difficult to understand and giving a negative verdict on usability. The last category of how the problems are addressed is a different story. In the scenario used there were up to fourteen problems built in. Some were crucial and had to be addressed while others that were less important could be resolved to aid with the crucial
problems. None of the solutions of both groups addressed all the problems (see appendix B.3.D). What is interesting here though is not the amount of addressed problems but how they were addressed.
Soldiers were quite careful with placing troops. In contrast, gamers, that did specify troop use, sometimes used all of the available units endangering the normal framework operations (NFO), which are the core, day to day, activities that have to be done. However, in regards to the NFO, the civilians did make use of affiliates they assumed they had built up during the NFO. These connections were not directly mentioned in the scenario but are likely to exist. They then used these connections to influence, to control the local populace and obtain additional resources for the bridge. However, these plans were often evaluated to be too dependent on the local population. The soldiers also used influencing tactics but their plans did not solely depend on the locals. They looked at the provided information, and used that to influence the local population, for example to increase interests in rebuilding the bridge or to get the locals to build the bridge. On how to deal with the bridge the two groups show the largest differences. Three of the four soldiers' plans were to rebuild the bridge and to use locals to do so if available. However, one soldier used boats as a backup plan. Therefore the evaluators found this solution to be the most creative one of the soldier group. From the civilian side six came up with alternatives to building the bridge. These alternatives ranged from building rope bridges, to boats to amphibious vehicles to helicopters to even dropping pontoon bridges from the sky with the aid of parachutes.
7. DISCUSSION
7.1 Usability of Creative Gamer Based
Solutions
In section 3 it was explained that it is crucial that the solutions generated by civilian gamers are safe, feasible, legit and acceptable. A solution is acceptable when the median is at least 50 and the mean is at least 55, Legitimacy is achieved with a score higher than 3 unless arguments can be found then a score of 2 is also a pass; lastly feasibility is considered achieved with a median value of >50. Together these criteria form the measure of usability. When taken the criteria strictly, only four solutions are useable as a subset of the seven deemed acceptable. This follows, as out of 14 solutions only 7 scored a median value of at least 50 in combination with a mean of at least 55% for acceptability. Now out of these seven three solutions have issues. Solution 6 has a safety issue, solution 15 fails on both safety and legitimacy and solution 20 goes against the mandate, therefore, failing on legitimacy as well. However, for solutions 6 and 20 there are mediating factors that make them acceptable. Solution 6 has a safety problem because it was not mentioned in the plan how to handle the safety of the bridge builders. However, the main basis of the plan is to make both clans aware of the fact that the bridge is necessary in order to remove the threat. Legitimacy is a problem for solution 20 as the mandate from the government is to remain neutral. However, both the creator as well as all three evaluators agreed that this is a problem of the scenario because there is a contrast the general assignment of TRUFOR and the scenario assignment of the government. Therefore it is concluded that 6 out 14 (42%) are civilian gamer based solution are valid. Two potential reasons can be thought of why the usability turned out so low, for each reason arguments will be presented if they are
applicable or not namely biased evaluators and the lack of similar environment combined with the lack of structure.
Even though the evaluators could have been biased against the civilian solutions this is certainly not the case. The reason for this is that the precautions taken prevented the evaluators from directly identifying a participant as being civilian. Additionally the evaluators thought solutions six & twenty to be of military personal, where six was thought to be a military engineer and twenty a Lieutenant colonel. Solution ten however, was identified by two evaluators as a "jeans wearer" solution, which is military slang for a civilian, and yet it received better evaluations than solutions six and twenty. Therefore, bias is excluded.
The second reason originates from the lack of a digital game. All civilian participants were video game players and had to play the game on paper. This is a change of environment for gamers that are used to get information visually instead of using literacy. However, the lack of structure is considered to be the biggest explainable cause for the low acceptability. Many civilian solutions scored badly because crucial aspects of the solutions were missing. Additionally the post game questionnaire revealed that the gamers would have preferred to have more support and training for the use of NATO symbols. This can be included into the anticipated digital game as a tutorial.
The results of the evaluations showed that, within this sample, the civilian based solutions have about 42% usability. This amount is too low to state that civilian based solutions are usable for military training purposes in large scales. However, with criteria that reduce the time to filter out useable solutions new creative insights can be obtained. Furthermore the results are limited by the amount of participants. With only three military solutions and fourteen civilian solutions no definitive statement can be made. However, it is believed that a digital game could improve the usability measure. Therefore it is suggested that further testing is done when the digital version of the game is finished.
7.2 Creative Differences between Gamers &
Military
To explore the difference between the creativity of gamers and military two methods were proposed in section 3. These were the use of the self-assessment questionnaire and analysing and comparing the solutions of both groups.
The results of the self-assessment questionnaire were not as expected. Based on the conceptual framework it was expected for the soldier to score lower on the self-assessment questionnaire. However, the fact that soldiers scored higher on all aspects except curiosity and being different can be explained. They are soldiers and they are trained to be persistent, to focus on the task given (output /direction), to collect and share intell (feedback) and take pride in what they do. The gamer group consisted for more than 64% out of students and lacks the training and organisational support that soldiers have. Most noteworthy are curiosity, resourcefulness and being different.
The results show that soldiers are a little bit more resourceful. After having a closer look on the solutions generated by the three soldiers a different result is found. From the soldiers only one made a backup plan in the case that the bridge was not finished on time. The other two did not stop to wonder if the bridge could be finished in the time set, can be related to curiosity. On the other hand five gamers did come up with alternatives to building the bridge. This makes it possible that the resourcefulness of soldiers
might be of a different kind namely working with what they got instead of finding additional resources.
The last major difference in the questionnaire results was “being different”. The fact that soldiers score lower on this was just like expected. During their training they are told not to be different. It is also very often not accepted as the example of Dwight Eisenhower showed.
Now the shape of the gamers in Figure 1 is unexpectedly quite similar to that of the soldiers, but it greatly explains what happened in the solutions. The solutions of civilian gamers were in the amount of problems addressed similar, however, the way they were addressed is quite different. As mentioned in section 6.2 the soldiers used what was known to them and used it to make a plan. This is a form of exploring conceptual space creativity as described in section 2.1. This is not a bad thing though, as there are many actions they could do, but it certainly makes the solutions look quite similar which is what happened. On the other hand the gamers and one soldier made a few assumptions. The gamers more than the one soldier though. This resulted in use of different kinds of creativity and different solutions. By assuming they could use boats or other transports both the gamers and one soldier used transformation creativity. They bend the rules of the scenario which did not state that what was and was not available. Additionally the gamers sometimes used novel combination creativity by combining multiple ideas like building a bridge quickly as using lighter vehicles to cross the bridge.
What needs to be considered is that the soldier sample is too small compared to the civilian group. In the civilian group only three out of seven used transformation creativity. With only three military solutions from which one uses transformation creativity it cannot be excluded that with the same amount of soldiers the results might have been the same. However it can also be that soldiers generating the solutions might be suffering from uncertainty bias. Therefore, it is suggested to do further research with a greater number of soldiers to see if these preliminary results can be confirmed in a large sample.
7.3 The Added Value of Civilian Solutions
In section 3.3 three requirements were mentioned in order for civilian solutions to have added value. Solutions need to be realistic and safe, use a different approach and have a different perspective. As for the former, the gamers and soldiers are not on the same line. Even though both, as mentioned in 7.2, used diplomacy, violence and other means to reach their goals, gamers made more assumptions and, therefore, had more options available to them then the soldier did. It is these assumptions though that got them into problems as well.
In section 7.1 it was discussed that only 42% of the civilian gamer based solutions meet the realistic and safety criteria. This is considered too low to be acceptable on a large scale. However, it is also stated that if the percentage of acceptable solutions is increased that there is potential in the gamer solutions with regards to transformation creativity and novel combination creativity as shown in section 7.2. Furthermore, when a good filter is put in place many new insights can be obtained, shared and used. One of the remarks that came from a lieutenant colonel after playing the game was: "If I had known that I had a pontoon bridge then I would have used it". This is an interesting case because the day before a gamer (solution 7) used a pontoon bridge in their solution even though this solution is not acceptable but for other reasons it opens possibilities. The two lieutenant colonels and
majors later continued with stating that in the game for creative planning making assumptions is not bad and the biggest surprise was they even stated that the solutions do not have to be 100% acceptable. Or as they put it:"It is the thought process that counts not the correctness of the solution". Taken these remarks into account, the added value of gamer based solution rises as eight were discarded because of not being acceptable.
This means that even without this the rejected solutions could have a potential use, namely for filling a toolbox per scenario. One lieutenant colonel said after creating his solution:" I think there should be a toolbox in the game with available units or resources that are useable in the scenario. The toolbox could stimulate more creative ideas". This means that even when solutions of gamers are rejected as feasible there might still be salvageable parts for the toolbox and, therefore, for novel combination creativity.
This experiment, therefore, revealed that currently 42% of the solutions, within this sample, have a full added value. However, there is potential to increase that number by using the digital game or by lowering the acceptability standard. Therefore, it can be stated that in the current state gamer solutions have some degree of added value even though it may only be as part of a toolbox or a full-fledged plan. However, further research is still necessary among the military to determine if the opinion of two lieutenant colonels and two majors is representative for the general military opinion.
8. CONCLUSION & FUTURE WORK
The mission of this paper was to investigate if "the use of civilian gamer based solutions has added value for tactical military decision making" in order to increase the acceptance towards crowd sourcing of military problems and increase creativity by means of this pilot study. To investigate this, the following products were developed: a paper based game, a background questionnaire and evaluation criteria. These were used in combination with an existing a self-assessment creativity questionnaire. The experiment was conducted with fourteen civilian gamers, four soldiers, and three military experts. First the civilians and soldiers played the game and filled in the questionnaire. The three experts evaluated the solutions with the aid of the evaluation criteria.
The analysis for the usability was done with the evaluated solutions and in combination with the background questionnaire. On the basis of this analysis only six of the fourteen civilians' solutions are usable and it is, therefore, concluded that it is not enough. However, there is reason to believe that this amount can be increased by using the digital game and it can be further increased by changing the demand of integral usability as two Lieutenant Colonels stated that the discussion about creative plans is of greater importance than the full-fledged plans themselves. The results of the self-assessment questionnaire for creativity showed higher values for the soldiers on all but two constructs of creativity. Given the fact how the soldiers are trained it is understandable that they scored higher on nearly all constructs except curiosity and being different. The biggest differences were found in the solutions themselves, whereas the soldiers make use of the limited scenario context available the gamers assumed a lot more and read between the lines. The gamers were, therefore, able to use and combine all three forms of creativity in their solutions, whereas the soldiers mainly used one.
The added value of civilian solutions is heavily dependent upon the usability and the opinion of the instructors. The results of evaluators are that only 42% of the solutions are useable. However, the opinion of the instructors differentiates from the evaluators as they mention that the solutions do not have to be fully fledged out and functional but that the discussion and individual parts are important. It is there for concluded that not all civilian based solutions have direct added value for tactical military decision making. However, indirectly parts of rejected solutions might have added value as a toolbox for new solutions. A digital crowd-driven tool like CDTDG could aid in sharing these pieces of inspiration.
The research done was as a pilot study and was, therefore necessarily limited in scope, due to time constraints, an unfinished digital game and the small number of participants available. A follow-up research is, desirable. In that research a fully functional game should be tested. That game should include a tutorial which teaches new players what is expected of them. Furthermore, the game should force and guide players to work out the aspects that are to be evaluated. The tests need to be conducted with a large sample and groups of equal sizes. Lastly the viability of the toolbox idea should be investigated.
9.
ACKNOWLEDGEMENTS
The author would like to thank the following people which aided in the realisation of this paper:
Anja van der Hulst (UvA, TNO), Frank Nack (UvA),Rudy Boonekamp (TNO), Floortje Thönissen (TNO), Hester Stubbé-Alberts (TNO), Martin Shaik (TNO), Nicolien Berkers, Isabelle Lamers and all participants, experts and evaluators involved.
10. REFERENCES
[1] 5 ways elephants changed history: A brief history of stomping victories and disastrous reversals: 2013.
http://www.independent.co.uk/voices/iv-drip/5-ways- elephants-changed-history-a-brief-history-of-stomping-victories-and-disastrous-reversals-8983414.html.
Accessed: 2015-04-08.
[2] Bentley, F. and Barrett, E. 2012. Building Mobile
Experience. The MIT Press.
[3] Beullens, K. et al. 2011. Excellent gamer, excellent driver? the impact of adolescents’ video game playing on driving behavior: A two-wave panel study. Accident
Analysis and Prevention. 43, 1 (2011), 58–65.
[4] Boden, M.A. 2004. The creative mind: Myths and
mechanisms. Routledge.
[5] Boden, M.A. 1996. What is Creativity. Dimensions of
Creativity. (1996), 75 – 117.
[6] Buiel, E. et al. 2012. Inzetmogelijkheden van Serious Gaming voor Public Order Management Training.
Soesterberg, TNO Rapport. (2012).
[7] Cohen’s kappa in SPSS:
https://statistics.laerd.com/spss-tutorials/cohens-kappa-in-spss-statistics.php. Accessed:
2015-08-02.
[8] Defense.gov News Article: Help Wanted: Military Seeks Creative Minds: 2002.
http://www.defense.gov/news/newsarticle.aspx?id=44180
. Accessed: 2015-04-04.
[9] Fischer, P. et al. 2009. The racing-game effect: why do video racing games increase risk-taking inclinations?
Personality and social psychology bulletin. 35, 10
(2009), 1395–1409.
[10] Gallic Wars: Battle of Alesia and Julius Caesar:
http://militaryhistory.about.com/od/battleswarsto1000/p/ alesia.htm. Accessed: 2015-04-08.
[11] Gerber, S. and Scott, L. 2011. Gamers and gaming context: Relationships to critical thinking. British
Journal of Educational Technology. 42, 5 (Sep. 2011),
842–849.
[12] Glover, J.A. and Sautter, F. 1977. Relation of four components of creativity to risk-taking preferences.
Psychological reports. 41, 1 (1977), 227–230.
[13] Hart, M. ‘t et al. 2014. Creative decision making.
[14] History channel THE SIEGE OF VIENNA (1529 A.D). History channel.
[15] Jackson, L. a. et al. 2012. Information technology use and creativity: Findings from the children and technology project. Computers in Human Behavior. 28, (2012), 370– 376.
[16] Land Warfare Centre 2014. Doctrine Publicatie
Landoperaties. Doctrine Commissie Koninklijke
Landmacht.
[17] Marshall, B. et al. 2013. Does Sample Size Matter in Qualitative Research?: a Review of Qualitative Interviews in Is Research. Journal of Computer
Information Systems. Fall (2013), 11–22.
[18] Mueller, J.S. et al. 2012. The Bias Against Creativity: Why People Desire but Reject Creative Ideas.
Psychological Science. 23, 1 (2012), 13–17.
[19] NTR 2015. Voormalig commandant der Strijdkrachten
Peter van Uhm over zijn tijd in het leger. NPO.
[20] Perla, P.P. 1990. The art of wargaming: a guide for
professionals and hobbyists. Naval Institute Press.
[22] Robson, C. 2011. Real world research: a resource for
Users of Social Research Methods in Applied Settings.
[23] Saul, D. 2012. Military Blunders. Constable.
[24] “Siege of the Moles” / How a Christian Serb helped save Vienna: 2001.
http://www.freerepublic.com/focus/news/2424922/posts.
Accessed: 2015-04-04.
[25] Spencer, E. et al. 2012. Progression in Creativity: Developing New Forms of Assessment: a Literature Review. May (2012).
[26] Spencer, E. et al. 2012. Progression in Creativity: Developing New Forms of Assessment: a Literature Review. May (2012).
[27] Stubbé-Alberts, H. and Jetten, A.M. (Andrea) 2014.
Creatief Vermogen constructen na analyse.
[28] The Battle of Pelusium: A Victory Decided by Cats: 2008. http://www.ancient.eu/article/43/. Accessed: 2015-04-04.
[29] Viera, A.J. and Garrett, J.M. 2005. Understanding interobserver agreement: The kappa statistic. Family
Medicine. 37, 5 (2005), 360–363.
[30] Wang, K. 2011. The encyclopedia of war.
[31] Youn, S. et al. 2003. Lifestyles of online gamers: A psychographic approach. Journal of Interactive
Advertising. 3, 2 (2003), 49–56.
List of appendixes
A. Conceptual Framework B. CDTDG Equipment
1. General Scenario Description 2. Pre-Pilot a. Scenario Description b. Stakeholders c. Map d. Symbolic 3. Experiment a. Scenario Description b. Stakeholders c. Map d. Symbolic C. Experiment Instruction D. Evaluation Results E. Restructured Categories F. Statistical analyses 1. Kappa overview
a. Kappa values all aspects b. Kappa values Feasibility c. Kappa values Acceptability d. Kappa values Legitimacy 2. Skewness & Kurtosis
3. Test for Normality G. Post Game Questionnaire
