Publisher: Taylor & Francis Journal: Ergonomics
DOI: http://dx.doi.org/10.1080/00140139.2015.1076059
Assessing the Structure of Non-Routine Decision Processes in Airline Operations Control
Floor Richters (corresponding author)
University of Twente, Ravelijn 3426, P.O. Box 217, Enschede, the Netherlands. E-mail: f.richters@utwente.nl. Phone: +31612971613.
Jan Maarten Schraagen
TNO Earth, Life and Social Sciences, P.O. Box 23, 3769 ZG Soesterberg, the Netherlands. Email: jan_maarten.schraagen@tno.nl. Phone: +31888665945
Hans Heerkens
University of Twente, Ravelijn 3426, P.O. Box 217, Enschede, the Netherlands. E-mail: j.m.g.heerkens@utwente.nl. Phone: +31534893492.
Assessing the Structure of Non-Routine Decision Processes in Airline Operations Control
Unfamiliar severe disruptions challenge airline operations control professionals most, as their expertise is stretched to its limits. This study has elicited the structure of airline operations control professionals’ decision process during unfamiliar disruptions by mapping three macrocognitive activities on the decision ladder: sensemaking, option evaluation, and action planning. The relationship between this structure and decision quality was measured. A simulated task was staged, based on which think aloud protocols were obtained. Results show that the general decision process structure resembles the structure of experts working under routine conditions, in terms of the general structure of the macrocognitive activities, and the rule-based approach used to identify options and actions. Surprisingly, high quality of decision outcomes was found to relate to the use of rule-based strategies. This implies that successful professionals are capable of dealing with unfamiliar problems by reframing them into familiar ones, rather than to engage in knowledge-based processing.
Keywords: decision making; airline operations control; fractionated expertise; macrocognitive activities
We examined the macrocognitive structure of airline operations control professionals’ decision process during a simulated unfamiliar disruption in relation to decision quality. Results suggest that successful professionals are capable of dealing with unfamiliar problems by reframing them into familiar ones, rather than to engage in knowledge-based processing.
Introduction
In the dynamic environment of Airline Operations Control, managing the network of schedules in the face of unexpected events is the main task of decision makers. Low impact disruptive events happen relatively often, and most operations control decision makers are experienced professionals who have dealt with such disruptions many times. High impact disruptions on the other hand, don’t occur frequently. This study focuses on the decision process of professionals during such disruptions at one airline. Most
studies suggest that expertise is adaptive, implying that Airline Operations Control professionals should be able to deal effectively with novel situations. However, operations control management of the airline in question has observed that under comparable non-routine circumstances decision outcomes vary considerably case by case. This is problematic, because wrong decisions under highly disruptive
circumstances can lead to loss of control, resulting not only in potentially high costs as flight delays accumulate, but also in reputational damage. Moreover, insight in the decision making process itself is lacking, and hence Airline Operations Control management believes they have no control over the decision process. To be able to effectively support and improve this process, in the present study we address the following question: what is the structure of the cognitive decision process of
professionals when dealing with unfamiliar situations, and how is this structure related to decision quality?
In this article, we will first outline how this research fits into the current body of naturalistic decision making (NDM) literature by describing the general context of airline operations control decision making under non-routine circumstances, after which we will elaborate on the expertise of professionals in operations control. Next, we present the model used to study decision processes. We will then elaborate on the methodology used, and finally results are presented and discussed.
Decision Making in Operations Control Disruption Management
Disruption management decision making in Airline Operations Control has mainly been studied from the perspective of developing decision support systems based on
mathematical grounds (see for example Abdelghany, Abdelghany, and Ekollu 2008; Bratu and Barnhart 2006; Clausen et al. 2010). However, such an approach not only fails to appreciate the complexity of the Airline Operations Control work environment,
but also ignores the underlying cognitive decision making processes (Bruce 2011; Feigh and Pritchett 2010).
The study of professional decision makers in their natural environment is typical of NDM research, especially when professionals need to make decisions under complex or difficult circumstances (Ross, Shafer, and Klein 2006). Many of the contextual factors defined by Zsambok (1997) that are considered ‘difficult circumstances’ apply to operations control decision making at an airline. Airline Operations Control
professionals often operate under high uncertainty as unexpected events, like internal failures such as maintenance or crewing issues, or external events, such as inclement weather, regularly disrupt the planned flight schedule (Ball et al. 2007). Multiple solutions are usually available to decision makers (Yu et al. 2003), but these often require making conflicting trade-offs between the main elements of an airline operation: passenger flows, aircraft and crew (Kohl et al. 2007). Inevitably this puts a strain on the airline’s profit margins, as disruptions will often lead to delays or cancellations
(Thengvall, Bard, and Yu 2000), which can result in excessive delay costs (Wu 2005). Also, the airline’s reputation is at stake if Airline Operations Control professionals fail to make the right decisions, and instead risk the airline to make the national headlines.
Although disruptive events happen quite often, their impact is usually limited to a small number of flights. Moreover, Airline Operations Control professionals are familiar with most events. Such disruptions include for example delays caused by unexpected aircraft maintenance or ATC delays at a destination airport. The current research is concerned with disruptions of a different kind; those that do not happen very often and hence are relatively unfamiliar, and that can potentially have a great impact on the flight schedule. In this study, this is considered to be the case when delays for the flight schedule on average add up to more than one hour per flight, as decision
processes under this condition are distinctly different from decision processes during solving individual disruptions. Disruptions like these would include delays of a large number of flights extending over multiple arrival and departure peaks within the network schedule, caused by irregular adverse weather conditions in the airport area. Note that, even though they do have a significant impact on the flight schedule, emergencies (for example, a crash) and contingencies (for example, events that cause political instability in certain flight destinations and result in severe safety issues) were excluded from this research, because the studied airline’s procedures and decision making processes under these circumstances extend beyond the boundaries of the Airline Operations Control departments.
Expertise in Operations Control
It is generally acknowledged that expertise is domain specific. However, what is often overlooked is that expertise is also not necessarily generalizable across different tasks (Shanteau, 2012). Although experts still have the ability to use certain types of decision strategies when working outside their area of expertise, such as mental simulation and progressive deepening, the quality of experts’ solutions is reduced to the level of beginners (Schraagen, 1993).
Bruce (2011) investigated for several types of disruptions how gaining initial situational awareness and generating various courses of action is affected by airline operations controllers’ experience and expertise. Experts were identified based on years of domain experience. Although this is an important predictor of expertise, it is not necessarily sufficient to reach an expert level of achievement (Ericsson 2006; Feltovich, Prietula, and Ericsson, 2006; Shanteau et al. 2002), especially under non-routine conditions. When confronted with high impact disruptions, those operations controllers classified as experts might not always be functioning on an expert level per se, due to the infrequent
occurrence of these types of disruptions, the unstructured nature of the environment, and consequently the non-routine character of the task at hand. As professionals are confronted with relatively few opportunities for developing expertise, amplified by the lack of feedback on the quality of decisions taken, they lack deliberate practice, which drives development of expertise (Ericsson and Charness 1994). This may result in ‘fractionated expertise’, as professionals will show expertise in some but not all of their activities (Kahneman and Klein 2009).
Fractionation of expertise is common in actual work situations, due to job rotation, lack of feedback in complex work environments, and increased pressure to develop expertise quickly. Yet, although this phenomenon is common in practice, it has been virtually neglected in most studies of expertise that assume full-time dedicated attention over a long period of time (for a review, see Feltovich et al. 2006). Hence, the current research aims to acknowledge the task specificity of expertise, by studying experienced
professionals who operate under conditions of fractionated expertise as they are confronted with unfamiliar disruptions within their domain.
Decision Process Structure
Multiple models of NDM have been developed that aid the elicitation of the cognitive structure of the decision process. The Recognition Primed Decision (RPD) model of Klein (1993) and Rasmussen’s decision ladder (1976) are the most frequently used (Lipshitz 1993; Naikar 2010). Several researchers have proposed models that integrate parts of these two models (see for example Hoc and Amalberti 1995; Lintern 2010; Vicente, Mumaw, and Roth 2004). Of these, we considered the modified version of the decision ladder by Lintern (2010) to be the most suitable to represent both implicit as well as explicit decision processes (Lintern 2009). This was an important requirement because we expect Airline Operations Control professionals to use both types (Bruce
2011). Moreover, the decision ladder can represent decision making under unfamiliar or difficult circumstances (Naikar 2010), and was therefore chosen for our analysis as a suitable model for representing professionals’ decision processes. The decision ladder is described here based on Lintern’s (2010) terminology of cognitive states (states of knowledge) and cognitive processes (information processes).
Originally, the decision ladder was intended to function as a formative model that displays which cognitive processes might be used rather than those that are used (Rasmussen 1986), and many researchers use it as such (see for example Jenkins 2012; McIlroy and Stanton 2015; Naikar, Moylan, and Pierce 2006). Other applications include, for example, analysis of complex socio-technical systems (Stanton & Bessell 2014), analysis of team cognitive work (Ashoori et al. 2014), and development of decision aids in healthcare (Dhukaram & Baber 2015). The decision ladder also has descriptive value, especially when used as a template with which to analyse verbal protocols (Vicente 1999). Our approach focuses on description and analysis of individual decision making behaviour, thereby explicitly linking the information processing steps on the ladder to macrocognitive activities of decision makers. This behaviour usually relies not only on knowledge-based transitions that follow the perimeter of the decision ladder, but also on rule- and skill-based transitions based on experience with prior solutions (Rasmussen 1986; Vicente 1999).
Macrocognitive activities
We will focus in particular on macrocognitive activities that can be fitted with the decision ladder (as shown in figure 1), such as sensemaking and planning (Klein, Klein, and Klein 2000; Schraagen, Klein, and Hofmann 2008). The macrocognitive activities identified in this research distinguish themselves from other cognitive activities based on the natural hierarchy of information processing proposed by Newell (1990). At the
lower end of this hierarchy, processes on the cognitive band are measured in milliseconds, whereas higher order cognitive processes are measured in seconds or minutes. Based on their higher information processing time, macrocognitive activities are comprised of the smaller individual cognitive elements on the decision ladder. The first macrocognitive activity, situation assessment or analysis, is represented in the left leg of the decision ladder (Lintern 2010). The three cognitive states in this leg coincide with the three levels of Endsley’s (1995) model of situation awareness: perception of elements (alerted to dimensions of the situation), comprehension of these elements (aware of current system state), and anticipating future events (aware of future system state). Klein, Moon, and Hoffman (2006) view sensemaking as the process by which situation awareness is achieved. Hence, the cognitive processes and states in this part of the decision ladder will be subsumed under the heading of ‘sensemaking’.
After making sense of the situation, decision makers are faced with response planning (Vicente, Mumaw, and Roth 2004), at least if the situation is unfamiliar and no pre-planned responses are readily available. Response planning starts with value
judgment (Rasmussen, Pejtersen, and Goodstein 1994). In the cognitive processes and cognitive states embedded in this part of the decision ladder, decision makers generate one or multiple options and assess these explicitly based on their goals, after which they will select a suitable option or ‘target state’. Hence, in this part of the decision ladder we do not only include the macrocognitive function of option generation, as activities here are also concerned with evaluating these options. We therefore named this macrocognitive activity on the decision ladder ‘option evaluation’.
Finally, in the right leg of the decision ladder, actions are planned that need to be carried out in order to arrive at the target state. In the bottom right cognitive state of the ladder, decision makers identify whether the chosen plan achieves the goal. Unlike
option evaluation, this involves no analytic comparison of options (Lintern 2010). Hence, the cognitive states and processes in the right leg of the decision ladder are part of the activity of ‘action planning’.
[Figure 1 near here]
Structure of macrocognitive activities for unfamiliar problems
Many studies have shown how different aspects of macrocognitive activities may differ between experts and novices during problem solving (see for example Glaser and Chi 1988; Kobus, Procter, and Holste 2001; Randel, Pugh, and Reed 1996; Simon and Simon 1978). However, as argued earlier, these differences cannot be directly translated to experienced professional’s problem solving strategies on novel tasks.
Experienced decision makers usually use a mix of decision making strategies (Salas, Rosen, and DiazGranados 2010), and switch more easily than novices between rule- and knowledge-based behaviour (Farrington-Darby and Wilson 2006). In general, rule-based processes are more reliable than knowledge-based processes (Hobbs and Williamson 2002). But reapplying past strategies to new situations based on stored rules can be inappropriate (Dane 2010), and can lead to severe and systematic errors (Tversky and Kahneman 1974). Especially in complex environments, decision makers run the risk of using the wrong strategies (Clewley and Stupple 2015; Kozlowski 1998).
In the current research, both knowledge and rule-based transitions are
interpreted in terms of Lintern’s integrated decision ladder model (2010). A transition between cognitive states is considered knowledge-based if it follows the perimeter of the decision ladder. Rule-based transitions in the centre of the decision ladder are redefined as recognition-primed activities that were originally part of the RPD model. At any time during sensemaking activities, decision makers can recognize actions and
commence with action planning. Based on mental simulation, decision makers can loop back to sensemaking activities and evaluate information more thoroughly.
In unfamiliar situations decision makers are more likely to rely on knowledge-based transitions, shown at the perimeter of the decision ladder, due to a lack of rules (Rasmussen 1986). Moreover, it is generally assumed that decision makers can only adequately deal with non-routine problems in an ill-structured environment by explicitly evaluating options (see for example Klein 1989; Lipshitz 1997). However, this does require that decision makers acknowledge the limits of their expertise and know when to switch between knowledge and rule-based strategies (Kahneman and Klein, 2009). Hence, the current study will examine the relationship between the use of different strategies for each of the macrocognitive activities and quality of the decisions taken.
Method
Participants
This research focused on decision processes of Airline Operations Control decision makers at a European airline. At the airline studied, each shift two operations controllers (one responsible for the European schedule, the other one for the intercontinental
schedule) work under the supervision of a duty manager. Ten out of twenty operations controllers and five out of seven duty managers participated voluntarily in the research: resulting in 15 subjects for the experiment.
During routine operations, the duty managers and operations controllers have distinct responsibilities. A routine disruption would include for example a delay of an inbound intercontinental flight due to a technical issue. The operations controller decides whether to delay several outbound flights to ensure connections for transfer passengers, or rebook passengers onto later flights. The duty manager’s main task is
monitoring the overall decision making process, and ensuring smooth cooperation between the different Airline Operations Control departments, such as maintenance and crew planning. For scenarios with a potentially big impact on the flight schedule, responsibilities of the duty managers and the operations controllers intersect due to either the timing and/or the size of the event. Non-routine issues are usually escalated to the duty manager, because these often involve problems that transcend individual flights or aircraft and hence have a bigger impact on the flight schedule. An example would be the issuing of an operator bulletin by an aircraft manufacturer, signalling a newfound technical issue with a specific aircraft type that needs to be addressed within a 24 hour period. In such a case, the duty manager has final responsibility in the decisions that need to be made, as he is also the one who needs to inform the Airline Operations Control management. He would however usually take those decisions in cooperation with the operations controllers.
Proficiency levels of participants were estimated by using two of the methods listed by Hoffman and Lintern (2006). All but one of the subjects have a minimum of 3000 hours of experience in their current function, and nine have more than 10.000 hours of experience. Only one participant had 9.000 instead of over 10.000 hours of experience within the Airline Operations Control domain. Interviews with Airline Operations Control management revealed that, based on performance reviews and hiring standards, all of the participants have at least reached the level of journeyman (Hoffman et al. 1995).
Procedure
Preparation
Woods (2003) distinguishes ‘natural history techniques’ (waiting to see what happens) and ‘staged world studies’ (staging situations of interest through simulation). As major disruptions do not present themselves frequently enough for our purposes, and to ensure comparability between cases, a simulated task was staged. Since the focus of this
research is on identifying decision strategies, think aloud was selected as the method of choice. Using this method it is possible to gain insight into various cognitive
components of thought processes, such as underlying goals, strategies and decisions (Ericsson and Simon 1993).
The experiment was conducted by two experimenters, one of which has ample experience as a quality process engineer at the studied airline’s operations control centre. In preparation for the experiment, the other experimenter (insert initials of first author) familiarized herself for a year with the organisation by observing and
interviewing the operations controllers and duty managers on the job, both during regular operational activities and during unexpected events that caused disruptions to the operations schedule. A fictional case was constructed for the simulation session, to prevent the risk of any of the participants recognizing the scenario and relying on memory to formulate decision outcomes instead of actually going through the decision making process. To ensure a high level of unfamiliarity with the case, interviews about actual severe disruptive events they had witnessed in the past were held with several different operations controllers and duty managers. These interviews revealed that although isolated elements of such events can occur more frequently and are familiar in isolation, it is the combination of elements that creates surprising dynamics which contribute to unfamiliarity of events. For example, heavy snow by itself does not occur
frequently, but is relatively familiar in isolation because its impact on runway capacity and hence on the flight schedule is usually clear. However, if this occurs in combination with other elements such as low hotel capacity, limited transportation means to and from the airport, and low availability of spare aircraft, what seemed to be a relatively controlled event can take a turn for the worse. Based on the interviews, a case was constructed in consultation with a former duty manager, thereby not only ensuring a high level of unfamiliarity but also keeping the case scenario sufficiently realistic. Finally, the case was compared to historical data, ensuring no comparable event had indeed happened in the past ten years.
A scenario was constructed based on heavy precipitation conditions in winter, which would severely impact the departure and arrival capacity at the airport.
Participants were first required to determine the impact of the weather conditions on the flight schedule, after which they had to specify what actions they thought would be appropriate. To be able to take such decisions, participants were allowed to look up additional real-time information (for example, about crewing or maintenance) as they would be able to do in reality. Since live data was used, all experimental sessions were scheduled on operationally similar days to ensure that simulation sessions were
maximally comparable. By using this approach, a balance was ensured between keeping the task both maximally simple and similar to a real life situation (Ericsson and Smith 1991).
Participants were requested to think aloud while contemplating the scenario and the implications for the airline’s flight schedule. Prior to the experiments, a pilot experiment was conducted with a duty manager still in training. By using him as a participant in a pilot study, we ensured that the case qualified for our research purposes.
Based on the results of the pilot, no changes to the case or the simulation and interview procedure were deemed necessary.
Experimental session
The experiment took place in a quiet room away from the operations control room. Participants were handed out paper and pencil, and were seated in front of a computer that they were allowed to use. They were given a short instruction, explaining the goal of the experiment to them. After this instruction, three short assignments that were not related to the simulation scenario were given to the participant in order to practice thinking aloud, as recommended by Ericsson and Simon (1993) when using the think aloud method. Next, participants were asked to start the computer and all of the live programs that they would normally also use. They were requested to give a short description of these programs. This both increased the level of fidelity of the simulation scenario and allowed the experimenters to be better able to understand what the
participant was doing when using the computer during the experiment. Finally, the participants were given a verbal description of the case and were handed out an assignment sheet with the same written description.
After the experimental sessions, a short interview was held to ensure participants did not have any objections or felt hampered by the instruction to think aloud. No participants indicated any such issues.
Quality assessment
Assessing solution accuracy can be problematic in an ill-structured environment, as there is not one correct solution. Moreover, due to lack of feedback it is difficult to determine for any solution if it is satisfactory. However, solutions can be judged based on other criteria than sheer effectiveness. For example, in Airline Operations Control it
is very important that solutions are robust. This means that a solution does not have to be optimal, but it will have to allow for the option of taking quick complementary measures if necessary in the spur of the moment. By using criteria like robustness, it is possible to at least determine whether a given solution is better or worse than any other.
An expert panel, consisting of two highly proficient Airline Operations Control decision makers at the airline based on job performance criteria, jointly judged solutions by rank ordering them. A bullet point summary containing the goals and actions of participants was extracted from each of the experimental transcripts and was handed out to them. One of these experts was also a research participant, but we did not consider this to be problematic, due to the two year time lag between the experimental sessions and the quality judgment. Moreover, in preparation for the quality assessment session the expert confirmed that he did not recognize the scenario. The presence of a second expert also enhanced the objectivity of their joint judgment.
Coding Procedure
All think aloud recordings were transcribed. Next, a coding scheme needed to be created that would allow for a detailed analysis of results. This scheme needed to be able to identify the smaller information processing elements that are incorporated in the various cognitive states of the decision ladder. We chose to use a coding scheme
modelled after Goel and Pirolli (1992) that identified these elements at a very fine-grained level. This allowed us to sift information carefully and eliminate all irrelevant information that participants came up with during the experiment. Specific
combinations of elements were then tied to individual cognitive states.
The coding scheme was validated by calculating inter-rater agreement levels. Three protocols were randomly selected and coded independently by two raters. The
correlation between the two raters was sufficiently high (Cohen’s kappa of resp. 0.75, 0.72 and 0.78) to allow one rater to finish coding the remaining protocols.
Measurement
For each participant, statements concerning the development of a plan were mapped onto the cognitive states of the decision ladder. Consequently, each statement could be linked to one of three macrocognitive activities: sensemaking, option evaluation or action planning. The structure of participants’ cognitive decision making process was studied based on those activities in two ways. First, all macrocognitive activities were assessed in terms of the number of statements relating to each of the activities relative to the number of statements in the other categories. Second, rule-based versus knowledge-based behaviour was identified by looking at the number of switches between different cognitive states of macrocognitive activities. A switch between cognitive states was labelled knowledge-based if it followed the perimeter of the decision ladder. All other transitions between cognitive states were labelled rule-based. In addition to analysing the nature of transitions throughout the entire decision process, they were also analysed per macrocognitive activity. For example, knowledge-based action planning consisted of all transitions towards cognitive states within the action planning activity that followed the perimeter of the decision ladder. Hence, switches from ‘aware of desired system state’ to ‘understand what needs to be done’, and switches between ‘understand what needs to be done’ and ‘satisfied plan achieves goal’ were labelled as such, whereas all other switches for action planning were labelled as rule-based.
Decision quality was measured by rank ordering solutions. The solution that was considered to have the highest quality received a score of 15, whereas the lowest quality solution scored 1. All other solutions were scored somewhere in between. Next to a
relative rank ordering of solutions, the experts provided brief comments on each solution, giving us general insight into the absolute quality of solutions as well.
Results
We first studied frequency of statements for macrocognitive activities and knowledge versus rule-based behaviour in the data, after which we tested if there were any correlations between these patterns and decision quality, using Spearman’s Rho. This test was chosen because our data do not follow the normal distribution, are rank
ordered on a continuous ordinal scale, and because the Spearman test is less sensitive to outliers, which could skew results considerably due to our small sample size. Results are summarized in table 1.
[Table 1 near here]
Examples from the protocols that illustrate the use of the different macrocognitive activities by participants are added in tables 2 and 3.
[Tables 2 and 3 near here]
Structure of Macrocognitive Activities
The general structure of the cognitive decision process, in terms of the different macrocognitive activities, shows that Airline Operations Control decision makers’ frequency of sensemaking statements is relatively high (49 percent , see figure 2), whereas the frequency of option evaluation statements is relatively low (17 percent). This is interesting, since this indicates that even though decision makers are dealing with a relatively unfamiliar event, they do not seem to be much engaged with prioritizing goals or establishing an appropriate target state.
[Figure 2 near here]
We tested whether the number of statements dedicated to any one of the macrocognitive activities was related to decision quality. No relationships were found between the quality of solutions and frequency of sensemaking, option evaluation or action planning statements (resp. rs (13) = -0.021, p>.05; rs (13) = -0.468, p>.05; rs (13)
= 0.274, p>.05). Frequency of statements dedicated to any one of the macrocognitive activities on the decision ladder does not reflect how good or bad solutions will eventually be. However, we did find a negative trend in the correlation between
frequency of statements related to option evaluation and decision quality, although this was not significant at the .05 level.
Knowledge and Rule-Based Transitions for Macrocognitive Activities
We also analysed whether switches between the cognitive states within each of the macrocognitive activities are based on stored rules, or reflect knowledge-based decision making. Figure 3 shows the relative amounts of knowledge and rule-based transitions for the decision making process in general, and for each of the macrocognitive activities separately. Rule-based transitions (69 percent) were used much more frequently than knowledge-based transitions (31 percent), and is the result of increased rule-based decision making for sensemaking and action planning. Option evaluation on the other hand, shows equal amounts of rule-based and knowledge-based transitions.
[Figure 3 near here]
The eight transitions most frequently used are summarized in table 4, and are graphically depicted in figure 4. Six of those are rule-based, with three feedforward and three feedback loops. What stands out first, is that the most frequently used transitions are all part of the lower half of the decision ladder. Second, most of the cognitive processes employed by participants are centred around understanding what must be done. This understanding appears to be largely based on the use of recognition primed
rules, that allow participants to deduce actions directly from the dimensions of the situation or from awareness of the participated future system state (resp. switches 4 and 6 in figure 4). Another rule-based transitions that is frequently used, is the recognition of the future system state based on the dimensions of the situation, without having to determine the status of the current system’s state first (switch 3 in figure 4).
[Table 4 and Figure 4 near here]
We analysed if knowledge-based transitions in the general decision making process were related to solution quality. This relationship proved to be statistically significant (rs (13) = -0.518, p<.05). The negative sign of the correlation indicates that
higher frequency of knowledge-based transitions was associated with lower solution quality. To further investigate this unexpected result, we examined the correlation between these transitions and quality of solutions for each of the separate
macrocognitive activities. For option planning, no such correlation was found (rs (13) =
-0.054, p>.05). A significant negative relationship was found between frequency of knowledge-based transitions during sensemaking and solution quality (rs (13) = -0.530,
p<.05), and between frequency of knowledge-based transitions during action planning and solution quality (rs (13) = -0.540, p<.05). This relationship was contrary to what one
would expect. An increase of knowledge-based transitions for both sensemaking and action planning was found to correlate with lower quality of solutions. In other words, quality of solutions increases with an increase in rule-based behaviour. Moreover, knowledge-based sensemaking and knowledge-based action planning also correlated with each other (rs (13) = 0.541, p<.05), as one decreases when the other does. This
suggests that the group of decision makers that has a low score for knowledge-based sensemaking and a high score for decision quality, consists largely of the same
individuals who have a low score for knowledge-based action planning and a high score for decision quality.
Discussion
The main purpose of the present study was to explore the structure of the decision making process of Airline Operations Control decision makers when dealing with non-routine decisions, and to assess the way this structure is related to decision quality. To this end, this study has first mapped the activities of decision makers on the decision ladder. Subsequently it was analysed if the relative frequencies of articulations of several macrocognitive activities on the ladder and their knowledge or rule-based nature was related to quality of decision outcomes. Results of this study show that when
confronted with novel problems, the structure of professionals’ decision making process does not seem to change, as it still resembles the structure of the decision process of routine experts. Remarkably, this did not impede the quality of decision outcomes. There was a positive trend between higher quality of solutions and low frequency of statements dedicated to option evaluation. Moreover, the positive relationship between higher quality of solutions and lower number of rule-based transitions shows that part of this routine expert behaviour is still effective, even under non-routine conditions.
Below, conclusions are first discussed in light of the findings of this research. Second, based on a number of informal observations that we made during the
experimental sessions, we also speculate on some additional explanations that could have led to these results.
The general structure of the decision process of Airline Operations Control decision makers showed that on average, there is a relatively high percentage of
statements for sensemaking compared to action planning and option evaluation. This is consistent with expert research showing that experts spend a great deal of time
analysing a problem qualitatively (Glaser and Chi 1988), particularly in ill-defined problems (Voss and Post 1988; Voss and Wiley 2006). Another type of behaviour that is generally associated with routine expertise was exhibited by Airline Operations Control decision makers for option evaluation. The low frequency of option evaluation statements reflects the general amount of rule-based cognitive processing, which is often shown by routine experts in naturalistic settings as they rely on recognition strategies (Klein 2003).
Another expertise based view that can help to account for the general high amount of rule versus knowledge-based transitions, is that reshaping the entire flight schedule resembles a design process. Identifying solutions early on in the decision process is often shown in design problems, as expert-designers are solution focused (Cross 2004). After initial solution identification, they will mould the solution until it is considered effective (Klein and Brezovic 1986), causing several loops back and forth between sensemaking and action planning. We speculate that time pressure, and in addition the common Airline Operations Control notion that doing nothing is usually the worst decision anyone can take, could have amplified this effect, as decision makers are eager to arrive at solutions early.
What was particularly interesting, is that rule-based behaviour was positively related to decision quality. In other words, quality of decision outcomes actually increased instead of decreased together with Airline Operations Control decision makers’ use of more rule-based decision making strategies. This was also reflected in the positive trend between lower numbers of option evaluation statements and higher decision quality. This result is remarkable, as it is generally assumed that decision makers can only adequately deal with non-routine problems in an ill-structured
environment by explicitly evaluating options (see for example Klein 1989; Lipshitz 1997; Rasmussen 1986).
Two explanations can be provided to account for this result: either the problem that was used in the experiment was not really unfamiliar, or Airline Operations Control professionals’ adaptive expertise enables them to reframe an unfamiliar problem into a familiar problem, allowing for recognition of solutions. This would be consistent with Simon (1973), who stated that experts will decompose ill-structured problems into multiple well-structured sub problems that can be solved by using their routine expertise.
The first explanation, that the problem was not really unfamiliar, seems unlikely. Due to the low frequency of large scale disruptions and shift rotations, Airline
Operations Control decision makers are confronted only one to three times per year with such events. Moreover, not only is it difficult to get adequate feedback on solutions due to the unstructured nature of the environment, at the airline studied feedback is also not formalized in the disruption management process.
It could be argued that in Airline Operations Control, regardless of the nature of the problem, solutions on a high level are to a large extent similar and thus familiar. For example, in case of disruptions, whether small or large, eventually most actions come down to delaying or cancelling flights. However, this can only explain why the decision making process is initially rule-based. Determining the size and timing of the actions is what really matters, as these will largely shape the flight schedule and the goals that are aimed for. For example: cancelling a large number of flights right at the moment of impact of a disruption might save costs as delays are limited, but cancelling the same number of flights over a longer time span might free up resources during the rest of the day and hence provide more flexibility in case of additional unexpected issues.
The second explanation, suggested by Simon (1973), is more plausible. It is possible that Airline Operations Control decision makers who are able to gain sufficiently adequate situation awareness through sensemaking, are still capable of identifying actions based on recognition principles. We speculate that, especially in Airline Operations Control, many actions or options that can be applied under non-routine circumstances probably have been applied before during non-routine operations, albeit on a smaller scale and perhaps not in combination with each other. For example, during a routine issue, such as unexpected prolongation of aircraft maintenance, decision makers will need to cancel a flight if a spare aircraft is unavailable. Focus during this action will be to determine which single flight that operates within the right time frame in the schedule, will minimize cancellation costs for the airline. During large scale disruptions, multiple flights in the schedule are impacted, and cancellation of flights will inevitably be one of the preferred actions. However, the focus will need to shift from just determining cancellation costs of single flights to include other schedule dynamics as well, as other issues such as crew rotations and positioning of aircraft also come into play. These issues are by themselves not unfamiliar, as they can be part of other small scale routine problems, but they are usually dealt with in relative isolation. The non-routine character of large scale problems is mainly caused by the combinations of factors that need to be dealt with all at once.
The notion that Airline Operations Control decision makers can reframe an ill-structured problem into multiple familiar sub problems, was strengthened by some informal observations during the simulation sessions. First, most participants would for example split the problem into issues caused by departure delays versus arrival delays. Departure delays would be further divided into problems caused by field closures at the end of the day, or delay problems caused by crew working hours. All of these sub
problems individually are familiar, and actions that are appropriate can be recognized. These actions were then refined and moulded until considered effective under the current circumstances.
Second, we observed that after initial adoption of a recognized high level action, such as cancelling flights, additional constraints were identified through sensemaking. These were used as cut-off points, and were used to determine the scale and timing of the high level actions in greater detail. Many participants then used mental simulation to ascertain that detailed actions would result in a desired target state, in this case the flight schedule.
Finally, constraints that were identified and elaborated on to limit the range of possible solutions, such as maintenance or crew issues, varied greatly between
individuals and seemed largely to depend on personal experience gained in previous functions. Previous experience ranges from other Airline Operations Control jobs, such as dispatcher or crew planner, to experience outside the Airline Operations Control domain (for example, as a cabin attendant, or maintenance engineer). As no relationship was found between frequency of statements dedicated to either sensemaking or action planning on the one hand, and quality of decisions on the other, we surmise that familiarity with decision making content, rather than structure of macrocognitive activities, is related to quality of decisions when professionals are confronted with non-routine issues (cf. Schraagen 1993).
Implications for Airline Operations Control
Although the current study has shed more light upon the structure of the decision making process, it has also revealed that the nature of this process itself relies largely on rule-based transitions. The relationship between this type of decision making strategies and quality of the outcome also cautions against forcing Airline Operations
Control decision makers to engage in more knowledge-based decision making. But this does not solve Airline Operations Control’s management problem of not having insight into the reasons why certain decisions are made, and hence believing that they are not in control of the decision process in their organisation. Moreover, solutions of different decision makers can differ widely, as the quality assessment reviewers’ comments revealed. As decisions are not evaluated afterwards, feedback on rule-based decision making strategies is not readily available. Consequently, less proficient decision makers have no opportunity to learn from their expert peers. Such feedback could ultimately help to reach more uniform decisions and hence ensure greater control over process quality, without impeding the use of rule-based decision making strategies or decision quality.
Obviously, using rule-based decision making strategies does not equate with uncontrolled decision making. It only implies that it is difficult for an outside observer to clarify which mechanisms are used by the decision maker that lead to good quality solutions. This makes it critical to elicit not just the structure of the decision process, but also the decision content, for example cues and subsequent actions that trigger the use of rule-based strategies for some but not for other decision makers. Only then can we hope to effectively support the decision making process. Eventually, this should reduce the risk of the airline of running into operational chaos in the face of unexpected events, keeping both the airline’s management and its customers satisfied.
Limitations and Future Research
Due to time and resource constraints, our research sample was relatively small. More research with larger numbers of participants would be necessary in order to confirm if the results we found in this study are solid. On the other hand, we have sampled
approximately half of the available operations controllers and duty managers within the
airline in our study. Another question is whether results can be extended to other
domains as well, and what other macrocognitive processes besides the ones studied here are related to decision quality under unfamiliar conditions.
Although afterward all participants agreed that the scenario used in the simulations was realistic enough, some of them mentioned that the main difference between the simulated scenario and real life is the absence of a sense of urgency and responsibility for severe consequences if the wrong decision is taken. This could have influenced the results of our study (Randel, Pugh, and Reed 1996). However, we believe that the selected research method is still to be preferred over examining the naturalistic environment, as this would have impacted the comparability of participants’ results. Moreover, it is difficult to predict when the events studied here arise (Farrington-Darby and Wilson 2006). Due to time constraints, this was considered a serious disadvantage.
Finally, we noticed that there were large differences between individual decision outcomes in terms of measures taken to adapt the flight schedule. Reviewer comments indicated that decisions on the low end of the spectrum were indeed relatively poor, whereas the highest ranked plans were considered to be excellent. This raises two questions. First, why is it that some of the Airline Operations Control professionals were capable of transferring their knowledge and skills to deal effectively with non-routine problems, whereas others were not? As all participants qualified at least as journeymen, years of experience in their current function, or even in the domain of Airline Operations Control, probably cannot or can only partly account for this, especially as participants’ focus on particular sub problems seems instead linked to previous experience. Moreover, although actions for sub problems were generated based on recognition, extrapolating actions from small scale routine disruptions to large scale non-routine disruptions requires certain creative problem solving skills as well.
This stresses the need for more research on the subject of adaptive expertise, and why it is gained by some experts, but not others.
References
Abdelghany, K. F., A. F. Abdelghany, and G. Ekollu. 2008. “An Integrated Decision Support Tool for Airlines Schedule Recovery during Irregular Operations.”
European Journal of Operational Research 185 (2): 825-848.
Ashoori, M., C. M. Burns, B. d'Entremont, and K. Momtahan. 2014. “Using Team Cognitive Work Analysis to Reveal Healthcare Team Interactions in a Birthing Unit.” Ergonomics 57 (7): 973-986.
Ball, M., C. Barnhart, G. Nemhauser, and A. Odoni. 2007. “Air Transportation: Irregular Operations and Control.” In Handbook in OR & MS, edited by C. Barnhart and G. Laporte, 1-67. Amsterdam, Netherlands: Elsevier.
Bratu, S., and C. Barnhart. 2006. “Flight Operations Recovery: New Approaches Considering Passenger Recovery.” Journal of Scheduling 9 (3): 279-298. Bruce, P. J. 2011. “Decision-Making in Airline Operations: The Importance of
Identifying Decision Considerations.” International Journal of Aviation
Management 1 (1): 89-104.
Clausen, J., A. Larsen, J. Larsen, and N. J. Rezanova. 2010. “Disruption Management in the Airline Industry – Concepts, Models and Methods.” Computers &
Operations Research 37 (5): 809-821.
Clewley, R., and E. J. Stupple. 2015. “The Vulnerability of Rules in Complex Work Environments: Dynamism and Uncertainty Pose Problems for
Cognition.” Ergonomics (ahead-of-print): 1-7. doi: 10.1080/00140139.2014.997804
Cross, N. 2004. “Expertise in Design: An Overview.” Design Studies 25 (5): 427-441. Dane, E. 2010. “Reconsidering the Trade-Off Between Expertise and Flexibility: A
Cognitive Entrenchment Perspective.” Academy of Management Review 35 (4): 579-603.
Dhukaram, A. V., and C. Baber. 2015. “Modelling Elderly Cardiac Patients Decision Making Using Cognitive Work Analysis: Identifying Requirements for Patient Decision Aids.” International Journal of Medical Informatics 84 (6): 430-443.
Endsley, M. R. 1995. “Toward a Theory of Situation Awareness in Dynamic Systems.”
Human Factors 37 (1): 32-64.
Ericsson, K. A. 2006. “The Influence of Experience and Deliberate Practice on the Development of Superior Expert Performance.” In The Cambridge Handbook of
Expertise and Expert Performance, edited by K. A. Ericcson, N. Charness, P. J.
Feltovich and R. R. Hoffman, 683-703. New York, NY: Cambridge University Press.
Ericsson, K. A., and N. Charness. 1994. “Expert Performance: Its Structure and Acquisition.” American Psychologist 49 (8): 725-747.
Ericsson, K. A., and H. A. Simon. 1993. Protocol analysis: Verbal Reports as Data. Cambridge, MA: Bradford books/MIT press.
Ericsson, K. A., and J. Smith. 1991. “Prospects and Limits in the Empirical Study of Expertise: An Introduction.” In Toward a General Theory of Expertise:
Prospects and Limits, edited by K. A. Ericsson and J. Smith, 1-38. Cambridge,
UK: Cambridge University Press.
Farrington-Darby, T., and J. R. Wilson. 2006. “The Nature of Expertise: A Review.”
Applied Ergonomics 37 (1): 17-32.
Feigh, K. M., and A. R. Prichett. 2010. “Modeling Work for Cognitive Work Support System Design in Operational Control Centers.” Journal of Cognitive
Engineering and Decision Making 4 (1): 1-26.
Feltovich, P. J., M. J. Prietula, and K. A. Ericsson. 2006. “Studies of Expertise from Psychological Perspectives.” In The Cambridge Handbook of Expertise and
Expert Performance, edited by K. A. Ericcson, N. Charness, P. J. Feltovich and
R. R. Hoffman, 683-703. New York, NY: Cambridge University Press.
Glaser, R., and M. Chi. 1988. “Overview.” In The Nature of Expertise, edited by M. T. H. Chi, R. Glaser and M. J. Farr, xv-xxxvi. Hillsdale, NJ: Lawrence Erlbaum Associates.
Goel, V., and P. Pirolli. 1992. “The Structure of Design Problem Spaces.” Cognitive
Science 16: 395-429.
Hobbs, A., and A. Williamson. 2002. “Skills, Rules and Knowledge in Aircraft Maintenance: Errors in Context.” Ergonomics 45 (4): 290-308.
Hoc, J. M., and R. E. N. E. Amalberti. 1995. “Diagnosis: Some Theoretical Questions Raised by Applied Research.” Current Psychology of Cognition 14 (1): 73-101.
Hoffman, R. R., and G. Lintern. 2006. “Eliciting and Representing the Knowledge of Experts.” In The Cambridge Handbook of Expertise and Expert Performance, edited by K. A. Ericcson, N. Charness, P. J. Feltovich and R. R. Hoffman, 683-703. New York, NY: Cambridge University Press.
Hoffman, R. R., N. R. Shadbolt, A. M. Burton, and G. Klein. 1995. “Eliciting Knowledge from Experts: A Methodological Analysis.” Organizational
Behavior and Human Decision Processes 62 (2): 129-158.
Jenkins, D. P. 2012. “Using Cognitive Work Analysis to Describe the Role of UAVs in Military Operations.” Theoretical Issues in Ergonomics Science, 13 (3): 335-357.
Kahneman, D., and G. Klein. 2009. “Conditions for Intuitive Expertise.” American
Psychologist, 64 (6): 515-526.
Klein, G. 1989. “Recognition-Primed Decisions.” In Advances in Man-Machine System
Research, edited by W.B. Rouse, 47–92. Greenwich, CT: JAI Press Inc.
Klein, G. 1993. “A Recognition-Primed Decision Model of Rapid Decision Making.” In
Decision Making in Action: Models and Methods, edited by G. A. Klein, J.
Orasanu, R. Calderwood, and C. Zsambok, 138-147. Norwood, NJ: Ablex Publishing Corporation.
Klein, G., and C. P. Brezovic. 1986. “Design Engineers and the Design Process: Decision Strategies and Human Factors Literature.” Proceedings of the Human
Factors and Ergonomics Society Annual Meeting 30 (8): 771-775.
Klein, D. E., H. A. Klein, and G. Klein. 2000. “Macrocognition: Linking Cognitive Psychology and Cognitive Ergonomics.” Proceedings of the 5th International Conference on Human Interaction with Complex Systems 173-177.
Klein, G., B. Moon, and R. R. Hoffman. 2006. “Making Sense of Sensemaking 1: Alternative Perspectives.” IEEE Intelligent Systems 21 (4): 70-73.
Klein, G., K. G. Ross, B. M. Moon, D. E. Klein, R. R. Hoffman, and E. Hollnagel. 2003. “Macrocognition.” IEEE Intelligent Systems 18 (3): 81-85.
Kobus, D. A., S. Proctor, and S. Holste. 2001. “Effects of Experience and Uncertainty during Dynamic Decision Making.” International Journal of Industrial
Ergonomics 28 (5): 275-290.
Kohl, N., A. Larsen, J. Larsen, A. Ross, and S. Tiourine. 2007. “Airline Disruption Management – Perspectives, Experiences and Outlook.” Journal of Air
Transport Management 13 (3): 149-162.
Kozlowski, S. W. 1998. “Training and Developing Adaptive Teams: Theory, Principles, and Research.” In Making Decisions under Stress: Implications for Individual
and Team Training, edited by J.A. Cannon-Bowers, and E. Salas, 115–153.
Washington, DC: American Psychological Association.
Lintern, G. 2009. The Foundations and Pragmatics of Cognitive Work Analysis: A
Systematic Approach to Design of Large-Scale Information Systems.
http://www.cognitivesystemsdesign.net/Downloads/Foundations%20&%20Prag matics%20of%20CWA%20(Lintern2009).pdf
Lintern, G. 2010. “A Comparison of the Decision Ladder and the Recognition-Primed Decision Model.” Journal of Cognitive Engineering and Decision Makin 4 (4): 304-327.
Lipshitz, R. 1993. “Converging Themes in the Study of Decision Making in Realistic Settings.” In Decision making in action: Models and methods, edited by G. A. Klein, J. Orasanu, R. Calderwood, and C. E. Zsambok, 103–137. Norwood, NJ: Ablex.
Lipshitz, R. 1997. “Naturalistic Decision Making Perspectives on Decision Errors.” In
Naturalistic Decision Making, edited by C. E. Zsambok, and G. Klein, 151-162.
Mahwah, NJ: Erlbaum.
McIlroy, R. C., and N. A. Stanton. 2015. “A Decision Ladder Analysis of Eco-Driving: The First Step Towards Fuel-Efficient Driving Behaviour.” Ergonomics (ahead-of-print): 1-17. doi: 10.1080/00140139.2014.997807
Naikar, N. 2010. A Comparison of the Decision Ladder Template and the
Recognition-Primed Decision Model: DSTO Tech. Rep. DSTO-TR-2397. Fishermans Bend,
Vic: Air Operations Division.
Naikar, N., A. Moylan, and B. Pearce. 2006. “Analyzing Activity in Complex Systems with Cognitive Work Analysis: Concepts, Guidelines and Case Study for Control Task Analysis.” Theoretical Issues in Ergonomics Science 7 (4): 371-394.
Newell, A. 1990. Unified Theories of Cognition. Cambridge, MA: Cambridge University
Press.
Randel, J. M., H. L. Pugh, and S. K. Reed. 1996. “Differences in Expert and Novice Situation Awareness in Naturalistic Decision Making.” International Journal of
Human-Computer Studies 45 (5): 579-597.
Rasmussen, J. 1976. “Outlines of a Hybrid Model of the Process Plant Operator.” In
Monitoring Behavior and Supervisory Control, edited by T. B. Sheridan, and G.
Johannsen, 371-383. New York, NY: Plenum Press.
Rasmussen, J. 1986. Information Processing and Human-Machine Interaction: An
Approach to Cognitive Engineering. New York: North-Holland.
Rasmussen, J., A. M. Pejtersen, and L. P. Goodstein. 1994. Cognitive Systems
Engineering. New York, NY: Wiley.
Ross, K. G., J. L. Shafer, and G. Klein. 2006. “Professional Judgments and “Naturalistic Decision Making”.” In The Cambridge Handbook of Expertise and Expert
Performance, edited by K. A. Ericcson, N. Charness, P. J. Feltovich and R. R.
Hoffman, 683-703. New York, NY: Cambridge University Press.
Salas, E., M. A. Rosen, and D. DiazGranados. 2010. “Expertise-Based Intuition and Decision Making in Organizations” Journal of Management 36 (4): 941-973. Schraagen, J. M. 1993. “How Experts Solve a Novel Problem in Experimental Design.”
Cognitive Science 17 (2): 285-309.
Schraagen, J. M., G. Klein, and R. R. Hoffman. 2008. “The Macrocognition Framework of Naturalistic Decision Making.” In Naturalistic Decision Making and
Macrocognition, edited by J. M. Schraagen, L. G. Militello, T. Ormerod, and R.
Lipshitz, 3-25. Aldershot, UK: Ashgate Publishing Limited.
Shanteau, J., D. J. Weiss, R. P. Thomas, and J. C. Pounds. 2002. “Performance-Based Assessment of Expertise: How to Decide if Someone Is an Expert or Not.”
European Journal of Operational Research, 136 (2): 253-263.
Simon, H. A. 1973. “The Structure of Ill Structured Problems.” Artificial Intelligence 4 (3-4): 181-201.
Simon, D. P., and H. A. Simon. 1978. “Individual Differences in Solving Physics Problems.” In Children's thinking: What develops?, edited by R. Siegler, 325-348. Hillsdale, NJ: Erlbaum.
Stanton, N. A., and K. Bessell. 2014. “How a Submarine Returns to Periscope Depth: Analysing Complex Socio-technical Systems Using Cognitive Work Analysis.”
Applied Ergonomics 45 (1): 110-125.
Thengvall, B., J. Bard, and G. Yu. 2000. “Balancing User Preferences for Aircraft Schedule Recovery during Irregular Operations.” IEE Transactions 32 (3): 181-193.
Tversky, A., and D. Kahneman. 1974. “Judgment under Uncertainty: Heuristics and Biases.” Science 185 (4157): 1024-1031.
Vicente, K. J. 1999. Cognitive Work Analysis: Toward Safe, Productive and Healthy
Computer Based Work. Mahwah, NJ: Lawrence Erlbaum Associates.
Vicente, K. J., R. J. Mumaw, and E. M. Roth. 2004. “Operator Monitoring in a Complex Dynamic Work Environment: A Qualitative Cognitive Model Based on Field Observations.” Theoretical Issues in Ergonomics Science, 5 (5): 359-384.
Voss, J. F., and T. A. Post. 1988. “On the Solving of Ill-Structured Problems.” In The
Nature of Expertise, edited by M.T.H. Chi, R. Glaser, and M.J. Farr, 261-285.
Hillsdale, NJ: Lawrence Erlbaum Associates.
Voss, J. F., and J. Wiley. 2006. “Expertise in History.” In The Cambridge Handbook of
Expertise and Expert Performance, edited by K. A. Ericcson, N. Charness, P. J.
Feltovich and R. R. Hoffman, 683-703. New York, NY: Cambridge University Press.
Weiss, D. J., and J. Shanteau. 2012. “Decloaking the Privileged Expert.” Journal of
Management & Organization, 18 (3): 300-310.
Woods, D .D. 2003. “Discovering how Distributed Cognitive Systems Work.” In
Handbook of Cognitive Task Design, edited by E. Hollnagel, 37-53. Mahwah,
NJ: Lawrence Erlbaum Associates.
Wu, C. L. 2005. “Inherent Delays and Operational Reliability of Airline Schedules.”
Journal of Air Transport Management 11 (4): 273-282.
Yu, G., M. Argüello, G. Song, S. McCowan, and A. White. 2003. “A New Era for Recovery at Continental Airlines.” Interfaces 33 (1): 5-22.
Zsambok, C. E. 1997. “Naturalistic Decision Making: Where Are We Now?” In
Naturalistic Decision Making, edited by C. E. Zsambok, and G. Klein, 3-16.
Mahwah, NJ: Erlbaum.
Table 1. Summary of results.
Correlation Result Accepted/Rejected
Frequency of sensemaking statements is related to level of quality of solutions
rs (13) = 0.021, p>.05 Rejected
Frequency of option evaluation statements is related to level of quality of solutions
rs (13) = 0.468, p>.05 Rejected
Frequency of action planning statements is related to level of quality of solutions
rs (13) = -0.274, p>.05 Rejected
Frequency of knowledge-based transitions is related to quality of solutions
rs (13) = -0.518, p<.05 Accepted
Frequency of knowledge-based transitions for sensemaking is related to quality of solutions
rs (13) = -0.530, p<.05 Accepted
Frequency of knowledge-based transitions for option evaluation is related to quality of solutions
rs (13) = -0,054, p>.05 Rejected
Frequency of knowledge-based transitions for action planning is related to quality of
solutions
rs (13) = -0.540, p<.05 Accepted
Frequency of knowledge-based transitions for sensemaking is related to frequency of
knowledge-based transitions for action planning
rs (13) = 0.541, p<.05 Accepted
Table 2. Examples of participants' statements for macrocognitive activities. Macrocognitive
activity
Citation
Sensemaking Participant 14: ‘Low ceiling [of clouds], 600 foot. And later on
conditions will get worse, because we will get freezing fog with low visibility. So capacity will be restricted from 1600 [Zulu] onwards. And this restriction will not so much be determined by runway use, but by our own de-icing capacity.’
Participant 35: ‘You look at a fairly uncertain probability. A prob40
tempo, which means there is a chance of 40 percent of something happening for a short period of time. So you have to take that into account. But if it happens you got to have a plan. Otherwise the operation will come to a halt.’
Action Planning
Participant 25: ‘… we have a cancellation tool that indicates which
[flights] are possible candidates [for cancellation], but there is a crew schedule attached [to the flights] as well which is not
connected to the tool. So we may want something, but we will have to check with cockpit and cabin [crew scheduling] if the proposal we have based on the tool, a priority list, whether that is feasible for them.’
Participant 33: ‘I also would cancel all short up and down flights that
have a crew that goes up- and down together with the aircraft. So the Paris and Manchester flights, which operate multiple times per day.
And the ones that can be reached by means of ground transportation.’
Option Evaluation
Participant 15: ‘We cannot accommodate all of our inbound flights.
So we will have to choose whether we want to accommodate them somewhere else, so whether we leave them at outstations or let them divert to a different airport.’
Table 3. Examples of participants' knowledge-based macrocognitive activities. Knoweledge-based macrocognitive activities Citation Knowledge-based Sensemaking
Participant 12: ‘17 flights per hour inbound and outbound
airport capacity from 1600 Zulu. [] Our last flights are scheduled to leave at 2300. [] I will process that in our Estimation tool. The tool shows an operational delay of 2,5 hours.’
Knowledge-based Option Evaluation
Participant12: ‘[I would cancel 40 legs at 1800 Zulu, and 5
legs after that.] Maybe we could leave those last 5 legs out, which would give us an average delay at 2200 Zulu of 50 minutes. Yes, [for crew] that should be legally
acceptable. At 2300 Zulu we’re clear.
Knowledge-based Action Planning
Participant 32: ‘… the ICA fleet will be protected, so we
will have to cancel most of our European flights.’
Participant 21: ‘… if we still want to operate flights in the
evening block, we need to cancel flights in the late afternoon block. We will have to check which flights we can still operate crew wise, and check which possibilities we have to dispose of passengers.’
Table 4. Participants’ most frequently used transitions between cognitive states. Transi tion Average use Used by # of participants Type Description 1 5 15 Feedback loop
The desired system state is
reconsidered based on possible actions
2 5 15
Knowledge-based switch
Actions are selected based on the desired system state
3 5 14 Rule-based
switch (based on
recognition)
The future system state is directly recognized based on the dimensions of the situation
4 4 15 Rule-based
switch (based on
recognition)
Actions are directly recognized based on the dimensions of the situation
5 4 15 Feedback
loop (mental simulation)
Dimensions of the situation are re-examined to check if actions are appropriate
6 4 12 Rule-based
switch (based on
recognition)
Actions are directly recognized based on the anticipated future system state
7 3 13 Feedback
loop
Dimensions of the situation are re-examined based on the anticipated future system state
8 3 12 Feedback loop
The anticipated future system state is re-evaluated to check if actions are appropriate
Figure 1. Macrocognitive activities mapped onto the decision ladder (adapted from Lintern 2010). Cognitive states are represented in squares. The arrows represent the various rule-based (in dotted lines) and knowledge-based (solid lines) cognitive processes that connect them.
Figure 2. Relative frequency of statements per macrocognitive activity
Figure 3. Relative knowledge-based and rule-based processing of macrocognitive activities.
Figure 4. Participants’ most frequently used transitions between cognitive states (numbered 1 to 8). Numbers are referenced in table 4.
