Executive Program in Management Studies Track Digital Business
Master Thesis
Bridging the Gap Between the Short- and Long-term Planning in the Scaled Agile Software Development Projects
Student Name: Siarhei Cherkas Student ID: 11999705
Supervisor: Prof. em. dr. ir. Hans J. Oppelland EPEC Approval: EC 20210125110101
Word Count: 14229
Submission Date: 22-03-2021 Version: v1.10 Final
Statement of Originality
This document is written by Siarhei Cherkas who declares to take full responsibility for the contents of this document.
I declare that the text and the work presented in this document is original and that no sources other than those mentioned in the text and its references have been used in creating it.
The Faculty of Economics and Business is responsible solely for the supervision of completion of the work, not for the contents.
Index
Abstract ... 5
1. Introduction ... 6
1.1. Research problem... 6
1.2. Research objectives ... 7
1.3. Research attributes ... 8
1.4. Thesis structure ... 8
2. Literature review ... 9
2.1. ASD overview ... 9
2.1.1. Agile Software Development’s inception ... 9
2.1.2. Scrum ... 9
2.1.4. Scaled ASD emergence... 10
2.1.3. Scaled ASD taxonomy ... 11
2.1.5. Scaled ASD methodologies overview ... 11
2.1.5.1. SAFe ... 12
2.1.5.2. LeSS ... 12
2.1.5.3. Disciplined Agile Delivery ... 13
2.2. Planning in Agile and propositions development ... 14
2.2.1. Goals gap ... 14
2.2.2. Frequent or Continuous Planning ... 15
2.2.3. Short-term decisions focus ... 16
2.2.4. Different approaches for balancing short- and long-term planning ... 16
2.3. Proposition’s summary ... 18
3. Research Design... 19
3.1. Conceptual model ... 19
3.2. Research strategy ... 19
3.3. Method selection ... 20
3.4. Sample design (cases selection) ... 21
3.5. Design of data collection instruments ... 22
3.5.1. Method selection and justification ... 22
3.5.2. Interview procedure development... 22
3.5.3. Semi-structured interview protocol development ... 23
3.6. Data analysis approach ... 24
3.7. Threats to validity and reliability ... 25
3.8. Pre-test ... 26
4. Data collection ... 28
5. Data analysis ... 30
6. Research results (presentation and interpretation) ... 32
6.1. Cross Case analysis ... 32
6.1.1. Short- and long-term gap estimation... 32
6.1.2. Goals alignment ... 34
6.1.3. Mismatch of the short- and long-term plan change frequency ... 37
6.1.4. Short-term decisions focus ... 39
6.1.5. Usage of roadmapping techniques ... 40
6.2. Thematic analysis for critical success factors ... 43
6.2.1. Rigorous requirements breakdown ... 43
6.2.2. Continuous plan monitoring ... 44
6.2.3. Tools usage ... 45
6.2.4. Horizontal alignment and dependencies management ... 45
6.2.5. Team engagement into road-mapping ... 46
6.2.6. Advanced prioritization ... 47
7. Discussion ... 49
7.1. Discussion of propositions ... 49
7.3. Discussion of critical success factors and induced themes ... 51
7.4. Conclusion ... 52
7.5. Managerial implications... 53
7.6. Research limitations ... 54
Bibliography (Literature) ... 56
Appendices ... 63
Appendix A – Interview questions development ... 63
Appendix B – A priory code set for thematic analysis ... 64
Appendix C – Interview meeting invitation template (access request) ... 65
Appendix D – Interview guideline (pre-test version). ... 66
Appendix E – Interview guideline after pre-test ... 68
Appendix F – A priori coding results (screenshot from MaxQDA software) ... 71
Appendix G – Open coding results (screenshots from MaxQDA software) ... 72
List of Figures
Figure 1 Conceptual model of the short- and long-term planning gap ... 19
Figure 2 A refined conceptual model of the short- and long-term planning gap ... 51
Figure F1 A priori coding results ... 71
Figure G1 Open coding results ... 72
List of Tables
Table 1 Proposition’s Summary ...18Table 2 Overview of the cases selected for the research ... 28
Table 3 Overview of the conducted interviews ... 29
Table 4 1st order in-vivo codes and 2nd order themes ... 30
Table 5 Propositions support summary across cases ... 32
Table 6 Short- and long-term planning gap across reviewed cases ... 33
Table 7 Summary of the critical success factors from this study and related work ... 52
Table A1 Interview questions development ... 63
Table B1 A priori code set ... 64
Abstract
Agile Software Development (ASD) is widely used across industries at ‘scale’ and is applied to multiple teams and large projects. This study focuses on the short- and long-term planning gap reported in the literature as one of the challenges in scaling ASD. The study has three research objectives (RO): to develop (RO1) and confirm empirically (RO2) conceptual model of how the short- and long-term planning gap appears within Scaled ASD projects, and to identify critical success factors (CSF) that can help solve or prevent the planning gap (RO3).
This study is qualitative and uses a comparative multiple case-study design. Four research propositions were derived from the literature review. Eleven semi-structured interviews were conducted to gather data from four Dutch companies that execute Scaled ASD projects.
Thematic analysis was used to test the propositions. The findings show that Sprint and Product Backlog goals alignment (P1) and the roadmap techniques usage (P4) decreases the gap between short- and long-term planning. Frequency mismatch of short- and long-term plan change does not lead to the planning gap (P2). In reviewed cases, moments of agility occurred between sprints, and changes occurred first in the long-term plan. Agile team’s short-term decisions focus does not increase the short- and long-term planning gap (P3), which is explained by the moderating effect of Agile control, teams’ alignment with Product Owners and Scrum Master, and long-terms goals communication to the team. Six additional CSFs were identified using open coding and thematic analysis. The CSFs are: rigorous requirements breakdown, advanced prioritization, tools usage, horizontal alignment and dependencies management, continuous plan monitoring, and team engagement into the roadmapping process.
Keywords: Agile, Scaling Agile, Agile Software Development, Planning, Long-term planning, Roadmap, Case Study, Critical Success Factors, SAFe.
1. Introduction
1.1. Research problem
Agile software development (ASD) became prominent in the early 2000s and quickly came to dominate the industry. ASD was initially adopted by small teams, but now its methods are often ‘scaled’ and applied to large projects with multiple teams. More than 70% of the respondents to a 2019 survey conducted by VersionOne reported that they work in organizations with more than 100 employees (CollabNet, 2019). There are many challenges associated with scaling ASD. Uludag et al. have identified 79 challenges, of which 41 are directly associated with large-scale Agile development (Uludag et al., 2018). This study focuses on planning-related challenges associated with Scaled ASD. The lack of rigorous empirical studies relevant to the industry’s understanding of ASD has been noted as ‘grand challenge’ (Hoda et al., 2017, p. 14). Empirical research on the topic of Scaled ASD is even more scarce. Dikert et al. identified that in 2016 only six research papers had addressed the topic ‘despite the relevance of the topic for practitioners’ (Dikert et al., 2016, p. 106). One of the suggested directions involves looking into scaling frameworks. On the other hand, the industry reports a high usage of the Scaled Agile Framework for Enterprise (SAFe) (CollabNet, 2019). Putta, Paasivaara, and Lassenius conducted a multivocal-literature review (MVR) and found only six peer-reviewed primary studies on SAFe and advised that future research should include in-depth primary studies on SAFe (Putta et al., 2018).
In a systematic literature review of studies that investigate the success factors and challenges associated with large-scale agile transformations, Dikert et al. report a phenomenon known as the ‘gap between long and short term planning’ (Dikert et al., 2016, p. 10). Uludag et al. report that ‘balancing short-term and long-term goals’ is a common challenge associated with large-scale agile development projects (Uludag et al., 2018, p. 194). It is not apparent why
the gap between short- and long-term planning appears, as both short- and long-term planning techniques are established in Scaled ASD. Short-term planning is covered very effectively in the most commonly used ASD method: Scrum (CollabNet, 2019). Scrum has two short-term planning activities: daily planning for the next 24 hours, and iteration planning for the next 2- 4 weeks (Schwaber & Sutherland, 2017). Multiple practices exist that focus on long-term planning. These include release planning (Cohn, 2005, p. 146), roadmapping or strategic release planning (Svahnberg et al., 2010), and program increment planning developed according to SAFe (Scaled Agile Framework, n.d.-b). The challenges discovered during literature review indicate that the knowledge gap exists. The knowledge gap can be formulated as the following research problem:
RP1. The software development industry lacks knowledge of how to solve the short- and long-term planning gap associated with Scaled ASD.
1.2. Research objectives
This study has three research objectives:
RO1. To develop a conceptual framework that demonstrates how the short- and long-term planning gap appears in Scaled ASD projects.
RO2. To confirm the reasons why the short- and long-term planning gap appears in Scaled ASD projects.
RO3. To identify critical success factors that can help solve and/or prevent the short- and long-term planning gap associated with Scaled ASD projects.
By developing knowledge of the critical success factors associated with solving and/or preventing the short- and long-term planning gap, this study will contribute to practical
managerial knowledge. This study also makes a theoretical contribution to the field in the form of an empirically validated conceptual model.
1.3. Research attributes
This study uses qualitative research methods to develop an in-depth understanding of the phenomenon in question. This study is a multiple comparative case study. The decision to conduct this type of study aligns with the recommendations of other studies that have debated which methods are most appropriate for conducting primary research on the topic of ASD (Hoda et al., 2017, p. 17). This study uses deductive logic to state and test propositions. The data used in the study was gathered using a semi-structured interview method. The data was collected from employees of four different companies. The data was analysed using thematic analysis.
1.4. Thesis structure
This study is structured in the following way: Section 2 is dedicated to the literature review and the development of the research propositions. Section 3 presents the conceptual model, research design, interview protocol development, and a priori code development carried out to assist with the thematic analysis. Section 4 presents the data collection report. Section 5 presents the data analysis process. Section 6 presents the results. Finally, Section 7 concludes with the discussion and the research limitations.
2. Literature review
2.1. ASD overview
2.1.1. Agile Software Development’s inception
ASD involves using self-organizing teams to develop software quickly, address frequently changing requirements, and meet the needs of real customers. ASD’s core principles, as articulated in the Agile Manifesto, include the following: ‘Individuals and interactions over processes and tools’, ‘Working software over comprehensive documentation,’ ‘Customer collaboration over contract negotiation’, and ‘Responding to change over following a plan’ (Beck et al., 2001). The publication of the ‘Agile Manifesto’ in 2001 can be considered the formal starting point of ASD. However, ASD can be traced back to 1998, when ‘the word “agile” was used in combination with “software process” for the first time’ (Dybå & Dingsøyr, 2008, p. 3). ASD can be clearly distinguished from traditional software development methods by its ability to respond to changing environments. ASD’s ability to change is the primary reason for ASD’s emergence (Conboy, 2009, p. 341).
ASD initially came in many different forms (‘frameworks’ or ‘methodologies’). These forms include Dynamic Systems Development Methods, extreme programming (XP), Crystal, Feature-Driven Development, Lean Development, and Scrum (Dybå & Dingsøyr, 2008).
However, Scrum method is the most used method: fifty-four percent of respondents to the
‘State of Agile’ survey reported that they use Scrum (CollabNet, 2019).
2.1.2. Scrum
Ken Schwaber introduced Scrum in 1997 (Schwaber, 1997). Initially, Scrum was defined as a sequence of ‘pregame’, ‘game’, and ‘postgame’ phases (Schwaber, 1997, p. 12).
Only pregame (planning and system architecture) and postgame consisted of defined processes or steps. The Sprint phase (game) was considered an empirical process (‘black-box’) that required external control but did not require a definition of all the processes within that phase.
The external controls for Scrum were ‘backlog’, ‘release’, ‘packets’, ‘changes’, ‘problems’,
‘risks’, ‘solutions’, and ‘issues’. Both the programming team and management shared the responsibility of using the different controls to manage the project. In the later version of the Scrum methodology, the ‘controls’ are not grouped, and some controls from the earlier version are not mentioned (such as risks). In addition, the backlog is split into ‘Product backlog’ and
‘Sprint backlog’ (Schwaber & Sutherland, 2011). While in the initial Scrum revision, the project team consisted of a ‘management’ team and a ‘development’ team (Schwaber, 1997, p.
16), the management team is now divided into two roles: ‘Product Owner’ and ‘Scrum Master’
(Schwaber & Sutherland, 2011). Regardless of which version of Scrum one chooses to work with, the development team must be small, and it has ultimate control over software development during the Sprint phase. Schwaber notes that the Scrum methodology has distinct advantages over waterfall, spiral, and iterative methodologies. These include Scrum’s ability to change a project at any point in time, its defined control mechanisms, and the freedom it gives developers to ‘devise the most ingenious solutions’ (Schwaber, 1997, p. 17).
2.1.4. Scaled ASD emergence
Following the widespread adoption of ASD for use by small software development teams, larger organizations began wanting to use ASD for large-scale projects. At the beginning of the 2000s, some researchers stated that ASD was not applicable to large-scale projects (Cohen et al., 2004, as cited by Dybå & Dingsøyr, 2008, p. 4). Scaled ASD was not the primary approach to software development in the mid-2000s. However, one case-study concluded that ASD could be feasibly integrated into the Stage-Gate models used for larger
projects (Karlstrom & Runeson, 2005, as cited by Dybå & Dingsøyr, 2008, p. 16). Similarly, Hossain, Babar & Paik studied multi-team projects and demonstrated that the Scrum framework can be applied to different projects with multiple teams. They conclude that Scrum might be used by ‘large number of project personnel or team size’ (Hossain, Ali Babar, & Paik, 2009, p. 178)
2.1.3. Scaled ASD taxonomy
To develop common terminologies associated with ASD, Dingsøyr et al. have proposed a taxonomy of scale that can be applied to discussions of agile software development projects.
This taxonomy is based on the number of teams working on a project. Small-scale projects have one team, large-scale projects have two to nine teams, and very large scale projects have ten or more teams (Dingsøyr et al., 2014). Dikert et al. have offered their own taxonomy and defined large-scale as ‘ 50 or more people or at least six teams’ (Dikert et al., 2016).
2.1.5. Scaled ASD methodologies overview
During the 2010s, practitioners and consultants developed several methodologies for applying ASD at scale, including the Scaled Agile Framework for Enterprise (SAFe) created in 2012, the Large Scale Scrum (LeSS) method developed in 2009, the Disciplined Agile Delivery method, and the Scrum of Scrums method. By 2019, SAFe had become the most popular and was named in the ‘State of Agile’ report as the most used agile scaling method (30% of respondents). Other popular methods mentioned by the report include Scrum of Scrums (16%), Internally created methods (8%), DAD (7%), the Spotify Model (5%), and LeSS (2%) (CollabNet, 2019, p. 12).
2.1.5.1. SAFe
SAFe was announced by its author Dean Leffingwell at the ‘Agile 2012’ conference. It was described as ‘a knowledge base of proven practices for scaling Lean and Agile development’ (Gale Academic OneFile, 2012). SAFe consists of detailed guidelines for building an organization’s structure, which roles to use, coordinating multiple teams, structuring requirements, and determining what engineering practices should be used (Scaled Agile Framework, n.d.-b). SAFe suggests dividing an organization into four levels: team, program, portfolio, and value stream. At the program level, all teams are grouped into what is known as the ‘agile release train’ (ART), and all teams follow the same two-week Sprint schedule (Putta et al., 2018). The ART delivers ‘Program Increments’ according to a set schedule that begins with ‘Program Increment (PI) planning’ and consists of 5 or 6 Sprints (10 or 12 weeks). PI planning is the core of SAFe (Scaled Agile Framework, n.d.-b). PI planning should take two days. During PI planning, teams plan work for 6 Sprints ahead.
2.1.5.2. LeSS
There are two versions of the LeSS framework: ‘normal’ and ‘Less Huge’ (Paasivaara
& Lassenius, 2016). The normal version of LeSS incorporates a minimal adoption of elements from the Scrum framework that are suited to large organizations: a single product backlog for all teams, the same definition of ‘done’, the same Sprints schedule, and the presence of a single product owner. ‘Less Huge’ divides the product backlog into several ‘areas’, and each area has its own ‘Area Product Owner’. Each area has its own sprint planning meetings, reviews, etc.
One of the few empirical studies of LeSS is a case study of a large globally distributed project conducted by Nokia. The study notes scaling problems associated with LeSS, and concludes that LeSS is applicable for software projects that are easily split into independent areas (Paasivaara & Lassenius, 2016, p. 83).
2.1.5.3. Disciplined Agile Delivery
Disciplined Agile Delivery (DAD) is a ‘hybrid process’ that enriches Scrum with techniques sourced from other methods (Alqudah & Razali, 2016. p. 830). According to Beecham et al., ‘DAD refers to its framework as a toolkit allowing the creation of an organization-specific scaling agile approach regardless of the organization’s size’ (Beecham et al., 2021, p. 4). So that it can be tailored to various organizational specificities, DAD has four different ‘lifecycles’: Agile/Basic, Advanced/Lean, Continuous Delivery Lifecycle, and Exploratory Lifecycle (Alqudah & Razali, 2016). The Basic lifecycle has three phases. The inception phase focuses on the development of the project vision and the initial modelling, the construction phase includes various practices adopted from SCRUM, and the transition phase includes deployment, testing, and feedback. The Advanced/Lean lifecycle has an inception phase with goals similar to those of the Basic Lifecycle, and its construction phase follows Kanban practices. In the Continuous Delivery lifecycle, the transition phase is very brief and can only be done daily or with other regularity. The Exploratory lifecycle is suited to research situations and has six activities: envisioning, prototyping, deploying, observing, measuring, and eliciting feedback (Alqudah & Razali, 2016). DAD is an Agile software development process that should value change and personal interaction as important facets of the process of developing extensive documents (Beecham et al., 2021, p. 5).
Though DAD has many roles that are similar to those of Scrum, it also incorporates several roles which distinguish it from Scrum. For example, its Product Owner, team member, and Team Leader roles are similar to various roles within Scrum. However, the Architecture Owner, Specialist, Independent Tester, Domain Expert, Technical Expert, and Integrator roles, which can be arranged temporarily, set DAD apart from Scrum (Alqudah & Razali, 2016, p.
829).
2.2. Planning in Agile and propositions development
2.2.1. Goals gap
Let us consider the most used ASD method, Scrum, which, as shown previously, is at the core of SAFe. The two most popular activities in Scrum are ‘daily standups’ and
‘Sprint/iteration planning’ (CollabNet, 2019). These practices are alternatively known as
‘Daily Scrum’ and ‘Sprint planning’ (Schwaber & Sutherland, 2017). Sprints are the fixed interval of time from 1 to 4 weeks and Sprints are characterised as ‘heart of Scrum’ (Schwaber
& Sutherland, 2017, p. 9). Sprint planning should involve the following steps: a specific Sprint Goal should be set by the Product Owner; the teams should use the Product Backlog as an input; the teams then collaborate to create a plan for the Sprint and decide how much of the Product Backlog can be implemented during the Sprint; and, finally, each team decides how the work will be done and presents their plans to the product owners. A Sprint is compared to a project in the Scrum Guide (Schwaber & Sutherland, 2017, p. 9). In the Scrum Guide, the daily standup is alternatively called the ‘Daily Scrum’, which is a meeting in which team plans work for the 24 hours (Schwaber & Sutherland, 2017, p. 12). The Daily Scrum is tied to Sprint planning in the sense that the team must determine how the daily plan will help achieve the Sprint Goal. As the Sprint duration should be no longer than one month, Sprint planning cannot be classified as ‘long-term planning’. Rather, Sprint planning is, like the Daily Scrum, a form of short-term planning.
Product Backlog is a form of long-term planning that can be derived from Scrum. The Product Backlog is the list of all requirements, that are needed to be built (Schwaber &
Sutherland, 2017, p. 15). Uludag et al. have identified and organized challenges associated with Large-Scale Agile development. These challenges include ‘balancing short-term and long-term goals’ (a challenge associated with the ‘requirement engineering practice’) and ‘defining clear
and visible priorities’ (Uludag et al., 2018). When the Sprint goal is aligned with the Product Backlog goal, the short- and long-term planning gap should not appear. The first proposition can be stated:
P1: Alignment of Sprint and Product Backlog goals decreases the short- and long-term planning gap.
2.2.2. Frequent or Continuous Planning
Agile methods, in contrast to traditional methods, are characterized by frequent or even continuous planning; traditional methods use up-front project planning (Conboy et al., 2011).
One of the challenges involved in the adoption of SAFe is the struggle to shift from ‘waterfall culture’ (Ahimbisibwe et al., 2015). ‘Waterfall culture’ in this context is a culture of plan- driven software development. Plan-driven software development is characterized by fixed phases, low release frequency, and extensive up-front design (Van Waardenburg & Van Vliet, 2013). The fixed, low-frequency delivery cycles associated with plan-driven software development prevent the long-term plan from changing and adapting flexibly to the current environment. Dikert et al. have noted ‘care had to be taken to avoid long term planning becoming a prescriptive practice’ (Dikert et al., 2016). There is no clarity in the reviewed literature, how teams change both the long-term and short-term plans, when they are reacting to change. One can imagine that if a team reacting to changes in the external or internal environment changes its short-term plan (e.g., Daily or Sprint level plans) without amending the long-term plan, a planning gap will appear. The opposite seems logical: if a team changes the long-term plan and does not change the short-term plan – the gap between the short- and long-term planning appears. This logic leads to this study’s second proposition:
P2: When short- and long-term plans are amended at the same time, short- and long- term planning gap decreases.
2.2.3. Short-term decisions focus
Let us review the decision-making perspective of ASD. In the 2012 study Drury, Conboy, and Power conclude that decision focus of Agile teams is on short-term, and not on long-term decisions (Drury et al., 2012, p. 1249). In a follow-up study, Drury, Conboy, Grogan and Action draw similar conclusions:
‘Regarding the value of adapting to change, the team made complex decisions quickly and only focused on the short term because their environment is so fast-paced and structured with a series of short, 2-week iterations … they focused on tactical decisions and ignored strategic decisions’ (Drury-Grogan et al., 2017, p. 259).
Focusing on tactical decisions privileges individual goals over team goals (Moe et al., 2009, p. 22). The autonomy of the individuals in a team can combine with pressure to complete iteration goals and thereby create a focus on individual goals rather than team goals. In this study, this phenomenon will be referred to as ‘short-term decisions focus’ (IV). This study’s third proposition develops out of an awareness of the possible negative consequences of short- term decision focus:
P3. Agile team's short-term decisions focus increases the short- and long-term planning gap.
2.2.4. Different approaches for balancing short- and long-term planning
As noted above, Scrum and other main-stream ASD frameworks have methodological arsenals that deal with short-term planning quite well. However, are there any approaches or techniques aimed at balancing short- and long-term planning? In Agile Planning and Estimation, Mike Cohn addresses this question with a model called the ‘planning onion’ (Cohn, 2005, p. 28). According to Cohn, Agile teams should plan at least at the Day, Iteration, and Release levels. Release planning covers time period longer than one Sprint, product level
planning looks beyond one release, and portfolio-level planning focuses on the selection of the products to be developed (Cohn, 2005, p. 28). The onion-model explains what a Scaled ASD project needs to do in order to align all planning horizons. However, no academic studies have examined this model.
SAFe uses PIs to develop a high-level plan of six iterations (see section 2.1.5.1 of this Master’s Thesis). SAFe also uses three kinds of roadmaps: the PI Roadmap, Solution Roadmap, and Portfolio Roadmap. Portfolio and Solution Roadmap covers 1-3 years of planning; PI Roadmap covers 1-3 PIs. These roadmaps act as the ‘glue that link strategy to tactics’ (Scaled Agile Framework, n.d.-a). The SAFe planning horizons model has an approach that is similar to that of Cohn’s onion model in the sense that it nests different levels of planning into each other.
As discussed in section 1.1, there is very little academic evidence that proves or disproves whether there are any benefits associated with using the various approaches to Scaled ASD. However, release planning and roadmapping are discussed in the academic literature.
Release planning is classified as ‘a coarse-grained long-time’ planning approach (in contradiction to the ‘fine-grained short-time ((iteration)) plan’) (Szke, 2011). In their systematic review of release planning models, Svahnberg et al. divide release planning into
‘strategic release planning’ and ‘road-mapping’. Roadmapping is the process of ‘assigning features to subsequent releases such that technical, resource, risk and budget constraints are met’. On the other hand, ‘Operational release planning’ is defined as planning that ‘focuses on the development of the identified features in a single software release’ (Svahnberg et al., 2010, p. 239). Roadmaps are particularly useful for solving the short- and long-term planning gap.
Kappel has observed that ‘One way that some organizations resolve the clash between the short- and long-term perspectives of participants was to do road-mapping at a higher level of abstraction–at the product category or platform level’ (Kappel, 2001, p. 47). However,
Svahnbert et al. have noted that organisations frequently use ad-hoc approaches, and there is no single model for roadmapping (Svahnbert et al., 2010). Release planning and roadmapping associated with Scaled ASD come in different forms. They may either be methods prescribed by practitioners (e.g., Cohn's planning onion and Roadmaps used in SAFe) or ad-hoc approaches developed within an organization. The use of any of these methods should either prevent or at least decrease the short- and long-term planning gap, which leads to the following proposition:
P4: Using release-planning or roadmapping techniques decreases the short- and long- term planning gap.
2.3. Proposition’s summary
The propositions developed during the literature review are summarized in Table 1.
Table 1
Proposition’s Summary
Proposition Related literature
P1: Alignment of Sprint and Product Backlog goals decreases the short- and long-term planning gap
(Uludag et al., 2018)
(Schwaber & Sutherland, 2017) (CollabNet, 2019)
P2: When short- and long-term plans are amended at the same time, short- and long-term planning gap decreases
(Dikert et al., 2016) (Conboy et al., 2011) (Ahimbisibwe et al., 2015)
(Van Waardenburg & Van Vliet, 2013) P3. Agile team's short-term decisions focus increases the short-
and long-term planning gap
(Drury et al., 2012, p. 1249) (Drury-Grogan et al., 2017, p. 259) (Moe et al., 2009, p. 22)
P4. Using release-planning or roadmapping techniques decreases the short- and long-term planning gap
(Cohn, 2005, p. 28) (Kappel, 2001, p. 47)
(Scaled Agile Framework, n.d.-a) (Svahnberg et al., 2010, p. 239) (Szke, 2011)
3. Research Design
3.1. Conceptual model
A conceptual model based on the propositions developed in the literature review section is depicted in Figure 1. The conceptual model is designed to explain why the short- and long- term planning gap appears in Scaled ASD projects. The conceptual model does not aim to capture all possible explanations for the gap. Rather, it only covers those deducted from the literature review.
Figure 1
Conceptual model of the short- and long-term planning gap
3.2. Research strategy
This study uses the mono-method strategy. The data gathered for the study was qualitative. The decision to use a qualitative research strategy was made before conducting the
actual research. The study takes a subjectivist stance towards its ontological assumptions: the
‘plan’ and the ‘planning gap’ are both socially constructed phenomena. They are ideas that have resulted from the process of using individual sense-making and imaginative activity to postulate possible futures and future actions. The ‘Plan’ and the ‘planning gap’ are not ‘objects’
that exist externally to the social activities of people. Nevertheless, they are real. This study assumes that different people can ascribe different meanings to the term ‘planning gap’ (in other words, this study acknowledges that multiple social realities exist). This study also assumes that the meanings associated with the phrase ‘planning gap’ may be shared by people operating in the same company. The literature recommends using qualitative research strategies to investigate ASD (Dybå & Dingsøyr, 2008). As far as the author of the present study can determine, there is no confirmed measurement scale in the literature that can suitably measure the planning gap with quantitative data.
3.3. Method selection
This is a multiple-case comparative study. The decision to use the mono method rather than the mixed method was based on two reasons: (1) the unclear boundaries between the phenomenon and the context, and (2) the recommendations found in the literature. The methodology used in this study is defined by Yin as ‘an empirical method that investigates a contemporary phenomenon (the ‘case’) in-depth and within its real-world context, especially when the boundaries between phenomenon and context may not be clearly evident’ (Yin, 2017). The boundaries between the short- and long-term planning gap and the context within which it appears are not clear. Hoda et al., in their recent tertiary study, recommend that scholars who wish to conduct primary research on the subject of ASD employ grounded theory, case study, and ethnography (Hoda et al., 2017, p. 17). Since the short- and long-term planning
gap is an organizational rather than individual phenomenon, the case (unit of analysis) was chosen as the organizational unit of this study (e.g., department, business unit).
The single case study method was considered during the research design phase because it is a method that offers an in-depth understanding of a given phenomenon. However, this method was not selected because it was not clear how to select either a typical, critical, or extreme case in the manner discussed by Saunders (Saunders et al., 2015, p. 186). A critical case search was deemed to be most likely very time-consuming and something that could not be feasibly conducted in the time frame available for the completion of this Master’s thesis. It was also not well suited to the resources available for this study. Another reason that a multiple case comparative study was conducted was that such a study can be used to determine if the findings of one case are replicable in other cases (Saunders et al., 2015, p. 187).
3.4. Sample design (cases selection)
This study uses non-probability purposive sampling. Since the short- and long-term planning gap is not specifically defined in the literature, it was not possible to directly select companies actively working to solve or close a gap that had formed during the development of one of their projects. To overcome this challenge, it was assumed that companies using Scaled ASD and having several teams would most likely have, due to their size and the demands of long-term planning, projects in which the gap had appeared. To be included in the study, each company had to have either medium or large organizational units with at least five ASD teams that had adopted any form of ASD (e.g., Scrum, SAFe, LeSS). To successfully conduct semi- structured interviews, the literature recommends a minimum sample size of 5-25 (Saunders et al., 2015). Consequently, this study examines four different cases and uses ten to twelve interviews (approximately three interviews per company). Access to companies was gained through professional networks.
3.5. Design of data collection instruments
3.5.1. Method selection and justification
A semi-structured interview method was chosen as the single method of data collection.
A research interview is a ‘purposive conversation between two or more people’ (Saunders et al., 2015, p. 388). Semi-structured interviews can be used both for exploratory and explanatory studies (Saunders et al., 2015, p. 393). The semi-structured interview method allows the researcher to use deductive logic to test the research propositions and reach RO2. They also allow the researcher to add exploratory questions and use inductive logic to collect critical success factors, best-practices used by practitioners, and reach RO3. The survey was considered as a possible data collection instrument, but it was rejected due to the risk that the survey questions and the notion of short- and long-term gap may have been interpreted differently by the researcher and the interviewees. Possible misinterpretation is one of the weaknesses of the survey approach (Saunders et al., 2015, p. 439). The observation method was also dismissed because permanent access to the selected cases was limited by the COVID pandemic. It also was not clear how to conduct observations during work-from-home conditions.
3.5.2. Interview procedure development
The preliminary search for interview candidates was conducted via professional networks and relationships with each company. Candidates were each sent an invite with a brief description of the interview, the necessary consent declaimers, and a copy of the interview. Confidentiality is crucial for overcoming organizational concerns associated with granting access to researchers, especially since some employees may be concerned that acknowledging the planning gap could damage the reputation of the project managers and the
company itself. Therefore, this study’s approach to confidentiality was described in the invitation email. The email explained that any identifiable information would be removed from the interview unless the interviewee explicitly asked otherwise. The interviewees were promised that the transcriptions of their interviews would not be published or shared with anyone, but some quotations might be used without identifiable information. Before each interview, the participants received the interview invitation letter (see a template of the invitation letter in Appendix C). The participants were interviewed via a video-conferencing service to eliminate risks related to the COVID pandemic. Video-conferencing services are assumed to provide a similar level of personal interaction as face-to-face interviews. Each interview was expected to last for 45-60 minutes. Each interview was recorded, and the recording began only once the interviewee explicitly agreed that the interview would begin and that it would be recorded. Every recording was transcribed, and the transcriptions were used to generate the study’s primary qualitative data.
For each case, at least two management-level practitioners were interviewed (e.g., Scrum Masters, Product Owners, Agile Project Managers, Project Managers, Software Engineering Managers, Release Managers).
3.5.3. Semi-structured interview protocol development
An interview protocol was developed to support the semi-structured interviews. The complete interview protocol can be found in Appendix A.3. Semi-structured interviews consist of open-ended and probing questions. The questions used in this study were formulated following the guidelines designed by Saunder et al. these guidelines recommend that the researcher should state both open and probing questions; avoid emotional language; and avoid leading and proposing types of questions (Saunders et al., 2015, pp. 408-409). Most of the questions were dedicated to testing propositions associated with RO1 and RO2. Some
questions were dedicated to an exploratory part of the research aimed at achieving RO3 and discovering any additional success factors associated with solving the gap between short- and long-term planning. Each interview began with a set of short, specific, and closed questions designed to help the researcher obtain basic information about the project, including the number of teams involved in the project being discussed (information which was crucial to determining the scale) as well as each interviewee’s role and position. After this first round of questions, open-ended questions relating to the research propositions were asked. The proposition-related questions are summarized in Table A1 in Appendix A. Each interview ended with several open questions designed to explore critical success factors associated with solving the short- and long-term planning gap.
3.6. Data analysis approach
Thematic analysis was used to analyse the qualitative data generated during the interviews. The purpose of thematic analysis is to ‘search for the themes, or patterns, that occur across a data set’ (Saunders et al., 2015, p. 579). Thematic analysis was chosen as this study’s data analysis method because it is a systematic approach that lends both rigour and flexibility to research. Thematic analysis allows researchers to use deductive logic to confirm research propositions and inductive logic to extract meanings from qualitative data. The thematic analysis procedures used in this study mirror those described by Saunders et al. These procedures include developing the initial set of theory-driven codes, also called ‘a priori’
codes; becoming familiar with the data (transcriptions of the interviews); coding the data using a priori and data-driven codes (derived from the data or actual terms used by participants);
searching for themes; and testing propositions (Saunders et al., 2015). Grounded Theory was considered as an alternative method for the qualitative analysis. However, Grounded Theory was rejected because there are already enough studies on this subject to generate propositions
that can be empirically tested. Furthermore, it was decided that deductive logic would be used to achieve RO1.
A priori codes were deveveloped before the interviews start. For list of a priori codes see Table B1 in Appendix B. CAQDAS (Computer Assisted Qualitative Data Analysis Software) was used to assist with the analysis process. MaxQDA software was used.
3.7. Threats to validity and reliability
The biggest threat to the validity of this study are the construct validity and the use of a single data collection method. As stated in Section 3.4, it was not feasible to directly observe the processes investigated in this study, nor was it possible to review documents that would most likely have provided valuable information. To strengthen this study’s internal validity, cross-case pattern matching was performed during the data analysis (Yin, 2017). Special attention was dedicated during the analysis stage to any negative cases, which are cases that are ‘counter to other cases’ (Saunders et al., 2015, p. 401). Close-ended questions were not used. This study’s construct validity was strengthened because the interviewer made sure to interview a different roles associated with Scaled ASD. A pre-test was conducted to improve data validity and to make sure that interview questions would be understood by the interviewees.
The literature review and discussion of deductive logic in Section 2 provide links to existing theories in a manner that improves this study’s external validity (Saunders et al., 2015, p. 400). To increase the reliability of the research, the study was rigorously designed before it was conducted, and the interview protocol that was developed (see section 3.5) was followed strictly. Other researchers can attempt to replicate the results of this study by using its case selection logic, a priori codes, and interview protocols.
3.8. Pre-test
Two pre-test interviews were conducted. Both interviewees were with IT professionals.
The pre-test was conducted following the exact interview protocol that was to be used during the proper interviews. Twenty additional minutes were given to each pre-test interviewee so that they could have an opportunity to reflect on the interview, explain which questions were not clear, and give general feedback to the interviewer. One of the interviewees works as a business analyst, and the other is a delivery manager. Both work on large software development projects (one consists of seven teams, and the other consists of about 300 people). The pre-test interviews suggested that the interview process would last for approximately 40 minutes. After the pre-test interviews, it was decided that there was room in the interview process for additional questions and follow up questions. In general, the questions were well understood by the interviewees, and, thanks to their open nature, they did not appear to lead the interviewees to give pre-determined answers. Furthermore, the open-ended questions included in the exploratory part of the interview process (5.1, see Appendix A3) yielded insightful and valuable information about critical success factors and valuable opinions about the planning gap.
The pre-test interviews also indicated that there were ways that the interview protocols and questions could be improved. First, the interviewees did not clearly understand the goal of the study. It seemed that they had not carefully read the introduction in the invitation email. To mitigate this issue, it was decided that a more detailed description of the study’s goal would be included at the start of each interview. Second, for one of the interviewees, it was not completely clear what the scope of the answers should be. For example, the interviewee was not sure if they should offer only their personal experiences or a broad range of information about their company and specific details of the case in question. It was decided to emphasis on answering only about the company and observed fact, and not broader professional during the
introduction portion of the interview process. Third, it was not obvious if the projects that the interviewees were working on had a gap between short- and long-term planning unless they were directly asked an open question about the existence of the planning gap. As a phenomenon, the short- and long-term planning gap appears to have numerous different meanings, meanings that might not be shared across different organizations. Based on this realisation, it was decided to begin the interview process with questions designed to gain information about the types of planning used in each interviewee’s Scaled ASD project as well as questions about planning-related issues in general. Though these changes ended up yielding more unstructured data at the beginning of the interviews, they were useful during the analysis phase for determining whether an interviewee had information about the planning gap and the meaning they attached to the concept itself. The pre-test interviews also revealed that decision- related question number 3.3, ‘How do you characterize the focus of the decisions made during Sprint or Iteration?’, was too broad of a question. The question was reformulated to make it narrower; ‘Can you recall any decisions made by Agile team during iterations which affected planning?’. Fifth, questions that focused on release planning generated answers that had already been given in response to earlier questions because the majority of the planning process had already been covered during the initial phase of the interview. To mitigate this issue, it was decided to include questions relating to release planning in the first part of the interview.
Finally, since ‘iteration’ was perceived as ‘program increment’ by one of the interviewees, it was decided to only use the term ‘Sprint’. For the initial interview protocol see Appendix D.
The amended interview protocol is described in Appendix E.
4. Data collection
Data was collected from four different Dutch companies that execute scaled ASD projects. The four different cases identified during the purposive sampling are summarized in Table 2.
Table 2
Overview of the cases selected for the research
Case Code Industry Organizational unit
Type of Project
Agile Methods Organizational Size, number of Agile teams
Number of interviewees
Case 1 - Bank Finance Data department
Data and compliance solutions
Scrum More than 20
teams
2
Case 2 – Media Media and TV
Engineering TV platform Scrum Six to seven teams
3
Case 3 – Food Retail
Retail Retail Core Retail backend platform
Elements of SAFe adopted in 2019.
Eight teams 3
Case 4 - Retail Retail Data Data and internal solutions
SAFe adopted prior to 2019.
More than ten teams in total
3
In total, eleven semi-structured interviews were conducted between January 7th, 2021, and February 12th, 2021. The interviewees were selected based on the different organizational roles they held at the time of the study and the applicability of those roles to the purposes of the study. Table 3 presents a short summary of the interviews. The interviewees were invited to participate through professional networks. The invitation to participate included a brief description of the study (see Appendix C). The interviews were conducted according to the final interview protocol (see Appendix E). Each interview lasted between 45 and 60 minutes.
All interviews but one were conducted online using MS Teams. One interview was conducted using Zoom video-conferencing software. Every interview was video-recorded using MS Teams and Zoom features.
Table 3
Overview of the conducted interviews
Case Code Interview Code Role Interview Date
Pre-Test I01 Delivery Manager 07-Jan-2021
Pre-Test I02 Business Analyst / Product Owner 07-Jan-2021
Case 1 - Bank I03 Product Owner 19-Jan-2021
Case 1 - Bank I04 Product Owner 26-Jan-2021
Case 2 - Media I05 Product Owner 28-Jan-2021
Case 2 - Media I06 Scrum Master 01-Feb-2021
Case 2 - Media I07 Scrum Master 04-Feb-2021
Case 3 – Food Retail
I09 Product Owners 05-Feb-2021
Case 3 – Food Retail
I10 Technical Lead / Architect 10-Feb-2021 Case 3 – Food
Retail
I11 Product Owner 11-Feb-2021
Case 4 - Retail I08 Scrum Master 05-Feb-2021
Case 4 - Retail I12 Scrum Master 12-Feb-2021
Case 4 - Retail I13 Senior Product Manager 12-Feb-2021
The interviews were transcribed using the AmberScript transcription tool (Amberscript, n.d.), and the transcriptions were manually checked and corrected against the video recording.
The transcriptions were then imported into MaxQDA software. The data was validated using participant validation, as suggested by Saunders et al. (Saunders et al., 2015, p. 206). This validation process included sending the interview transcripts to the interviewees via email. The interviewees were then asked to validate the accuracy of the transcriptions and give comments if necessary. All the interviewees except interviewee I07 responded to the validation email and confirmed the validity of the data, making only minor amendments. It was decided to completely remove the I07 interview transcript data from the analysis step due to nonresponse.
The reasons for the nonresponse were not clear.
5. Data analysis
The data analysis process consisted of several steps. First, every interview was coded using a priori codes developed during the literature review. The screenshot of the a-priori coding in MaxQDA software is shown in the Figure F1 (see Appendix F). Second, open coding was conducted over each interview (see Figure G1 in Appendix G for open coding screenshot from MaxQDA software). In total, 519 segments were coded using both a priori and in-vivo codes. Third, a cross-case analysis was performed. Cases were compared to identify commonalities, build an explanatory narrative, and test propositions. The results are presented in the next chapter.
For the fourth step in the data analysis process, a thematic analysis of in-vivo codes was performed in order to identify the critical success factors that could be extrapolated from the nine identified themes. A summary of the themes and their corresponding in-vivo codes is presented in Table 4, and the results of the study are presented in the next chapter.
Table 4
1st order in-vivo codes and 2nd order themes.
1st Order In-Vivo Codes 2nd Order Themes
- KPIs
- too high-level goals - refinement
- Epics breakdown (+)
- different kind of backlog categories
- no specifics regarding how to do refinements
T1. Rigorous requirements breakdown
- constant puzzling
- reporting on the roadmap - staff meetings
- PO Sync
T2. Continuous plan monitoring
- stakeholders
- un-attended backlog items
- continuous roadmap change process - cross-team alignment (+)
- alignment
- Sprint Review meetings - transparency
- monitoring
- no common view on planning (tool perspective) - experience in tools usage
- planning based on statistics - tools
T3. Tools usage
- aligning priorities across different teams - community of practice
- cooperation - communication - dependencies (+) - commitment
- get stakeholders committed
T4. Horizontal alignment and dependencies management
- the roadmap is communicated to the team - estimations
- engage teams in roadmapping - feedback from the teams - trust
- story mapping
- the team does not understand the roadmap - spike usage
- roadmap use is difficult at the team level
T5. Team engagement with road-mapping
- business alignment - business value sizing - competitive product owners - identify stakeholders - protecting the roadmap - Business Case
- Need validation
- WSJF
- no time pressure - amount of work is huge - roadmap rush (pressure) - overask
T.6. Advanced prioritization
6. Research results (presentation and interpretation)
6.1. Cross Case analysis
The result of the propositions support and cross-case analysis is summarized in Table 5. Strong support is indicated with a double plus sign; support is indicated with a plus sign;
weak support is indicated with a ‘+ / −’ sign; and no support is indicated with a minus sign.
Table 5
Propositions support summary across cases
Proposition Case 1
– Bank
Case 2 – Media
Case 3 – Food Retail
Case 4 – Retails P1. Alignment of Sprint and
Product Backlog goals decreases the short- and long-term planning gap.
++ ++ ++ ++
P2. When short- and long-term plans are amended at the same time, short- and long-term planning gap decreases
− − − −
P3. Agile team's short-term decisions focus increases the short- and long-term planning gap
− − − −
P4. Using release-planning and/or roadmapping techniques decreases the short- and long-term planning gap
++ ++ ++ ++
6.1.1. Short- and long-term gap estimation
The researcher estimated the short- and long-term gap in all cases. The estimation was based on interviewees' opinions since there is no established measurement scale for the phenomenon. The summary of the extent of the short- and long-term planning gap across all cases is summarized in Table 6.
Table 6
Short- and long-term planning gap across reviewed cases
Short- and long-term
planning gap
characteristics
Case 1 – Bank
Case 2 – Media Case 3 – Food Retail
Case 4 – Retails
The extent of the short- and long-term planning gap
Small Small Small Very small
Examples and issues related to the short- and long-term planning gap
Big initiative unattended in the
long-term plan;
estimation of big initiatives is challenging
Long-term strategies are too
high level;
pressure on the long-term plan to
delivery more;
estimations
Pressure on the long-term plan to
deliver more;
team’s understanding of
the long-term goals
None are mentioned
‘Case 1 – Bank’ Product Owner reported the gap as ‘spreading’ of very big initiatives over prolonged time: ‘if in the long-term planning we have one Epics [with] the huge amount of work … then … I see the tendency for most of the teams to postpone as much of this work and to kind of spread it … it’s kind of laying there unattended, and no one is really splitting it into smaller chunks’. Product Owners could recall some rare incidents when the team was not working accordingly to the roadmap due to the external dependencies ‘we are not on track due to some external dependency … but usually that’s not the case’. I04 saw more challenges in planning big, long-term initiatives ‘I think the biggest gap I feel is estimation because planning really depends on estimates’. I04 described the gap as a delay in such cases: ‘let’s say for a long term project of four quarters, about a year, maybe at the end of Q1, you will see, oh, no, this might take another [quarter] or another one year’.
‘Case 2 – Media’ Product Owner described the gap in terms of having too high-level goals: ‘I felt the gap between the strategies and the initiatives … because the strategies were so high level … so you could basically put anything there [short-term]’. I05 recognized estimations as one of the challenging areas: ‘we see the gap in our estimations. When we just defined the road map, we give very high-level estimations, and then we start working on it. We
discover more impediments and risks. [Estimations] were approved, and of course, we did not promise [to delivery], but predicted. Sometimes it doesn’t match what we produce at the end’. Product Owner perceived that gap as rather small and explained: ‘I think we work pretty closely with the road map’. I06 did not recall any critical incidents concerning the short- and long-term plan gap: ‘I cannot give you an example of not meeting some kind of big thing during long-term plan’. Based on the analysis, the author estimates the extent of the gap between short- and long-term planning as small in this case.
‘Case 3 – Food Retail’ Product Owner perceived the gap as small and mentioned that all inconsistencies are usually solved: ‘in the end, it’s always solved’. Product Owner saw the gap in the pressure from the business side: ‘there’s always pressure … they always want it tomorrow, which will never happen’. Technical Lead stated that the long-term plan is usually well executed: ‘everything is going accordingly to the roadmap’. Technical Leads mentioned issues related to short- and long-term planning, when team members did not always understand the long-term reasons of particular requirements: ‘the big gap is that due to its too small stories, sometimes teams don’t know the reason of this story and the result of this story’.
‘Case 4 – Retail’ Scrum Master perceived the gap as very small: ‘So we will have some deviation from the [long-term] plan doing something faster or doing something longer’ and
‘My personal opinion is that it is going well because it's already on rails … it’s a predictable and we can estimate well now because team is mature’.
6.1.2. Goals alignment
Every case involved in the study was found to use an elaborate process for aligning long-term goals with those of the Sprint plan. A variety of techniques are used, including Product Backlog management, roadmaps, and OKR (Objectives and Key Results). The data reveals that goal alignment is primarily done by Product Owners. The Product Owners
frequently use a ‘breakdown’ process that breaks high-level goals, typically called ‘Epics’
(product backlog items, roadmap items) into smaller tasks or work-items called ‘Features,’ and
‘Stories.’ Epic, Feature, and Story, though their relationships vary from case to case, follow a hierarchy of priorities and are arrived at via an analytical process driven by either the Business Analyst or Product Owner. The tasks associated with each Sprint are prioritized in accordance with the priorities of the Stories and Epics, which, in turn, are clearly aligned to the Product Backlog and roadmap priorities. This prioritization process was described by I05:
‘For each roadmap item, I create an Epic. Onto this Epic, I have features and after and the features that have user stories. So I start working on the Epics, which are on the top of this prioritized roadmap. I prepare user stories, and … they just end up at the top of the backlog, and we start working on them, so it's always from the top and next from the top, the next’.
The ‘breakdown’ process was described by the Product Owner from ‘Case 1 – Bank’
in a similar manner: ‘So we use Epics and then the POs generally break down the epic on refinements with the team. So, we can have, maybe, a couple of very high-level features, three or four features, and I think breaking down the epic into features’. I13, when discussing ‘Case 4 – Retail’, said the following: ‘We handle one milestone in the planning, and that milestone will be broken up into Epics. …and, later on, we'll break down an epic into user stories or maybe into spikes if we don't know the entire scope of a particular story’.
The cases demonstrated additional priority and goal alignment mechanisms, including alignment with management, advanced backlog management practices, and frequent meetings.
I03 stated: ‘And actually before Sprint planning, we have Sprint alignment with the management … in which I discuss what I have planned for the next Sprint’. ‘Case 3 – Food Retails’ adopted a ‘data-driven backlog’ mechanism that helps Product Owners align long-term planning at the strategy level with priorities on the level of two Sprints. This was described as
