Controlling agile development from a vendor perspective

(1)

Controlling agile development from a vendor perspective

Author Ing. S.M. Zeinstra BSc E-mail : s.zeinstra@gmail.com Student number : S1760564 Institution Rijksuniversiteit Groningen

Faculty of Economics and Business MSc. Technology Management

University of Groningen Prof. Dr. H.G. Sol (Supervisor)

Mr. N.R.T.P. van Beest, MSc (Co-assessor)

Supervisors COOLProfs Mr. A. Willemse

Mrs. I. Duivenvoorde

Date

(2)

Management Summary

COOLProfs is an organisation which develops tailored software solutions using the OutSystems model driven development environment. This development environment has been in use by the company for four years and around ten projects have been delivered using an agile software development method. Not all projects showed a discrepancy between estimated and actual development effort. In a number of cases more than the budgeted development effort was used. This was the main motive for the management to investigate the causes for the problems and design a solution for it.

After an analysis and diagnosis of the situation, problems have been identified with:

- Unclear vision of the solution, between developers and customer as well as with the customer organisation itself.

- External dependencies on interfaces to other systems or information provision. - Unclear division of tasks within the development team.

- Insufficient information about the progress of the development.

Based on the above stated problems, a redesign has been made in the form of a number of recommendations for the development process. These recommendations have been made in line with the preconditions for a redesign as they have been found in the interviews with during this research. These recommendations are described in more detail in chapter 7.2.

- The projects need a clear boundary. By identifying the external dependencies beforehand, risks associated with these dependencies can be avoided.

- A clear goal for development is needed. This is done by creating a vision which is supported by all stakeholders and gaining more insight into technical complexity.

- The customer must accept delivered functionality in time and has to play a more active role in definition acceptance criteria.

- Members of the development team must know what their responsibilities are and the tasks have to be divided realistically.

- Regular feedback sessions with all stakeholders will help getting insight into how the stakeholders are aligned and enables adjusting the development goals.

(3)

6.1 Case 1: Global workflow and supply chain management system 33 6.2 Case 2: Process control console 35 6.3 Case 3: Customer portal 37 6.4 Case 4: Online Insurance calculation 38 6.5 Case 5: Maintenance advise for window frames 39 6.6 Control theory in practice Fout! Bladwijzer niet gedefinieerd. 7 Redesign ... 41

7.1 Preconditions for redesign 41

(4)

7.3 Redesign overview Fout! Bladwijzer niet gedefinieerd. 8 Conclusion ... 44 9 References ... 46

9.1 Literature & books 46

9.2 Web & Other resources 49

10 Appendix ... 50

10.1 Appendix I: Organisational Chart 51

10.2 Appendix II: Problem overview Fout! Bladwijzer niet gedefinieerd.

10.3 Appendix III: OutSystems task-division 52

(5)

List of abbreviations

API

Application programming interfaces

DSDM

Dynamic systems development method

ERP

Enterprise resource planning

MDD

Model driven development

RAD

Rapid application development

RFP

Request for proposal

(6)

1 Introduction

This report describes the research which has been done for the Dutch software company COOLProfs. The goal of this research is to find possibilities to improve the efficiency of their agile software development process. This introduction chapter introduces the company, the used technology and the development process in question. The end of this chapter gives a short description of the current problem situation and a description of the of the document structure.

1.1 COOLProfs

COOLProfs is an organisation that has been developing tailor made transactional information systems for automating administrative tasks at banks, insurance companies, wholesale, retail and government since its foundation in 2000. The company detects a trend in the market where more and more standardized software solutions are being used, which does not always match with the specific needs of their customers. COOLProfs delivers its solutions on the areas where its customers need specialized software, tailored to their specific situations.

The main part of the COOLProfs business is software solution development. The company was founded in 2000 as a CA:Gen consultancy firm, developing transaction intensive systems by using the model driven development (MDD) platform CA:Gen. The services delivered span the whole software life cycle, including analysis, design, build, test, implementation, maintenance and project management. Over the years, the popularity of the CA:Gen platform decreased. For COOLProfs, this means that there are no new customers using CA:Gen and the current business comes from extending and supporting existing systems. The management realizes the need for new business to ensure continuity of the company.

1.2 OutSystems agile platform

The search for new business led to the selection of a new development platform. This platform had to support the development of tailor made software and in 2008 the choice was made to invest in the OutSystems agile platform. This platform is aimed at supporting the whole lifecycle of delivering and managing web based business applications. The OutSystems agile platform uses similar modelling-techniques as CA:Gen and was selected from a range of model driven development platforms, because this tool best reflected the whole development process from idea to modelling of the transaction to implementation of the solution.

(7)

1.3 The delivery process

As said in the previous paragraph, the OutSystems platform and the development process is designed around agile development principles. The term agile has been introduced in 2001 by a group of software developers who were discussing lightweight development methods. Together, they defined the principles of Figure 1.1 in the agile manifesto.

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools

Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more. Figure 1.1 - Agile manifesto [1]

The agile methods can be seen as a reaction on traditional sequential software development methods. Sequential methods emphasize a rationalized engineering-based approach which is based on the assumption that problems are specifiable and that optimal and predictable solutions exist for every problem (Dybå & Dingsøyr, 2008).

In contrast, agile methods see software development as a process that cannot be defined in too much detail, because too much change occurs during the time that the team is developing the product (Williams, 2003). The agile development methods are based on the empirical process, which necessitates short “inspect-and-adapt” cycles and frequent, short feedback loops (Williams, 2003). These short inspect-and-adapt cycles are suggested to help agile methodologies better handle conflicting and unpredictable demands.

(8)

Figure 1.2 - OutSystems development method

The purpose of the pre-development phase is to understand what will be built, why it should be built, how much it will likely cost to build and in what order it should be built. The analysis phase starts when a customer deposits a request for proposal (RFP). The detail of these requests can range from a high level idea, containing only general outlines of the envisioned system to more detailed plans when the customer has done his own preliminary analysis. During the analysis, a shared vision of the solution between customer and COOLProfs should emerge. The analysis process will lead to a development proposal, comprising a list of features which have to be developed, estimated development effort and the cost for the delivery of the proposed solution.

When the customer accepts the proposal, the features from analysis phase are prioritized, together with stakeholders from the business, on its added value to the business. The goal is to deliver the most value adding functionality first.

Most of the OutSystems development projects are initially contracted by fixed quality, fixed time, fixed price contracts. Working with fixed price contracts is important for COOLProfs because it is aligned with the value proposition “Our customers get what they need, at the fixed date, for a fixed price”. With this value proposition COOLProfs wants to distinguish their selves from competitors in the market.

1.4 The problem situation

The fixed price projects were problematic in a number of the completed projects. Many of the fixed price projects have been delivered using a higher consumption of development effort than estimated in the analysis phase.

(9)

Figure 1.3 - Project effort estimation versus actual effort spent

The graph in Figure 1.3 shows the results of the initial analysis of the project information as it is collected from the administrative system from COOLProfs. The overshoot percentage is based on the hours spent on the project by the COOLProfs employees. This information is gathered from the back-office system where the employees who are assigned on the projects enter information about their time-spending. This development effort is compared to the estimated effort as it is described in the contracts which are used to fund projects. To ensure that the problem of the difference between estimated and actual development effort is not a problem caused by a too high target, it has to be shown that the goal of having a better alignment between the estimated and actual development effort is possible.

1.5 Document structure

(10)

2 Research design

In this chapter the research design is presented. In section 2.1 the used structure is explained. Following this, section 2.2 discusses the problems with the development process in more detail. Using this description, the research goal and problem statement are described in section 2.3. Finally, section 2.4 introduces the control theory of de Leeuw (1980) which is used as analysis framework.

2.1 Research structure

In the introduction chapter it becomes clear that the management wants advice on how to improve the development method of COOLProfs which is a design oriented goal. It means that this research will identify possible improvements for the development process which reduce discrepancy between estimates and actual development effort. Because of the design orientation, this research is structured along van Striens' (1997) regulative cycle as shown in Figure 2.1.

Figure 2.1 - Regulative cycle (van Strien, 1997)

The regulative cycle consists of the diagnosis, design, implementation and evaluation phases. The problem selection and therefore the decision to do research on the OutSystems development process result from insights of the COOLProfs management. The characteristics mentioned in the quote by van Aken et al., (2007) below make the problem with development process a typical business problem selected from a ‘mess’ of issues. The remainder of the redesign chapter will be used to clarify the problem mess and problem definition.

Busin

es

s

p

rob

(11)

The Oxford dictionary defines an analysis as “the detailed study or examination of something in order to understand more about it” (Hornby, 2010).To better understand the given problem, first, a study will be made on the development process as it is currently practiced by COOLProfs. Secondly, literature will be explored to broaden the view on the problem identifying possible alternative explanations for the problem. In this report, chapter three provides an overview of literature on agile software development, while chapter four provides a deeper view into the current development process. The insights from these chapters will lead to a framework the diagnosis, which is described in chapter five.

Diagnosing is defined in the Oxford dictionary as “the act of discovering or identifying the exact cause of a problem” (Hornby, 2010). To diagnose this business problem, means, to identify what elements and relations can be determined as actual causes for the problems. This will be done by diagnosing the problems of a number of projects using the framework as described in the analysis.

Because of the limited time available for this research, this thesis will follow the cycle from the problem mess until the plan of action. The steps of intervention and evaluation have to be done by the organization itself.

2.2 Problems with the delivery process

To get a better understanding about the current problem mess, interviews have been done with the COOLProfs management. An overview of the management structure, including the responsibilities of the members is provided in appendix I.

The introduction of the OutSystems platform meant a change in the development process of COOLProfs. Traditionally, the CA:Gen development processes were based on the dynamic systems development method (DSDM). The company built a stable base of knowledge about this development method and has over 12 years of experience in applying it to the development of transactional processing systems.

The introduction of the OutSystems platform has caused a number of changes within the company. Firstly OutSystems is primarily designed for the development of Web-based business applications and the change in the tool also meant that different types of customers has been reached. Where the focus for the CA:Gen projects was mainly on the financial and insurance sector, the switch to Web-based development has also lead to new customers in the Governmental sector and certain Small and Medium Enterprises.

(12)

business value, fixed time boxes, significant customer involvement and developing high priority items first.

As is stated in chapter one, the organisation is having problems with the control of the development effort. A number of complaints revolve around the initial analysis phase. Most of the interviewees state that they do not feel comfortable with the current level detail of requirements specification. This does not give sufficient insight to be able to give an accurate estimation of the development effort.

The agile principles are partly based the premise the not all uncertainty can be foreseen (Boehm & Turner, 2003). One of the causes of uncertainty is that customers cannot exactly know what they want before they have seen some working functionality (Schiel, 2010). Therefore it is very likely that customers will gain new insights, causing requirements to change. Therefore, agile methodologies have less focus on specifying requirements beforehand. The definition of requirements takes much time and since they are prone to change, writing specifications and documentation beforehand can is a wasteful activity.

However, COOLProfs does need to make an offer to the customers and therefore needs an estimation of required effort to realize a solution. As explained in chapter one, estimates are based on the features that need to be built to realize a user story. This should deliver a “good enough” estimate for the scope. Based on this estimate, the overall time-box and number of sprints is determined. The customer knows that changes are possible during the development process. Some rework is calculated in the estimation of the projects, but one of the possible problems mentioned in the interviews with the management, is expected to come from a large amount of rework. The management is asking itself how the projects can be organized in such a way that there is flexibility for changes, but are managed in such a way that they stay more within a the margins of the estimation, or that the margins of the estimations are more realistic towards the development.

2.3 Problem statement

The previous section gives more insight into the motivation for this research and why this problem is selected. In this section, the research objective and question are defined.

It is unclear what the exact causes are for the current problems with the delivery process. A diagnosis of the completed projects is needed to come to an explanation and solutions for the current situation. Therefor the object for this research is stated as follows:

Rese

ar

ch

ob

ject

iv

e

(13)

In order to reach the research objective, the following research question is formulated:

Rese

ar

ch

ques

ti

on

What are the main causes for the current discrepancy between estimated and actual effort and what interventions are needed to reduce the discrepancy between project estimations and actual development effort?

Now the research goal and question are clear, the next section will introduce the framework which will be used throughout the analysis and diagnostic phases of this research.

2.4 Agile development: a control problem

This research will try to explain the problems with the current development process by examining experienced problems in some of the completed projects. This problem is based on a problem which is measured in the output of the delivery process by looking at the used development effort. Formulated as a control problem, the development process is not being controlled to match the expected development effort. The theory used in this research to explain the main concepts of process control is the control theory of de Leeuw (1980).

The control theory describes how to accomplish effective control of a system and for that purpose connects elements as shown in Figure 2.2. The theory describes a system which is controlled by a governing body. The steered system needs control because operation of the system is influenced by its environment. The environment causes contingencies which have influence on the output of the system. The governing body tries to influence the system to make it produce an output as wished. To control the steered system, the governing body should have a number of tools to influence the steered system called control mechanisms. How the control mechanisms are used should be based on the information about the actual state of the system and the goals which guide control decisions.

Governing body

Steered system Information Control

mechanism

Input Output

Figure 2.2 – Control model (de Leeuw, 1980)

De Leeuw (1980) gives five conditions that are required for effective steering of a system:

(14)

- Information about the environment and the state of the system. The governing body has to take measures based on the state of the system, if there is incomplete or invalid information there can be a wrong impression of the system, possibly leading to ineffective decisions.

- Control mechanisms which have effect on the disruptions are needed that have influence on these disruptions.

- Information processing capacity. The information about the environment and state of the system has to be processed by the governing body. To be able to process the information, the governing body needs to have sufficient processing capacity.

- Model of the controlled system. A model can be used to predict the effects of a control mechanism.

(15)

3 Agile software development

Agile software development is a topic attracting more and more attention in the world of the practitioners (West & Grant, 2010). Also amongst scholars, the number in published journal papers has increased over the last decade (Dingsøyr, Nerur, Balijepally, & Moe, 2012). The literature review of Dybå & Dingsøyr (2008), states that agile development methods are easy to adopt and work well, they note improvements in the areas of customer collaboration, work processes for defect handling, learning in pair programming, thinking ahead for management, focusing on current work for engineers and estimation. A limitation of agile which was repeatedly mentioned in literature, is the lack of attention to design and architectural issues (Dyba & Dingsoyr, 2009).

This chapter provides an overview of the literature that is written about agile software development. Firstly the history behind agile is described to show some of the reasoning behind the agile principles. Then based on the recent developments in literature a working definition is defined. Next, the impact of agile on the lifecycle of software development is described. Finally, home grounds for both sequential and agile methods are given.

3.1 Roots of agile development methods

As said in the in the introduction, the agile methodologies can be seen as a reaction on traditional sequential software development methods. Not all ideas of agile development are new, and actually most of them have long roots, originating from before the creation of the agile manifesto (Abbas, Gravell, & Wills, 2008). The newness of the agile stream is that it is now presented as a method for developing software (Abbas et al., 2008).

Since the eighties, problems have been identified with sequential software development. For example, a number of authors believed that most users do not have a clear idea about their needs and that a system’s users seldom know exactly what they want and cannot articulate all they know (Gladden, 1982; Parnas & Clements, 1986). Even if all requirements could be stated, there are many details that can only be discovered once we are well into implementation because only a limited amount of complexity can be mastered (Gilb, 1985; Parnas & Clements, 1986). Furthermore, Parnas & Clements (1986) say that, apart from mastering complexity, software development processes are influenced by external forces that can lead to changes in requirements, some of which may invalidate earlier decisions. To be able to deal with those problems the authors suggest using multiple development iterations that allow number of practices to overcome the previously stated difficulties.

(16)

be more positively influencing to the success of the project than to see the proposed system in operation (Gladden, 1982). This is in line with the agile value: “working software over comprehensive documentation”.

Secondly, incremental development approach can also have advantages for the developing party, since they can take advantage of what was being learned during the development (Basil & Turner, 1975). This reduces complexity because instead of delivering software in one release with a lot of complexity, there are multiple small releases with limited complexity. For example, in the construction phase of Rapid Application Development (RAD), each transaction that is developed has its iteration, consisting of detailed design and coding (Martin, 1991). This way it is possible to “respond to change” instead of trying to “follow a plan”.

In 1982, Gladden also emphasised the need of a focus on customer value. He reasoned that system objectives plus physical demonstrations will result in a successful product. By successful project he meant that a product that performs the function intended and satisfies the customer’s need. Therefore it is necessary to measure the added-value and be oriented on results instead of processes.

McCracken & Jackson (1982) recommended a close relation with the user: “development proceeds step-by-step with the user, as insight into the user’s own environment and needs is accumulated”. Gilb(1985) stressed being user oriented: “With evolutionary delivery the situation is changed. The developer is specifically charged with listening to the user reactions early and often.” This has some similarity to stating that “customer collaboration” should be valued over “contract negotiation”.

3.2 Definition of agility

Before jumping to a definition of agility, it is important to look at what exactly is a software development method and what is a software development process. Hevner & Chatterjee (2010) make a clear distinction between the two:

Soft

war

e

d

ev

elo

p

men

t

p

roce

ss

A software development process is a pattern of activities, practices and transformations

that support managers and engineers in the use of technology to development and maintain software systems (Hevner & Chatterjee, 2010).

Soft

war

e

d

ev

e

lo

p

men

t

m

et

h

od

A software development method is a set of principles, models and techniques for

(17)

In other words, the development process sets the order of development phases and the criteria for transitioning from one phase to the next, while the development method defines what is to be done in each phase and how the artefacts of the phase are represented.

In the literature about agile software development processes as well as methods, many interpretations can be found. This is explained in the research by Abrahamsson, Conboy, & Wang (2009) who state that in the context of software development, agility as a concept needs to be multifaceted and contextual, and that agility is achieved through various different means depending on the environment. Based on this argument, the authors say organisations needs to adopt their own individual interpretation of what agility means to them, as opposed to relying on a single commercial version such as XP or Scrum. However, in order to significantly advance this area of research, it is important to have some solid platform on which to build a cohesive body of knowledge (Abrahamsson et al., 2009).

Conboy (2009) provides a comprehensive definition of software development agility by systematically examining its various facets and definitions from related disciplines. He explains agility as flexibility and leanness. Flexibility relates to the ability of a systems development method to “create change, or proactively, reactively, or inherently embrace change in a timely manner, through its internal components and its relationships with its environment”, while leanness is defined as the “contribution to perceived customer value through economy, quality, and simplicity.” The following definition will be used in this research to refer to the agility of a software development method.

Defi

n

it

io

n

of a

gi

li

ty

Agility is “the continued readiness to rapidly or inherently create change, proactively or reactively embrace change, and learn from change while contributing to perceived customer value (economy, quality, and simplicity), through its collective components and relationships with its environment (Conboy, 2009).

The two facets of Conboy's (2009) definition can be related to the research question. The flexibility aspect is regarded as important by the COOLProfs management. Proactively embracing change is done by the quick demonstration and delivery of working functionality and thereby quickly receiving feedback from customers. The potential of improving quality and increasing customer value is recognised. However, a too high level of leanness is expected to cause some of the problems because the projects are insufficiently delineated.

3.3 The agile process

(18)

Figure 3.1 Agile & traditional lifecycles (Shore & Warden, 2007)

Even though different development processes have different characteristics, the agile methods contain similar phases as the sequential methods: planning, design, coding, testing and deployment (Shore & Warden, 2007). However, they are carried out in agile working practices and therefore do not take place in a sequential order. This is graphically shown in Figure 3.1. In general, even though there is some upfront planning done, it can be said that the activities focus more on the current iteration than the overall project and in each iteration, the design, coding and testing are effectively performed simultaneously (Shore & Warden, 2007).

Not all development methods based on the agile lifecycle support the same phases in the software development life-cycle (Abrahamsson, Salo, Ronkainen, & Warsta, 2002). Figure 3.2 shows which phases of software development are supported, and into what depth they are supported by the Scrum and DSDM methods. The methods are divided in three blocks, where the first shows if a method supports project management, the second whether a process is described, the third indicates if practices, activities and work products are given that could be followed and used.

(19)

Figure 3.2 - Agile methods, adopted from Abrahamsson et al. (2002)

(20)

3.4 Method home grounds

Traditional life cycle models are “process-oriented” and “plan-driven”, heavyweight life cycle models advocate extensive planning, codified processes, and rigorous reuse to make development an efficient and predictable activity (Boehm, 2002). These life cycle models are document-driven as the major focus is on a plan which includes documented process procedures. Each documented process procedure involves tasks and milestone plans as well as product development strategies that involve requirements, design, and architectural plans (Guntamukkala, Wen, & Tarn, 2006).

Boehm & Turner (2003) describe a number of characteristics which show the home grounds for agile methods compared to what he calls “plan-driven methods”, referring to sequential, incremental or spiral methods. In between these home grounds is a grey area where organizations have to find there place, based on their specific situation. Boehm & Turner (2003) point out the different project characteristics of different development methods from a risk-driven perspective. Table 3.1 shows the characterization of both approaches.

Table 3.1 - Agile and Plan-driven methods (Boehm & Turner, 2003)

Project characteristics Agile home ground Plan-driven home ground

Application

Primary goals Rapid value, responding to change Predictability, stability, High assurance Size Smaller teams and projects Larger teams and projects

Environment Turbulent, high change, project focus Stable, low change, focus on project and organization

Management

Customer relations Dedicated on-site, focus on product increments

As needed interactions, focus on contract provisions Planning and control Internalized plans, qualitative control Documented plans, quantitative control

Communications Tacit interpersonal knowledge Explicit documented knowledge

Technical

Requirements Prioritized informal stories and test cases

undergoing unforeseeable change Formalized project, capability, interface, quality, foreseeable evolution requirements Development Simple design, short increments,

refactoring assumed inexpensive

Extensive design, longer increments, refactoring assumed expensive

Test Executable test cases define requirements, testing

Documented test plans and procedures

Personnel

Customers Collaborative, representative, authorized, committed, and knowledgeable (Crack)

Crack performers, not always co-located Developers Skilled personnel Skilled personnel early, during development less

skilled personnel Culture Comfort and empowerment via many

degrees of freedom

Comfort and empowerment via framework of policies and procedures

These characterisations show that between the agile home grounds, there is a large grey area, where organisation will have to find their own place and adapt their method to tailor it to their own situation.

(21)

4 The OutSystems development process

After providing some theoretical insights in the previous chapter, this chapter will give more insight into how the OutSystems development method offers control on the agile development process. In the OutSystems documentation [2] the following quote is found “Our focus on timely delivery and constant business alignment lead us to choose Scrum as the framework to support our delivery process”. How this timely delivery and creation of business alignment is embodied in the development process is described in this chapter. Firstly, a description of the persons are usually involved in the process and what artefacts are used for inspecting and controlling the process. Then it is shown how these roles and artefacts are used in the pre-development and development processes. The last chapter offers a comparison with the Scrum method.

4.1 Roles and artefacts

Within the OutSystems development method there are a number of role described. Each of these roles has an accompanied training program which is offered partly on the OutSystems website, the other part is given as training. A description of the tasks for each main role is given in Table 4.1.

Table 4.1 - Roles in the OutSystems process

Role Task description

Engagement manager

The engagement steers the project from a business perspective, working between the client and the delivery team and ensuring constant alignment with the business needs and concerns.

Delivery manager The delivery manager is in charge of building the solution that addresses the customer requirements.

The delivery manager handles all technical management activities and key architectural decisions, guiding the work of the developers in the team.

Developer The developer is in charge of actually implementing the target solution and performing integrations with external systems. There are three levels of developers, trainee, associate and professional. For small projects, one professional developer is recommended. Medium complexity projects will often have a mix of experienced and less experienced developers.

Business manager

The business manager is the customer representative empowered to take project decisions when necessary in direct coordination with the Engagement Manager

Business experts The business experts are the stakeholders with hands-on domain knowledge

Sales team The sales-team consists of an commercial employee who does the initial vision building and an engagement manager responsible for the estimation

During the development there are three main artefacts which are used to control the development, and to be informed about the progress of the development, these artefacts are described in Table 4.2.

Table 4.2 - Artefacts of the OutSystems process

Artefact Purpose

(22)

that are planned to be processed in the current or next sprint.

Project backlog The Project backlog is used to list the features, functions, technologies, enhancements and bug fixes that are planned to be processed in the current or next sprint.

Development Progress bar

A graphical representation showing actual development progress compared to initial plan.

4.2 The delivery process: pre-development

Every new project starts with an initial planning phase called the development phase. The pre-development phase follows the process flow of Figure 4.1 and uses a customer request for a proposed system as an input and leads to a proposed solution, complete with a quotation and initial planning.

The first step is the analysis. In the analysis phase, the customer needs are captured and a business vision is defined. The business vision has to be aligned with the needs of the customer and gives insight into what business value the proposed solution should deliver to the customer.

Figure 4.1 – Pre-development phase

After agreeing on the vision, the next step is to describe scope of system in more detail. This description is done on three different levels, packages, stories and features. Packages are collections of user stories that revolve around one big module or component of the solutions architecture. A user story summarizes in a couple of informal sentences a small block of desired functionality by describing a process or action that the system must implement. A user story aggregates a set of features, which on their turn, detail specific functions or requirements of the system. The user story must be detailed enough to give an estimation of the development effort.

After the solution is described to the level of the features, the development effort is estimated. This is typically done by an engagement manager but can also be done by an experienced developer. The first step in the sizing process is linking each feature with one or more general descriptions of an actual functionality. These descriptions of functionality are called patterns. Examples of these patterns are: listings used to show a list of entities, edit-screens to create or modify an entity, integrations with external data sources and implementation of business rules. These patterns all have a predefined estimation of development effort, which can be influenced by setting the complexity of the feature as simple, average or complex.

(23)

phase to calculate the overall effort estimate for the project. The categories for mark-ups is shown in Table 4.3.

Table 4.3 - Estimation mark-ups

 Project Management  Analysis  Design  Setup  Bug Fixing  installing infrastructure  Deployment  Training  Demo’s  Documentation

 Change request slack

 Post-Production Tuning

 Warranty Effort

 Testing

The relative weight of the mark-ups for these activities are influenced by factors as customer complexity, project complexity, if it is the first OutSystems installation, if the project is a first release, how many analysis resources are available and the length of the warranty period. Added together, this leads to an estimation of development effort for the project as a whole, a quote for the delivery of the solution and an initial plan for the development and delivery of the solution.

The final step in the pre-development phase, the project proposal is re-evaluated with the customer. Here the complete plan with stories and features is proposed to the customer and a negotiation takes place to re-align the proposal with the customer.

4.3 The delivery process: development

When the customer accepts the proposal, the development phase commences. As explained in the first chapter, the development of the solution is an iterative process. The steps of the process are shown in Figure 4.2.

Figure 4.2 - Development process model

(24)

feedback and clarify features. Everybody has to be aware of the time box approach and there is a fixed budget. All the concepts should be clearly explained to and acknowledged by the customer. Also the responsibilities of all people involved should be clearly explained. If project risks are worth mentioning, these should be addressed during this meeting. A project plan will be presented focussing on the demo’s en delivery dates. A tight schedule should be agreed upon for the analysis step.

The initial analysis is a set of meetings led by engagement manager, the goal is to go through all of the modules to detail initial requirement. If new requirements are identified, they will be added to the project backlog. It is not a fully detailed analysis that will be done at the start of each sprint. The delivery manager together with the engagement manager should both attend the initial analysis to understand the business requirements and to map them into a solution.

Based on the initial analysis, it should be possible to define the solution architecture. The mapping of the business requirements to the solution is done in the solution design meeting, this will make clear what business value is associated with what part of the solution. The architecture consists of solution’s modules definition, database entity model and integration application programming interfaces (API. Test cases should be created here so that the solution can be validated during the development.

Development

Each sprint starts with sprint backlog settlement. During this meeting, the engagement manager, delivery manager and business manager will agree on the goals for the current sprint. This is done by defining and negotiating what backlog requirements will be realized in next sprints. Since new requirements might have emerged during previous project phases, the project backlog is reviewed in the backlog settlement. If new items have been added, the backlog has to be reprioritized. New features will be estimated on expected effort by the delivery manager, so the engagement manager has all the information to negotiate about the features that will be built in the next sprint.

The focused analysis step is used to get detailed information of each feature that is selected for the sprint. This is typically done in conjunction with the business users that have domain knowledge of the requirements. The perception of the requirements might change during this analysis and changing estimates might change the planning for the current sprint.

(25)

The demo and feedback Session conclude the iterative development cycle. The demo session demonstrates the built functionality to the quality insurance environment (key users and Responsible Business manager). The demo will be given by the engagement manager, who should get the acceptance from the customer on the demo, this is the sprint acceptance. There will be a feedback session where new issues can be captured from key users which are placed on the product backlog. The presented version will be kept on the acceptance or testing environment so the customer can keep on testing and giving feedback after the session has ended. When the development of the solution is not yet done, the cycle will start again with the sprint backlog settlement.

Solution release

After the last sprint is closed it is time to prepare and launch the solution. All of the functionality that fits into the time box has been implemented, and the end-users will be trained. Several training sessions will be planned to give the ownership of the solution to the end-users. When there are many end-users or the end-users are not yet known, a train-the-trainer approach will be used. During the training sessions feedback is welcome, however it is not a feedback session and the time spent on feedback should be short. Some of the feedback can be used for the post-production tuning, where the application is tuned for better user adoption.

Testing is critical to ensure solution quality and adoption. The defined tests are executed by the development team or the user’s quality assurance team. A business validation should be done by the key users to ensure that everything has been implemented according to plan. Any detected bugs should be fixed and identified change requests should be added to the product backlog for upcoming releases.

For the launch phase, the production environment should be in place and the rollout plan will be executed so that the solution can be rolled out into production. Sometimes a soft launch may be used, this means that the solution is used by a limited number of end-users to ensure everything is working according to plan, this is often done when an existing application is replaced.

The post-production tuning phase is aims at creating higher end-user adoption. This phase is used to implement issues that are critical to the end-user. For example this can be usability or performance improvements based on issues from the training phase.

4.4 OutSystems vs. Scrum

(26)

In Figure 4.3, an overview of the Scrum process can be seen. The frequent inspection is done on different levels. Firstly, after each sprint, the goals for development are re-evaluated with the customer. This is done in the product backlog refinement. Secondly, the progress of the actual development is monitored on a daily basis in the daily Scrum meeting. The OutSystems method uses the same meetings and artefacts for process control. However, the OutSystems method goes into more detail into how and where meetings should take place and what stakeholders should be involved. Scrum has one meeting that is not present in the OutSystems process, this is the sprint retrospective. During this meeting, the project team performs a review of what went well, what did not go well in the most recent sprint and anticipates the next sprint.

Figure 4.3 - Scrum process (Sutherland, 2010)

In Scrum, the product owner is the one and only person responsible for managing and controlling the product backlog. This is the person who is officially responsible for the value of the work done. This person maintains the product backlog and ensures that it is visible to everyone. The product owner has to make sure that all members of the development team know what items have the highest priority, so everyone knows the order in which the items will be addressed. The corresponding role in OutSystems is the engagement manager; the next table highlight the differences between the product owner and the engagement manager. A difference between both methods however is that the Scrum product owner is often a business manager, where in OutSystems these tasks are done by the engagement manager.

Table 4.4 - Comparison Scrum & OutSystems [2]

Scrum: product owner OutSystems: engagement manager

Product management Establish Target market, positioning, marketing, distribution channels and pricing

Engagement manager core focus is on product design and alignment with the Business Manager

Roll-out Roll out procedures and the roll out itself is mostly the operations team concern.

(27)

The Scrum master is responsible for ensuring that Scrum values, practices and rules are enacted and enforced. The Scrum master is the driving force behind the entire Scrum and helps the Scrum team and the organization adopt and use Scrum to produce a higher quality product. The Scrum master is not the manager but leads by coaching, teaching and supporting the team. The Scrum master helps the team understand and use self-management and cross-functionality. The corresponding role in the OutSystems method is the delivery manager. The differences between these roles are shown in Table 4.5.

Table 4.5 - Scrum master versus Delivery manager [2]

Scrum Master OutSystems: delivery manager

Sprint Backlog Sprint planning and settlement occurs between the product owner and the team during sprint planning and backlog reviews, the sprint Backlog is owned by the team.

Sprint planning and settlement occurs between the engagement manager and the delivery manager who articulates decisions with the team when needed. The sprint backlog is owned by the delivery manager.

Solution Architecture

The solution architect emerges or is selected by the team. With large systems some members specialize on each sub product.

The delivery manager owns the solution architecture, sharing its design with experienced team members and is the ultimate responsible for keeping it aligned with the engagement manager and the rest of the team.

Team Coordination

Teams self-organize to turn the requirements and technology into product functionality.

The delivery manager coordinates and performs activities together with team members.

To summarize, the main differences between the OutSystems and Scrum method are:

 OutSystems has a more detailed description of activities.

 The product owner is typically business manager, engagement manager is on the side of the development team.

 The product owner in Scrum is responsible for prioritizing tasks, in OutSystems this is a shared responsibility between engagement manager and business owner.

 Roles of the Scrum master and product owner are divided over two different roles in the OutSystems development method.

 The sprint retrospective meeting is not present.

(28)

5 Analysis

This chapter applies the information from the analysis and comes to a number of possible causes, based of control theory. Furthermore this chapter explains how this model can be used to do research on the projects and what data is needed to come to an explanation for the current situation.

5.1 Control theory in an agile process

When the control theory is applied on the OutSystems software development method, the five preconditions for effective control give a number of possible causes for the current situation. Firstly, the concepts of the control model will be described based on the current development method. The steered system in this research is the iterative development.

In the OutSystems development method, the governing body consists out of the business manager and the engagement manager. Together they make the decisions about what functionality should be developed. These decisions should be based on the information from the system about the progress of the development and the available budget and time constraints. The decisions should be based on creating as much business value as possible.

Governing body - Business manager - Engagement manager Steered system: - Iterative development Control mechanisms: - Backlogs Information: - Development progress - Used effort Input: - Customer insight Output: - Solution

Figure 5.1 – Control model for OutSystems development Goal

Schwaber & Beedle (2001) state that an agile development method is a “management and control process that handles complexity by focusing on building software that meets business needs”. In agile development, the central goal for the development of the solution is the creation of business value within the available timeframe. COOLProfs as a software vendor does have an agreement with the customer to deliver a working piece of software which creates this business value for a fixed price.

(29)

The goals are determined in the shape of the time-box for development, stories and features which have to be developed and the agreed amount of development effort. During the analysis phase of this research it has to be confirmed that the goals are viable and a realistic time box for the envisioned system is set. Because the project scope is subject to constant change, creating accurate cost and schedule estimates for the entire project is difficult (Cao & Ramesh, 2008), however the development progress should often be checked against the set goal.

Control mechanisms

The main control mechanisms of the development process are the value prioritized backlogs (Sutherland, 2010). These are used to ensure that the most valuable items are developed first. The development team chooses the items from the project backlog which can be built within the time box of a sprint. These selected items are moved to the sprint backlog. This way they control how much development effort is planned within a sprint. When selecting the functionalities for the upcoming sprint, business value is leading and therefore functionality at the top of the project backlog should be developed first. This ensures that valuable items are constructed and is meant to develop the value for the customer. Controls on the used development effort should be enforced by the time box approach. Working in fixed time boxes with planned availability of the developers should give control on the development effort spent.

Model of the controlled system

To be able to control the development process, it needs to be clear what characteristics of the development process can be influenced. Based on the perception of the development process, a prediction can be made on how the influencing might lead to a desired change in the outcomes of the development process.

The model in this research is the iterative development phase, for which the necessary artefacts and interactions are specified in the OutSystems documentation. These elements are used to create a development process which can be controlled and is flexible enough to incorporate changes. To be able to understand how the system works, the process must be understood, as well as to how to use the artefacts and meetings as explained in chapter 4.

Information about the environment and the state of the system

(30)

When new insights and features emerge because of new insights, these changes have to be assessed on the impact on the time schedule by the delivery manager as said in the explanation of the OutSystems development method. Small changes might be developed within schedule, while for larger changes in requirements, it might have to be exchanged with other functionality. Analysis of the projects will have to show whether the information is available and correct.

Governing body and information processing capacity

The governing body has to make decisions based on the available information. However, according to the control theory, to be able to make the right decisions, they need to be able to understand the available information. This means that the business manager will need to have insight in the consequences of the choices that he makes. On the other hand, the engagement manager needs to have insight into what business value the solution is supposed to bring and combine that with a logical development order from a technical perspective to plan the sprint.

When more than one customer group is involved, with each concerned about different aspects of the system, achieving consensus or compromise in the short development cycle is challenging. Extra effort to integrate requirements through negotiations is necessary (Cao & Ramesh, 2008). Cao & Ramesh (2008) also report that customers sometimes find it difficult to understand or trust the agile process. In the context of this research this might mean that the associated business manager does not understand or trust the process enough to be able to make the right prioritizing decisions for prioritizing the user stories. Since both engagement manager and business manager have to make the decisions, these issues can ask extra effort from the engagement manager, causing limitations in the processing capacity of the governing body.

5.2 Research model

This chapter introduces the research model that will be used the answer to the question: “What are the main causes for the current discrepancy between estimated and actual effort?”. The previous chapters introduce the control theory and the five conditions for effective control and apply this to the control theory, leading to the following model for research.

Figure 5.2 - Research model

(31)

 How do invalid or indistinct goals lead to a discrepancy between estimated and actual effort? • Diagnosis of the projects have to show whether (1) there was a common picture of the

business vision, estimated effort and available time (2) if the estimates resulted in a valid goal for the development.

 How do the lack of / ineffective control mechanisms lead to a discrepancy between estimated and actual effort?

• Diagnosis of the projects will show whether (1) the backlog and time-boxes are used and sufficiently enforced and (2) there are sufficient control mechanisms to steer the project.

 How does an incorrect model of the system lead to discrepancy between estimated and actual effort?

• Diagnosis has to give insight into whether both engagement manager as well as associated business manager have a good understanding of the development process.

 How does incorrect or incomplete information lead to discrepancy between estimated and actual effort?

• Diagnosis of the projects will show whether the information about the development progress was available and correct.

 How does lack of information processing capacity lead to discrepancy between estimated and actual effort?

• The diagnosis will show whether (1) the engagement manager and business manager both understood the information available and (2) with this information were able to make correct use of the control mechanisms.

5.3 Data collection

So far, the analysis of this research has led to the identification of a number of possible causes for the problem. The diagnosis, has a purpose to discover the exact causes of the current problem. For this, the research model and its questions will be tested on a number of projects.

To select the projects for the diagnosis, the following criteria have been used: 1. The project must be based on the OutSystems development method.

2. The solution which has been delivered by the process has to be accepted by the customer and has been released in the production environment.

3. There has to be sufficient data available to do the diagnosis.

(32)

development effort measured in total number of spent hours by of all COOLProfs employees and external consultants who are directly assigned to the project by COOLProfs.

Table 5.1 - Quantitative data sources

Source Information

Back-office system Used development effort, budgeted development effort.

Project management platform Budget spending, project planning.

Documents Contracted delivery dates and project planning.

The qualitative data is done by firstly studying project related documents to get an understanding of the project. For most of the projects, documentation is available which is supplied by customers to give an insight into the initial customer vision for the project. The project contracts also give an outline for the development. This understanding of a project, combined with the quantitative data was used as a basis for interviews with the responsible engagement managers of the projects. The goal of these interviews was to identify the experienced problems during the projects and find how these relate to the research model.

Table 5.2 - Qualitative data sources

Source Information

Project management platform Project plans, documents

Contracts, presentations Project goals, customer, customer situation

(33)

6 Project diagnosis

The diagnosis of is done by analysing a number of projects. As explained in the previous chapter this has been done by studying project documents, information from administrative systems and interviews with the project managers.

The description of the projects is done by first giving a global overview of the project, the motivation for the project and its goals. Then, there will be elaborated on the problems with the control of the projects as they are found by the interviews with the responsible engagement managers. The project descriptions are concluded by giving an overview of the project results. When problems are identified and can be related to a characteristic in the research model, this is noted between parentheses.

This chapter will end with an overview of the project problems, based on the research model. This overview will show what relations of the research model are actually causes for the main problem.

6.1 Case 1: Global workflow and supply chain management system

This project was done for an international express and mail delivery service company in the Netherlands. The company was having trouble to organize the handling of incidents during transportation. Before the development for the system, a spread sheet was used to store incidents during transportation combined with the best practices for solving these incidents. This solution was difficult to keep up-to-date and not well searchable.

Table 6.1 - Project data case 1

Planned throughput time 8 Weeks Number of sprints 2

Actual throughput time 8 Weeks Weeks per sprint 2

Team Size 3

For this company, an incident registration knowledge base is created. When an incident occurs during shipping, the responsible account manager can use the web application to look into the database with prior incidents to find the best practice to solve the incident. The design of the web-application was based on the spread sheets from the prior solution and the existing business process.

6.1.1 Project results

(34)

Figure 6.1 - Project course. Global account and retention tracking system

(35)

6.2 Case 2: Process control console

The process control console is built for the department of functional management of a Dutch bank. The bank used about 20 systems that all send messages to each other. The prior solution consisted of two parts, one Access application which showed what processes should have run during the day and a CA:Gen application showing the error messages. However both of these applications were limited in their user interface and functionality. With the large amount of sent messages, the prior solutions did not give enough structure to the employees to be able to handle the errors efficiently.

Planned throughput time 36 Weeks Number of sprints 4x3

Actual throughput time 36 Weeks Weeks per sprint 2

Team Size 3

The solution was to create a web-application which shows the state of the systems, by looking at the total number of messages, where they are sent and show the efficiency of the sent messages (errors/total). The error messages are analysed and grouped when they have similar characteristics. These groups of messages can be assigned to a “call” to ensure further processing of the errors. This way the system would offer the bank a way of reacting pro-actively to errors, leading to higher quality of data processing.

The dashboard also helps in analysing chain effects. The batch processes of the bank are often depending on each other. An error in the start of the chain can have effects on the rest of the chain. By giving a comparison of the processes that should have been executed and an overview of the processes that were actually executed, chain effects are exposed and problems can be more effectively solved.

6.2.1 Project results

(36)

Figure 6.2 - Project course, main phase of process control console

During the first four phases it became clear that not all wished functionality could be developed. Also, after the final release there were some change requests because of feedback from end-users. It was agreed with the customer to plan a follow-up phase. The course of this development is shown in Figure 6.3.

Figure 6.3 - Project course, follow up phase process control console

(37)

6.3 Case 3: Customer portal

For the Dutch division of a multinational security services company, an online portal for ordering cash and scheduling cash transports has been developed. The client company attracted a large new bank as customer that was going to outsource the cash management of their customers this client company. Prior to the development of the web portal, the customers of the company were small and medium sized enterprises (SME) and orders were handled by a call centre and an existing web-portal. The existing web portal was specifically designed for the SME clients. The new client brought many large companies. To be able to handle the sudden increase of customers, the client decided to develop a new web-portal to serve the new customers. The goals of the new customer web-portal were (1) to prevent the overloading of the call centre (2) prevent and reduce mistakes that are made in the call centre.

Planned throughput time 8 Weeks Planned nr Sprints 4 Actual throughput time 11 weeks + 14 weeks follow up Weeks per sprint 2

Team Size 3

The online process for ordering the cash transports partly existed online, as a web portal for the SME customers. However this web portal did not function optimal and the processes were not always connected properly. The customer portal was built from scratch and is based on the processes as they take place in the call centre.

During the project there was intensive collaboration with the business manager and business specialists. In the beginning of the project, there was no contact with the end-users because these were not available at the start of the project. The business manager had the function as manager of information technology and innovation. He was end-responsible and had decision power on the area of requirements. The business experts in this project were business specialists; a graphical designer for customer contact, a transport specialist and the head of the call-centre.

6.3.1 Project results

(38)

Figure 6.4 - Project course, customer portal

6.4 Case 4: Online Insurance calculation

This customer in the insurance sector wanted to launch a new insurance product. For this launch, the customer wanted to build an online form that enables customers and intermediaries to make an online request for an insurance offering. To be able to do that, a new online request module was needed.

The online request process for the new insurance product was comparable with processes for already existing insurance products. The added value of the new request form is that the intelligent form only shows required fields, based on the information that has been entered on-line. The business manager for this project was the product manager, responsible for the launch of the new product.

Planned throughput time 12 Weeks Number of sprints 4

Actual throughput time 20+ Weeks Weeks per sprint 2

Team Size 3

The sizing phase was done by a sizing team in which the estimations were done by a delivery manager, who was involved in the project but was not engagement manager. Project results

As can been seen in Figure 6.5, the course of the project has been problematic. Both the throughput time and the used effort exceed the predetermined goals. The solution was supposed to be delivered in 13 weeks but has been delivered after 36 weeks, using more than three times more the budgeted development effort. Sprin t 2 Sprin t 3 Sprin t 4 Sprin t 5 Sprin t 1 Sprin t 0 Sprin t 2 Sprin t 3 Sprin t 4 Sprin t 5 Sprin t 1 Sprin t 0 0 200 400 600 800 1000 1200 1400 1600 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 E ff o rt (h o u rs )

Troughput Time (weeks)

Effort Effort budget Initial delivery date Delivery date 2 Delivery date 3

911

1136

Controlling agile development from a vendor perspective