“COST MODEL FOR AUTOMATED TESTING AND
MANUAL TESTING”
Clifford Vornis
S1541218
Master of Science Business Administration Business & ICT
Faculty of Economics and Business
COST MODEL FOR AUTOMATED TESTING AND MANUAL TESTING March 2008 Author : C. Vornis BSc Student Number : S1541218 Email : s1541218@student.rug.nl cmvornis@hotmail.com Date : 18-03-2008 Version : 1.0 Organization : Logica Project Leader : R. Tuinhout Architect : Ing. P. Blaauw
ACKNOWLEDGEMENTS
After two and half years my period as a student at the University of Groningen is reaching the end. In a period of seven months, I completed my Master thesis and thereby graduating in the Masters program in Business & ICT. This period was very intense. Not only reading many articles, conducting interviews, but also the design of a cost model. Although the amount of time invested in the thesis, it was a very interesting period for me. I had the opportunity to do my thesis at Logica, Groningen. At Logica, I was able to execute an internship under the Working Tomorrow program. The Working Tomorrow program gives students the possibility to conduct innovative researches in the field of ICT.
I would like to thank all the employees of Logica that help me during this process. I would especially like to thank René Tuinhout, Pieter Blaauw, and Jolanda Huizinga for their supervision, support, and feedback during my research.
Conducting an academic research could not be done without the staff of University of Groningen. Therefore, I would like to thank Elad Harison for providing me input and helped me to write and structure this research. Further, Professor Berghout kept me on the right track during the process with various suggestions.
Last but not least I would like to thank my family and friends, especially Josette Servage and Andre Joubert, who supported me in the graduation process.
Clifford Vornis
MANAGEMENT SUMMARY
These days companies in various industries are depending heavily on information systems (ISs). Not only does the IS support the management in their business functions, but workers on the operational level are able to execute their work effectively and efficiently. These ISs can be divided into several categories of systems such as sales order processing, stock & control systems, accountancy systems, and payroll systems.
ISs are end products of a software development life cycle (SDLC). An SDLC is a method used in project management that contains various phases that are involved to produce the ISs. Before implementing an IS, one important phase is the testing phase. In this phase the IS is checked if it is performing as expected in the deployed environment.
There are two testing approaches which a project team can choose to implement. The test team can choose between automated testing approach and manual testing approach. A combination of both can be applied by test team is. Currently there is limited information which a testing approach needs to implement. This research aims at developing a cost model, the Information System Functional Testing Cost Model (ISFTCM). The ISFTCM will give a recommendation which is the best testing approach a test team can apply for a IS project. This model bases its recommendation on the cost perspective of the approaches. This model is developed at Logica.
Logica was founded on December the 30th of 2002. Logica is an international company that provides IT and business services. This organization provides their clients with the following services: management- and ICT consultancy, system development, and system integration. They have clients from various sectors, like telecom services, financial services, energy and utilities, industry, distribution, transport and the public sector.
The main research question of this study is:
The main research question is divided into several sub questions. The research methods applied in this study are literature survey, interviews and modeling techniques. The literature survey is performed to study what ISs are and their various functions in organizations. Further, the survey is done to analyze how the ISs are developed through the SDLC. Types of SDLC that are described in this study are: structured design, rapid application development and agile development. Due to the fact that this study is focused on the IS testing, it is essential to focus on this part. Not only does this study describe a brief history regarding IS testing and how it started, but it presents also the definition of testing and the various reasons why testing is important. There are various categories of testing. These are: unit testing, system testing, acceptance testing, functional testing, white-box testing, and black-box testing. Interviews were conducted with test managers, testers and navigators at Logica and also IB-Groep. These professionals insight was necessary to give a practical view on IS testing. Both research methods were used to identify variables that have an influence on functional testing. Functional testing is chosen to be the base for developing the ISFTCM.
In this research the testing process is highlighted. The TestFrame testing methodology is used to explain the various sub phases a team needs to pass before the execution of the testing. An analysis of testing must be conducted, test cases must be designed. If the test team implements automated testing, the navigation scripts must be created. After the testing is executed it needs to be analyzed. The test team has the freedom to implement the phases of the TestFrame methodology in a different order.
Components that are part of the ISFTCM are: Total number of testers and navigators with hourly rate, number of test cases, number of test-runs, test environment costs, license and maintenance cost of testing tool, design hours for test cases, hours for creating navigation scripts for test cases, hours for execution of test cases, hours for analysis results for test cases, maintenance hours per year for each test case, budget, expected release per year.
Total cost for automated testing: TCa= TFCa + TVCa Total cost for manual testing: TCm =TFCm + TVCm
The TFCa is total fixed cost for automated testing and the TVCa is the total variable cost for
manual testing. The TFCm is the total fixed cost for manual testing and the TVCm is the total
variable cost for manual testing. To reach the break-even point the number of test-runs needs to be identified by using the formula below:
TVCa – TVCm = TFCm – TFCa
The TVCa and the TVCmconsist the number of runs and the variable cost for each
test-run. This leads to the following formula:
VCa x (Ntest-runs) – VCm x (Ntest-runs) = TFCm – TFCa ∆VC (Ntest-runs) = ∆TFC
(Ntest-runs)= ∆TFC/∆VC= X
As illustrated in the formula, the total variable costs are influenced by the number of test-runs (Ntest-runs). One important aspect of the ISFTCM is the break-even point. At this point, both
TABLE OF CONTENTS ACKNOWLEDGEMENTS ... 3 MANAGEMENT SUMMARY ... 4 1 INTRODUCTION... 9 1.1 Introduction ... 9 1.2 Logica... 13 1.2.1 Logica Testing... 14
1.2.2 Logica Testing Projects ... 17
1.3 Structure of the study ... 17
1.4 Research Setting ... 18
1.5 Conclusion... 19
2 RESEARCH DESIGN ... 20
2.1 Problem Definition ... 20
2.2 Research Question and Sub questions... 21
2.3 Conceptual Model ... 21
2.4 Research Methods ... 23
2.5 Research Scope and Constraints... 24
2.6 Conclusion... 27
3 INFORMATION SYSTEM TESTING... 29
3.1 System Development Life Cycle... 29
3.1.1 Structured design... 30
3.1.2 Rapid application development ... 32
3.1.3 Agile Development... 35
3.2 Information System Testing ... 36
3.3 The importance of IS... 38
3.4 Categories of Testing... 39
3.5 Testing in a Software Development Life Cycle ... 42
3.6 Conclusion... 43
4 TESTING APPROACH ... 45
4.1 The Executing of IS Testing... 45
4.2 Manual Testing Approach ... 49
4.2.1 Benefits... 49
4.2.2 Disadvantages... 50
4.3 Automatic Testing Approach ... 51
4.3.1 Benefits of the Automatic Test Approach... 52
4.3.2 Disadvantages of Automated Testing... 53
4.4 Combining automated testing approach with manual testing approach... 56
4.5 Conclusion... 57
5 FACTORS OF FUNCTIONAL TESTING... 58
5.1 Factors Influencing the Execution of Functional Testing ... 58
5.1.1 Total number testers and navigators and test managers with hourly rate ... 59
5.1.2 Motivation of team members ... 60
5.1.3 Number of test cases... 63
5.1.4 Number of test-runs... 64
5.1.5 Test environment cost ... 65
5.1.6 License and maintenance cost of testing tool... 65
5.1.7 Design hours for test cases ... 67
5.1.8 Creating navigation scripts for test cases ... 67
5.1.9 Execution hours for test cases ... 68
5.1.11 Maintenance hours per year for each test case ... 70
5.1.12 Total effort costs... 70
5.1.13 Budget ... 71
5.1.14 Expected release per year ... 72
5.1.15 Function points for testing... 73
5.1.16 Risk factors... 75
5.2 Conclusion... 76
6 SOFTWARE COST MODELS... 77
6.1 Introduction of Cost Model ... 77
6.2 COCOMO II... 81
6.2.1 Application-Composition model ... 82
6.2.2 Early Design Model... 83
6.2.3 The Reuse Model... 85
6.2.4 Post-Architecture Model ... 86
6.3 Putnam’s Software Life-cycle Model (SLIM) ... 87
6.4 Model of Ramler and Wolfmaier (2006)... 89
6.5 Hoffman cost model ... 91
6.6 Conclusion... 93
7 EVALUATING SOFTWARE COST MODELS... 94
7.1 COCOMO II... 96
7.2 Slim model ... 100
7.3 Model Ramler and Wolfmaier... 102
7.4 Hoffman model... 105
7.5 ISFTCM... 108
7.6 Conclusion... 110
8 INFORMATION SYSTEM FUNCTIONAL TESTING COST MODEL... 112
8.1 Break-even scenarios... 112
8.2 Information System Functional Testing Cost Model ... 115
8.2.1 Total fixed costs for automated testing ... 116
8.2.2 Variable cost for automated testing... 121
8.2.3 Fixed costs for manual testing... 124
8.2.4 Variable costs for manual testing ... 127
8.2.5 Break even automated testing and manual testing ... 130
8.3 Limitations of ISFTCM... 131
8.4 An example of ISFTCM... 132
8.5 Conclusion... 141
APPENDIX 8-1: Variables of ISFTCM... 143
APPENDIX 8-2: Information System Functional Testing Cost Model ... 144
9 CONCLUSIONS ... 145
10 RECOMMENDATIONS AND REFLECTION ... 147
10.1 Future Research... 147
10.2 Recommendations for Logica ... 148
10.3 Reflection ... 148
REFERENCES... 150
APPENDIX A: Characteristics and subcharachteristics in ISO/IEC 9126 ... 156
APPENDIX B: Function Point Estimation Worksheet ... 157
APPENDIX C: COCOMO II productivity ranges ... 158
APPENDIX D: Factors that are used for calculation of the “B”... 159
APPENDIX E: Interview questions and results ... 165
1 INTRODUCTION
1.1 Introduction
During the last few years, information systems1 (IS) have played an important role in various industries all over the world (Powell & Dent-Micallef, 1997). This conclusion is also found in Sommerville (2006). For instance (Powell & Dent-Micallef, 1997):
• McKesson, a pharmaceuticals distribution company, provided their pharmacists with computer terminals that allow them to enter orders. This system improved the customer service and increased switching costs.2
• Federal Express improved their service and reduced costs by implementing an advance of data management system. Additionally, drivers were equipped with hand-held computers. This firm was able to deliver services that were lucrative and affordable for their clients by implementing these systems.
• Merril Lynch, a financial and advisory company, improved their operations by launching the Cash Management Account (CMA) system. This information system combines the details of checking, savings, credit card, and securities accounts of each customer into a single statement. Automatically this organization has invested unused funds in the interest-bearing money market funds.
• Xerox made their master production schedule available online to their suppliers, which facilitates just-in-time deliveries. Not only does it lead to lower inventory costs, it also improves supplier relationships.
Information systems help firms to collect data and manage information efficiently and effectively. These systems also support business functions and support business processes in a holistic way (Boddy, et al., 2005). Not only do these systems support managers and staff other employees in the organization also depend heavily on these kinds of systems. For many organizations these are core systems that support the organization (Davies, 2004).
1Information system is defined as “a set of people, procedures, and resources that collects data which it
transforms and disseminates” (Boddy et al., 2005: p9).
2 Switching cost is defined as “the negative costs that a consumer incurs as a result of changing suppliers,
brands or products. Although most prevalent switching costs are monetary in nature, there are also psychological, effort- and time-based switching
In the first place organizations have sales order processing systems. This type of information system helps the organization to manage their sales effectively and efficiently. Secondly, many organizations depend on stock & control systems. Employees are able to monitor the inventory of raw materials and goods that need to be sold to customers. In the third place organizations depend on purchase order processing systems. With these ISs employees are supported when making purchase orders. Not only the staff can create purchase orders, but the stock control system can create the order automatically. Next, there is accountancy system that is another core system found in almost all firms. These accounting systems consist of three parts, namely accounts receivable, accounts payable and general ledger. The accounts receivable is the subsystem that is needed to manage the cash flow of the business. It registered the amount that customers need to pay for the goods or services received. The accounts payable is the subsystem that is important to manage the cash flow. This part of the IS registered the amount that the firm needs to pay or paid to suppliers for goods or service received. The general ledger is used to record all the transactions of the organizations such as income, expenditure and assets.
Further, Davies (2004) identifies payroll systems as core system. This IS is needed to help management or human resource employees to generate two outputs which are payment that needs to be made to each employee and the transaction details of the payment made. In this IS information such as pay rate and tax details are some of the information needed to produce the correct outputs.
to estimate the financial effects of investment. Also IS has a knowledge function. This function utilizes human knowledge to help a person to make choice. An example is the knowledge system used at The Royal Bank of Scotland. This knowledge IS evaluates bank loans requests based on lending history information. This makes it possible for employees that are less experienced to be able to evaluate the requests. The last IS function described by Boddy et al (2005) is the communication aspect. With the communication capability of ISs employees are able to communicate better and it eliminates barriers of distance and time. An example of this is the e-mail and video conferencing.
According to Boddy et al. (2005), implementing an IS may lead to a competitive advantage. Companies are able to raise the entry barriers, enter markets more easily, differentiate products and services, and create new products and services. The value of an IS continues to grow due to the fact that software suppliers are expanding their product lines to support more and more functions.
Before implementing an IS, it needs to go through a software development life cycle. A system development life cycle is “a process of understanding how an IS can support business needs, designing the system, building it, and delivering it to users (Dennis et al., 2005-p1)”. In other words, SDLC is a conceptual model applied in project management that explains the stages involved in an information system project development. It starts with a feasibility study and it ends with maintenance of completed IS. The term used in the literature of Dennis et al. (2005) is not clear enough. Therefore, this study is describing another term which similar to the SDLC, which is software engineering. In Sommerville (2006) software engineering is defined as: “An engineering discipline that is concerned with all aspects of software production from the early stages of system specification to maintaining the system after it has gone into use” (Sommerville, 2006-p7).
An important phase of the software engineering is testing. IS testing tries to assure that the delivered product is performing as expected in the deployed environment (Hailpern & Santhanam, 2002).The main objective of IS testing is to enhance the reliability of the software. Additionally, many organizations attempt to reach a higher level of product quality through testing (Burgt & Pinkster, 2003).
In order to execute the test activities, the software company needs to take into account the costs. It is estimated that testing accounts for 50 % of the total project costs (Hetzel, 1993). Further, Ramler and Wolfmaier (2006) reached the same conclusion about the percentage of testing costs related to the total project costs. In the article presented by Engel and Last (2006) they demonstrated that the costs related to testing are 60% of the development budget. In some projects, testing accounts for 75% of the total cost (Sommerville, 1996). According to Malaiya (1996), total test costs are influenced by the testing approach that is implemented. This research focuses on two testing approaches, namely manual testing approach and automatic testing approach (Leitner et al., 2007).
This study aims at developing an Information System Functional Testing Cost Model (ISFTCM). This model recommends a testing approach. The recommendation of this model is based on specific values of variables entered in the model. By applying the model, software companies are able to determine the best approach. This decision leads to a higher level of efficiency and effectiveness of the SDLC process. Software testing consists of various testing categories (Dennis et al., 2005). Each category focuses on certain aspects. This study is going to focus on functional testing only. The arguments are described in section 2.5.
Functional testing is defined by Forrester as “tests to verify that application functionality conforms to predefined specifications” (Shawber & Gilpin, 2005-p3). Functional testing is a category of testing that analyzes the requirements of the system (Bertolino, 2007). This type of testing selects the test scenarios without taking the source code structure into consideration (Whittaker, 2000). Moreover, functionality is one aspect of the ISO/IEC 91263 (Bhatti, 2005). “ISO/IEC 9126 defines a hierarchy of software quality metrics i.e. issues related to the quality assurance” (Bhatti, 2005-p1.)
1.2 Logica
Logica was established on December the 30th of 2002. This company was established from the merger between Logica plc (60%) and CMG plc (40%). January the 13th of 2006 Unilog became also part of the Logica group. Besides Unilog, WM-data later on became part of the Logica group. End of February 2008, the company has changed the name LogicaCMG to Logica. The mission statement of Logica is: “To help leading organizations worldwide achieve their business objectives through the innovative delivery of information technology and business process solutions”.4
Logica is an international company that provides IT — and business services. The company has 40 000 employees in 41 countries around the globe. Logica is listed, based on turnover, as one of the top 20 international ICT service firms. Europe and Australia are the markets that account most of the sales. With the knowledge and experience of the employees of Logica in various industries, this firm is able to focus on helping its clients to build and maintain competitive advantages. This organization provides their clients with the following services: management- and ICT consultancy, system development, and system integration. Logica has clients from various sectors, like telecom services, financial services, energy and utilities, industry, distribution, transport and the public sector. Beside these facts, Logica has its headquarters in Europe and is listed on the London Stock Exchange and Euronext. In The Netherlands Logica has several offices in the following cities: Alkmaar, Amstelveen (head office), Arnhem, Eindhoven, Groningen, Hoofddorp, Leeuwarden, Maastricht, Nieuwegein, Rijswijk and Rotterdam. Table 1.1 presents the revenue for each market sector in which Logica operates.
Revenue by market sector FY ’06 £’m FY ’05 £’m Pro forma FY ’05 £’m Actual % Growth FY ’06 on FY ’05 Pro forma % Growth FY ’06 on FY ’05 Actual Telecom Products 244.5 255.9 254.7 (4.5) Telecom Services 225.0 180.8 116.4 24.4 Public Sector 619.3 594.5 471.3 4.2 Industry, Distribution & Transport 663.9 660.8 343.2 0.5 Energy & Utilities 427.5 452.8 352.9 (5.6) Financial Services 485.0 416.0 295.6 16.6
Group 2,665.2 2,560.8 1,834.1 4.1 45.3
Table 1.1: Revenue for Logica market sector 2005-2006 (Source: LogicaCMG Annual report 2006)
Below the organizational structure of Logica in The Netherlands is shown and the Working Tomorrow program.5
Figure 1.1: Organizational chart Logica Nederland and the Working Tomorrow program (Source: Logica internal documents)
1.2.1 Logica Testing
Logica executes several testing projects in various countries, and in particular in the Netherlands, United Kingdom and Germany. This firm has 7000 testers around the world. Logica provides managed testing and managed test environment services, business acceptance management, quality assurance and domain-specific testing services across diverse market sector. This company provides testing services like, test consultancy, testing projects, test automation, specialist and test factories. This company is able to provide an integrated approach by combining their domain and industry knowledge with their testing expertise. “Logica’s testing mission is to help leading organizations worldwide manage their business
5Source: www.workingtomorrow.nl
Working Tomorrow
risks while improving predictability and quality, reducing time to market and lowering their IT costs by 30% or more. This will be achieved by optimizing the testing process to make it more effective and more efficient”.6
Logica keeps developing and innovating with structured yet flexible keyword driven methodology for testing specification and testing strategies. This methodology is based on business and project risks and it provides testing and test situations as managed services. Logica currently uses the test execution methodology TestFrame.
TestFrame is a structured test execution method developed by test experts of Logica. This methodology can be compared with the structure of a Greek temple (See Figure 1.2 on page 16 ) and can be described as follows.
Quality to market & time. Quality to market & time to market reflect the symbols of the company’s objectives and are therefore consist of the upper part of the model. Companies aspire to deliver a high quality end product onto the market in a short time period. To achieve these objectives, a firm foundation is needed.
Re-Usable test products. In the TestFrame methodology the base is Re-Usable Test products. These test products must be used more than once and they have to be simple to maintain in order to be cost effective. There are three critical success factors that influence the re-usable test products (three pillars in Figure 1.2). The three critical success factors are:
Fitting. In the preparatory phase it is important to reach the best test execution method that fits the firm and which optimizes its usability. It needs to offer a direct link with the development or maintenance processes of the IS that needs to be tested.
Structuring. This part helps a company or project team to design the various tests in a structured way; it needs to be transparent and uniform. Further, the organization can rely on the testing standards time after time.
Tooling. TestFrame makes use of various modern tools. Applying the various testing tools in the process will have various benefits. These benefits are: faster test executions, a better overview and it helps to conduct a consistent testing.
Figure 1.2: TestFrame Model (Source: TestFrame Research Centre, 2006)
1.2.2 Logica Testing Projects
Logica has conducted various projects were IS testing was part of the project.
For example, a worldwide bank organization desired to shorten the throughput time of the implementation of new systems. The bank wanted to implement various electronic-banking applications and contracted Logica to execute various tests before launching the system. Logica created a team consisting of consultants from the bank and experts from Logica. They used the TestFrame methodology to execute the various tests.
Another client of Logica is a financial organization. As part of their activities to have an efficient Operation and IT, this financial organization decided to outsource activities like application development, application management and tests services. Logica was assigned to execute this project. The agreement of this project has a value of over 200 million Euro based on a six-year period.
Finally, one of the world’s largest mobile communications company is another customer of Logica. This firm was facing with an increasing complexity of products and services that needed to be tested in a fully integrated manner in a single environment. Therefore, this company needed a partner who could help design, build and implemented the new services. The project came into service on the 1st of February 2005.
1.3 Structure of the study
In Chapter 2 the research design is presented. Various aspects such as problem definition, the main research question and sub questions are described. Besides that, the various research methods that are part of the study are presented. In the final part of the chapter attention is given to the research scope and constraints.
In Chapter 3 a literature survey is conducted. It starts with a description of the various
In Chapter 4 a description of the several activities is provided that are performed during the testing process. Besides that, the disadvantages and advantages of automated testing and manual testing approach is presented. The last part of the chapter contains discussion about combining the testing approaches in a IS testing.
Chapter 5 presents the various factors that have influence on functional testing. Several of the factors described are part of the ISFTCM. Others are not part of the ISFTCM. The decision to either applies a factor or not, is based on arguments presented.
Chapter 6 contains several models that are used to make estimations. Some of the models are used for software estimation and others are used for IS testing.
Chapter 7 provides an in-depth analysis of each models described in Chapter 6. This evaluation is done based on several criteria.
Chapter 8 presents the ISFTCM. This chapter does not only describe how the model is constructed, but also presents an example how it can be applied in an organization. Chapter 9 and 10 concludes the research and presents recommendation not only for future research, but also several suggestions for Logica.
1.4 Research Setting
The research was conducted under the following circumstances:
• The assignment started on the 3rd of September 2007 and ended on 25th of March
2008.
• This research is executed at the Logica office in Groningen, The Netherlands. • The final products are:
1. A report for Logica;
2. A graduation report for the University of Groningen;
• The supervisors of Logica for this study are Ing. P. Blaauw and R. Tuinhout. • The supervisors of the University of Groningen are Prof. Dr E.W. Berghout and
1.5 Conclusion
2 RESEARCH DESIGN
2.1 Problem Definition
According to Coopers and Schindler (2003) before stating a research question, the management dilemma must be discovered. Therefore, this study presents the management dilemma. When software companies aim at reaching lower tests costs and increasing the quality of the product, they assume that automated testing to be a silver bullet (Hoffman, 1999). On the other hand, in some projects manual tests are seen as the best approach for the information systems especially in fast-paced developments (Masciana, 2005). Thus, the following question arises:
When should a company choose to implement automatic testing or manual testing?
There are various factors that influence the testing phase (Ramler, Wolfmaier, 2006). These are growing international competition, fast paced development processes, tight schedules and budgets. These factors emphasize that cost plays an important role (Hoffman, 1999). Nowadays the cost aspect is important, because management of companies aspire to obtain return of investment in the various costs that have been made. Additionally, managers of several companies wish to see effective and efficient allocation of test costs. Further, various senior managers view IT projects not as an investments, but merely as costs. A cost model is recommended to support the decision for a testing approach for functional testing.
Various software costs models have been designed (Boehm, 2000; Hoffman, 1999; Pham, 2003; Menizies, T. Port, D. Chen, Z. Hihn, J. 2005; Zhang X. & Pham, H., 1998; Ramler & Wolfmaier, 2006). Although there are various cost models that are able to estimate the total costs of software development, they are limited to calculating which approach is the best for a functional testing of an information system update. In other words, project teams execute a testing approach based on certain assumptions they have. In addition, senior managers demand IT departments to reach more objectives with a lesser amount of resources and in a faster way (Charette, 2005).
their hourly rate, number of test cases, number of test-runs, test environment cost, license cost and maintenance of testing tool per year, design hour for the test cases, hours for creating navigation scripts, execution hours of test cases, analysis hours of test results, maintenance cost of test cases, total effort costs, budget, annual expected releases, the risk factor. These variables are discussed in Chapter 5 and presented in the ISFTCM in Chapter 8.
2.2 Research Question and Sub questions
Based on our problem definition the following main research question can be formulated:
“How can organizations determine which testing approach should be applied on the basis of cost-benefit analysis in an information system update that uses an Information System Functional Testing Cost Model (ISFTCM)?”
The main research question is divided into various sub questions:
1. What is information system testing and what it importance in an information system update?
2. Which testing approaches exist?
3. Which variables influence the functional testing costs of the automatic testing approach or the manual testing approach?
4. Which software (testing) cost models are used by companies to predict the test costs?
5. How can these software (testing) cost models contribute to the information system functional cost model (ISFTCM)?
6. Can an ISFTCM be developed, that recommends a manual or automatic testing approach for system update based on a cost-benefit analysis?
7. In which circumstances can the ISFTCM be applied and what are its limitations? 8. Can there recommendations be made that can further improve the ISFTCM?
2.3 Conceptual Model
Besides formulating a clear problem statement and sub questions, a conceptual model is also presented. A conceptual model specifies the following aspects (Cooper & Schindler, 2003):
• It shows the components of the system that are part of the study;
This research consists of two kinds of variables, namely independent variables (IV) and dependent variables (DV). Independent variables are components that are not being influenced by any factor. On the other hand, there are dependent components, which are being influenced by the independent variables.
The conceptual model of this study (See Figure 2.1) is based on the following hypothesis: The information system (IS) update is not influenced by any factor, but it influences the variables. The values of variables entered, that describe the IS, influence the recommendation of the ISFTCM.
The main objective of this thesis is to design an ISFTCM to help organizations like Logica, in deciding which testing approach is the best for a new IS version. As illustrated in Figure 2.1, the IS update is an independent variable. ISs updates influence the values of variables into the model. In other words, these variables are dependent variable. Further, the ISFTCM is an independent variable. The ISFTCM uses the values of various variables entered and recommends a testing approach. One dependent variable in the conceptual model is the cost model. The cost model results are dependent on the variables values that are entered. The last dependent component is the testing approach. Based on the costs entered in the ISFTCM, it recommends a specific testing approach. The recommended testing approach is part of the information system update. To conclude this section, this study presents all the variables in the problem statement:
“How can organizations determine which testing approach (DV) should be applied on the basis of cost-benefit analysis in an information system update (IV) that uses an Information System Functional Testing Cost Model (ISFTCM) (IV)?”
Figure 2.1: Conceptual Model
Testing approach (DV)
2.4 Research Methods
The research methods used for this study are literature survey, field research and modeling techniques. The literature survey is conducted by gathering relevant scientific journals such as: the ACM Journal, Business Source Premier, and the IEEE Journal. Not only articles from scientific journals are used, but also books. For instance System Analysis and Design with UML version 2.0 (Dennis, et al. 2005), Software Engineering 5th edition (Sommerville, 2006). Less scientific sources, such as www.investopedia.com, www.webopedia.com are also used as sources. This is necessary, because some terms are described extensively in these sources. Further, test experts are approached to give more insight about the values of variables that have effect on the testing approaches.
The literature survey consists of several stages. First, the literature survey describes the various development methodologies of information systems. Next, it is important to define the concept of information system (IS) testing. The importance for IS testing will be also argued. Moreover, various testing categories are described based on the literature survey. Further, this survey gives insight how testing is applied in the development process of an IS. The results are presented in Chapter 3.
The next stage of the literature survey covers the automatic and manual testing approach. Before presenting the automated and manual testing approach, details are given regarding the several phases of testing. In this part the various stages, technologies, procedures and tools are discussed. This chapter analyses the various strengths and weaknesses of automated testing and manual testing. Further, a discussion presented was various authors indicate that automated testing and manual testing should not be compared. A discussion of the factors that influence the decision for applying a certain testing approach is also part of this study. This discussion is presented in Chapter 4.
The next part of the literature survey focuses on the several factors that have influence on the functional testing. Using several articles and books these factors are presented. Arguments are given why it is or is not included in the research. This part of the research is presented in Chapter 5.
described using various books and scientific articles. Chapter 7 the various software cost models are analyzed.
In the next part of the study, the ISFTCM is designed by using modeling techniques. An accountancy book and article are used to build a qualitative model. Before developing the model, an in-depth analysis is conducted of variables that influence functional testing was conducted. To guarantee the correct outcomes of the model, an example is provided. Further, the limitations of the ISFTCM are discussed. The results are presented in Chapter 8.
The last part of the thesis consists of recommendations and opportunities for future research. Implementing these recommendations may further improve the ISFTCM.
To gain more knowledge in the testing world, various interviews were conducted with test managers, testers and navigators. Not only were these persons from Logica, but also the IB-Groep. The interviews were done using a list of questions (See Appendix E). Results of the interviews can be found in various parts of this study.
2.5 Research Scope and Constraints
It is necessary to define the scope and constraints in order to conduct an efficient and effective research (Cooper & Schindler, 2003). The scope indicates the area on which the research concentrates. Constraints are detailed limitations or restrictions that are influenced by the researcher. For this research the following constraints are:
• This study focuses on functional testing.
There are testing categories that can be difficult to use (Segue, 2005). For instance, to execute load testing manually, it may be complex to arrange more than 300 people to test the systems (Marick, 1998). According to (Burnstein, 2003) loading testing manually is not cost effective. The company needs to pay for additional computers needed, catering and also persons to monitor the 300 persons. With functional testing the testers are able to verify if the IS is functioning according to the specification. Moreover, functionality is one aspect of the ISO/IEC 9126 (Bhatti, 2005), as “ISO/IEC 9126 defines a hierarchy of software quality metrics i.e. issues related to the quality assurance” (Bhatti, 2005-p1.).
Finally, various problems with testing arise from unclear, imprecise, incomplete and ambiguous specifications (Tretmans & Belifante, 1999). These problems can be the categorized in several defects (Burnstein, 2003).
Requirements and Specification Defects. These defects are mainly due to incorrect specifications. The complete description of the tasks that the product does and how it should perform is not correct. Additionally, features that are not properly identified or described in the functional requirements are not complete or are missing. Furthermore, the interaction between the features is not specified or is missing. For instance, a feature to add customers is not interacting correctly with the feature to categorize the type of customer. The last aspect that is part of this type of problem is the interface description. The problem that testing encounters occurs due to missing descriptions of how the IS is going to interact with external aspects such as software, hardware, and users.
Design defects. These types of defects are the consequence of system components and their interactions, external software and hardware that are not properly designed. Two aspects of the design defects are the algorithmic and processing defects. These defects are due to the pseudo code7 that the processing steps are not properly described. Further the data may be incorrectly structured, incorrect parameters, missing or unclear design elements, incorrect design description of interfaces where some commands or messages are missing.
7Pseudo code is defined as “An outline of a program, written in a form that can easily be converted into real
Coding defects. These defects are due to errors that are found when the IS is implemented. These types of defects are almost the same as the design effects. One of the most common consequences of these defects is the knowledge of the software engineering regarding how the programming language works.
Testing defects. These types of defects are not related to the code. These types of defect are found in the test plans, test cases, and test procedures.
• The ISFTCM can be used in a new version or an update of an IS.
There are some cost models, which are presented in Chapter 6 that may indicate if an automatic testing approach or manual testing approach is the best option (Hoffman, 1999; Ramler & Wolfmaier, 2006). The Hoffman model (1999) based on the entered values for the variables of the model, results in a return of investment value. This value indicated that either automated testing or manual testing is the best option to implement. Besides that, the result of the ROI can illustrate that both approaches is favorable. Variables that are used to reach the number are expenditure for test specification and implementation, expenditure for test specification, expenditure for test interpretation after automated testing, expenditure for single manual test execution, number of automated (and manual) test executions, number of automated only test executions and number of manual test executions.
• Only specific variables that are part of the functional testing will be part of the ISFTCM.
There are several activities that need to be executed in the testing phase. All these activities may cost time and money. This research is going to concentrate on variables like testing costs and test-runs. These factors are discussed in Chapter 5.
2.6 Conclusion
Nowadays when ISs need to be tested, an important question arises: When should a company choose for automatic testing and when for manual testing? One aspect that is plays an important role is the cost aspect. Management of several companies aspires to obtain return on investment in the various costs that have been made. A cost model is recommended to support the decision for a testing approach for functional testing. There are various cost models developed that can support project teams to make the right decision. Moreover, senior managers demand IT departments to reach more objective with a lesser amount of resources and in a faster way.
For this research the following research question is formulated:
“How can organizations determine which functional testing approach should be applied on the basis of cost-benefit analysis in an information system update that uses an Information System Functional Testing Cost Model (ISFTCM)?”
The main research question is divided into several sub questions. Further a conceptual model is presented that consists of independent and dependent variables. The dependent variables are: the variables of functional testing and the testing approach. On the other hand the independent variables are: ISFTCM and the IS update.
Another important research method is the model techniques that are used to design the cost model. The final part of the study consists of a literature study to bring forward recommendations for future research.
3 INFORMATION SYSTEM TESTING
3.1 System Development Life Cycle
3.1.1 Structured design
Structured design methodologies deals with a formal step-by-step method to the SDLC that advances from one phase to the next. Waterfall development (See Figure 3.1) and parallel development (See Figure 3.2) are examples of structured design methodologies.
Waterfall development. Waterfall development typically consists of a several phases, including planning, analysis, design, implementation, delivery, that needs to be followed in a sequential order (Avison & Fitzgerald, 2003). As showed in figure 3.1, one phase needs to be completed before the next one can begin. Each stage must be seen as individual process (Sommerville, 2006). The deliverables for each phase are presented to the project sponsor8, which gives the approval to let the project move to the next phase (Dennis et al., 2005). This approach has two important advantages. First, it analyzes and identifies the requirements early before the programming starts. Second, by analyzing and identifying the variables properly, it reduces the changes that requirements are changed during the process of developments.
Figure 3.1: Waterfall Development-based Methodology (Source: Dennis et al, 2005)
On the other hand there are also two disadvantages. In the first place the design must be finished and specified before the programming can start. In other words without the design, the projects is not able to continue with the next stages. According to Avison & Fitzgerald (2003) the waterfall method is not flexible. Another disadvantage is that the time frame that the development has to deal with compared with methodologies such as rapid application development methodologies (Dennis et al., 2005). The elapse time between the completion of the IS in the analysis phase and the implementation phase of it takes could be long.
8 Project sponsor is the person/organization that initiates the project and who serves as the primary point of
It increases the chances for rework, because of changes in the business domain. These changes were not discussed when the analysis phase was executed. Avison & Fitzgerald (2003) came to the same conclusion and describe it the waterfall method as unstable.
Parallel Development. Parallel Development is almost the same as the waterfall development methodology (See Figure 3.2). This methodology tries to solve the problem of delays that occur between the analysis phase and the delivery of the information system (Dennis et al., 2005). Parallel development does not apply to the implementation stage after finishing the design. Project teams applying this methodology perform a general design for the IS and split the project into several subprojects that are designed and implemented in parallel. When all the subprojects are ready and completed, they are integrated and the IS can be delivered.
Figure 3.2: Parallel Development-based Methodology (Source: Dennis et al, 2005)
3.1.2 Rapid application development
Rapid application development (RAD) evolved from fourth-generation languages (Sommerville, 2006). RAD methodologies are methodologies that modify the SDLC phases to develop certain parts of the IS rapidly, which afterwards are presented to the users afterwards (Dennis et al., 2005). In this way, the users can understand the IS better and suggest revisions to improve it. Tools that are part of the RAD are: database programming language, interface generator, links to office applications, and report generator (Sommerville, 2006). Phased development (See Figure 3.3), prototyping (See Figure 3.4) and throwaway prototyping (See Figure 3.5) are examples of rapid application development methodologies.
Phased Development. This subcategory of the rapid application development divides the IS that needs to be develop into versions that are developed sequentially. As illustrated in figure 3.3, after the planning phase is finished, the project continues with the analysis phase. The objective of the analysis phase is to identify and categorize the overall system concept, the project team, users, and system sponsor and categorize them into a series of versions. Requirements that are crucial and fundamental are putted together in the first version of the IS. The design and the implementation stages continue and deal only with the requirements defined in the first version. After implementing version 1, the execution of version two starts. In other words, the same phases that were applied for version 1 is started to implement version two. Thus in version two, the IS needs to go through the phases analysis, design and implementation. Although these phases are the same, the analysis phase in version two is formulated based on requirements, new ideas and issues that rise from the experience of the users with version 1.
Figure 3.3: Phased-based Methodology (Source: Dennis et al, 2005)
Prototyping. In a prototyping-based methodology the analysis phase, design phase and implementation phase are executed at the same time. These phases are repeated till the IS development is completed. The analysis and design phase are quickly executed to work on a system immediately with minimal features. Many times the first prototype consists of features that are important for the users. After the first prototype is launched, the project team is going to concentrate on the next prototype. The second prototype is the results of the provided comments project sponsor, which are utilize to analyze, design and implement the system again. The process goes on till the analysts, project sponsor, and users agreed that the prototype has all the necessary features. The next step will be the installation of the system.
The main advantage of the prototyping is to provide a working system very fast which the users can use. Using the prototyping methodology it guarantees the users that the project team is working on the system. In addition, having a prototyping it supports the users and the project team to quickly refine the requirements. On the other hand, a disadvantage of the prototyping methodology is that can be difficult to conduct an in depth analysis in fast-paced systems releases. By not doing the right analysis, a prototype may become useless later on. This is the result of significant changes of the requirements. The fundamental issues are not discovered and needs to be included in the next prototype.
Throwaway Prototyping. This subcategory rapid application development is almost similar to the prototyping. Although both strategies produce prototypes, they are used for different purposes and appeared different. The analysis phase of the throwaway prototyping is done more carefully compared with the prototyping approach (See Figure 3.5). In the throwaway prototyping a design prototype is used. A design prototype is a system that is not functional, it only represent a part of the system that needs refinement.
An advantage of the throwaway prototyping is that is it balances the development of a carefully analysis and a using the advantages of prototyping. A disadvantage of the throwaway prototyping is that it takes longer to finish the system compared with the prototype-based methodologies.
3.1.3 Agile Development
Agile development focuses on streamlining the SDLC through eliminating most of the modeling and documentation overhead, and the time spent on those tasks. This development was needed due to the changing environment of the firms (Sommerville, 2006). An IS needs to be developed in a short time frame to be implemented to take advantage of the new opportunities and to the necessary actions (Dennis et al., 2005). Due to the fact that the sometimes these methods consisted of large overhead, new agile methods were introduced (Sommerville, 2006). The new agile methods concentrate more on the software than the design and documentation. The objective of the software is to provide the software very fast to the end-user. By quickly providing a system, the user is able to address the changes and new requirements that need to be part in future iterations. These methodologies have a small number of rules and practices which are easy to follow. An example is Extreme Programming.
Figure 3.6: Extreme Programming Methodology (Source: Dennis et al., 2005)
3.2 Information System Testing
An important sub stage in the implementation phase of the system development life cycle is testing. Testing started in 1950 (Hetzel, 1993). It commenced almost simultaneously with the first practice of writing programs for machines. Testing was an activity that used to be associated with engineering and manufacturing processes. In the 1950s, people described testing as “debugging”9, which refers to correcting and removing errors. The launch of the
personal computer in the past led to a tremendous increase in commercial software development (Dustin et al., 1999). It became clear that these information systems (IS) needed testing. An application test was considered successful if there was no error found in the program (Goodenough, Gerhart, 1975). The process of finding errors was considered as testing the execution of the program or system (Hetzel, 1993).
Although these days authors like Myers (2004) define testing as a practice which is executed in a project with a focus on finding errors, others approach and define testing from a different point of view. Testing is defined by Hutchison (2003-p388) as “the process of analyzing a software item to detect the differences between existing and required conditions (that is, bugs) and to evaluate the features of the software item”. In book published by Burnstein (2003-p27) describe testing as “the process of exercising a software component using a selected set of test cases, with the intent of revealing defects, and evaluating quality”.
9 The process of debugging involves analyzing and possibly extending the given program that does not meet the
According to Dennis et al. (2005) it is not feasible to detect every error in the software looking from a cost-effective point of view. Subsequently, these days IS testing is defined as the process10 of executing activities to determine whether the IS matches its specification and can run in its intended environment as expected (Whittaker, 2000). In other words, it tries to assure that the delivered product is performing as expected in the deployed environment (Hailpern & Santhanam, 2002). Tretmans and Belinfante (1999-p3) defines IS testing along the same lines: “An operational way to check the correctness of a system implementation by means of experimenting with it”. Experimenting with the IS includes checking the main functions are working properly (Tretmans & Belifante, 1999). The main objective of IS testing is to enhance the reliability of the software. But Sommerville (2006) mentioned two main goals of IS testing. In the first place the objective is to demonstrate to the software engineering and the customer that the IS meets the requirements described. The other objective is to search and identify errors or defects in the IS where it does not work properly or not executing the tasks conform the specification.
IS testing activities include design reviews, code inspections and static analyses of the source code (Hailpern, Santhanam, 2002). Even though testing and verification may be seen as a single activity, they are complementary approaches to analyze and check the correctness of ISs (Tretmans and Belinfante, 1999). The aim of verification is to test the IS properties by formal manipulation on a mathematical model of the system. Additionally, testing is exercised under real execution implementation.
System testing is more critical for IS developed nowadays than for IS implemented in the past, because of various factors like polymorphism11 and dynamic binding12, inheritance13 and reuse (Dennis, et al. 2005). The same is concluded by other authors too. Dustin et al 1999 argued that, instead of testing single closed applications that are operating on a single system, these days testers are challenged with IS that are based on a client-server architecture. This type of architecture is based on three separate parts; server, client and the network. Testers need to take into consideration with three components that may contain errors.
10 Process is the flow of information, control or materials from one activity to another (Harmon, 2003) 11 Polymorphism means that the same message can be interpreted differently by different classes of objects.
(Dennis et al. 2005-p28)
12 Dynamic or late binding is a technique that delays typing the object until run-time. Polymorphism is made
possible through dynamic binding. (Dennis et al 2005-p28)
3.3 The importance of IS
Since the late 1950s and 1960s professionals began to be more aware of testing because of their experiences with testing and its economic impact (Hetzel, 1993). This awareness increased due to many deficiencies. The costs involved and the influence of fixing or recovering from these problems were considerable. Nowadays, ISs need to function under higher requirement levels (TestFrame Research Centre, 2006). Although in the past few years there were many technological advances in programming are achieved through the use of patterns, frameworks, class libraries and components, IS testing must be handled systematically and the outcome must be properly documented (Gelperin & Hetzel, 1988). According to Gelperin Hetzel (1988) testing activities need to strongly interact with development activities from the beginning of the IS project. They needs to be coordinated in the early stages, because testing may not only impact the IS implementation, but could also influence the requirements and the design.
Testing will continue to have major responsibilities, because of the increased: complexity of products, shortened development cycles and higher customer expectations (Hailpern & Santhanam, 2002). In a research conducted by Standish Group International (1994) 50% of IT executives said that there are fewer or the same number of failures today compared to five and ten years. However testing remains an important part of the project. If IS testing is not executed carefully, it may lead to serious consequences such as costly customer complaints, loss of goodwill (Pham, 2003), lawsuits, non-lucrative deals, and lower sales (NIST, 2002). For instance, FoxMeyer Drug Co. poorly implemented a resource planning system (Charette, 2005). This implemented system failed because of many system errors. These errors had an enormous impact that led to the bankruptcy of this wholesale drug distribution company in Carrollton, Texas. Moreover, in April 2003 a large student loan company in the Unites States made incorrect calculations of 800,000 loans due to a software error.14 Although the fault was corrected, they reported a loss of 8 million dollar in interest.
In addition, table 3.1 illustrates that the aerospace industry has lost more than a billion dollars that might be related to problematic applications. For example, the Ariane 5 Galileo rocket that exploded 40 seconds after the launch was due to a software design error and insufficient software testing of the flight control system.
According to previous study in the Unites States costly customer complaints that are related to IS errors are estimated between 60 billion to 70 billion dollars (Charette, 2005).
For example, Denver International Airport attempted to implement an automated baggage-handling system. Not only an unrealistic time schedule was used, but also the testing of it was minimal. According to Gibbs (1994) at one point the delay was costing the city of Denver over $1million a day in interest and operating costs. Nevertheless, the system was introduced. The result of this poor implementation was that the system chewed up baggage and the carts that used to transfer the luggage around regularly derailed. United Airlines sued the system contractor for these costly mistakes.
Table 3.1: Losses due to software failures. Source: NASA IV&V Center Fairmount, West Virginia, 2000
Information systems with errors can even lead lost human lives (Hailpern & Santhanam, 2002). On March the 31st of 1986 a Boeing 727 of Mexicana Airlines crashed into the mountains in Maravatio, Mexico. The software on board failed to calculate the mountain position correctly. Similar scenario occurred in U.S cities Georgia, Massachusetts and Texas in the period from March 1986 until June 1986. In this space of time the massive Therac-25 radiation therapy machines overdosed various cancer patients, because the computer program was not performing as expected. After thorough inspection, it was concluded that the program was defect.
3.4 Categories of Testing
Unit testing is defined by Burnstein (2003-p670) as “Aggregate of technical activities
involved in demonstrating that an individual software unit has been implemented correctly, that the code and the design of a unit are consistent, and that the unit design is correct”. Whittaker (2000-p77) defined is as “tests individual software components or a collection of components. Unit testing focuses on a single component of the IS (Dennis et al., 2005). Thus, unit testing examines a piece of code (Bertolino, 2007). Subsequently, in Sommerville (2006) it is called component testing. Testers determine the input domain. This type of testing ignores the other parts of an IS. The goal of the unit testing is to discover the error in each component separately. The same goal of unit testing is described in Burnstein (2003). According to Bertolino (2007), by examining individual isolated units of the system, the errors in the system can be detected earlier if they are tested as a complete system.
Integration testing analyses a group of classes that need to execute various tasks together
(Dennis et al., 2005). It verifies whether the interface and linkages between the several components are working correctly. For instance, the flow of control between the classes and the data exchange. Not only components, but also modules, programs, application, or system that programmers put together to produce the final IS (NIST, 2002). This type of testing may be difficult to execute because a class can be in many dissimilar aggregations 15 (Dennis et al., 2005).
System testing is defined by Whittaker (2000-p77) as “testing a collection of components that
constitutes a deliverable product”. System testing is executed to ensure that all classes work together without any errors (Dennis et al., 2005). This testing deal with integrated components that are needed to execute system functions (Sommerville, 2006). Thus the testing is executed to check if the components are working properly. The objective of the system testing is to examine the functional behavior and quality requirements (Burnstein, 2003). Quality requirements are reliability (ISO/IEC 9126), usability (ISO/IEC 9126), performance and security. This category of testing concentrates on the external software and hardware interface. This test is executed by a system analyst (Dennis et al, 2005). Although system testing is similar to integration testing, it is broader in scope. Integration testing focuses on whether the classes are working together without errors while system testing checks whether the system meets the business requirements, whether it is suitable for use, is secure, performs
15 Aggregation illustrates the is part or is subpart of a relationship set. For example: the aggregate of a car could
under a heavy workload and has exact documentation. Integration testing verifies if the correct data is transferred at right time over the interfaces (Sommerville, 2007). Further, Myers (2004) has criticized authors that only label system testing as testing the whole system. System testing examines other types of errors. These errors may occur when the translation of the objectives is made to the external specification.
Acceptance testing is executed mainly by the end-users with assistance from the project teams (Dennis et al., 2005). The objective is to verify that the information system is complete, that it meets the business needs and that the end-users agree with it. There are two types of activities that are executed in this test. Subsequently, users test the system with fictional data or the users test the system with real data.
Functional testing, as described in section 1.1, are tests that are executed using test scenarios without taking the source code structure into consideration (Whitaker, 2000; Canna, 2001). These tests are designed from the user’s perspective. Functional testing can be considered as a quality assurance process to examine the IS if it is working correctly (Segue, 2005). It is necessary to execute this test, because it confirms what users expect of the performance of the system to perform. Although functional testing seems to execute the same tests as unit testing, they are executed from a programmer’s perspective. System testing and System integration testing are sub testing categories of functional testing. In other words, with functional testing the software behavior is compared and evaluated against the business functionality16 (Everett & McLeod, 2007). According to Whittaker (2000) functional testing is also named specification-based testing, behavioral testing and black-box testing.
White-box testing tests whether the source code contains any errors. It looks inside a class to review the code (Dennis et al., 2005). White-box testing gives the test team the opportunity to test the internal structure of the IS (Myers, 2004). The same is described in Burnstein (2003). To execute white-box testing the testers need to have knowledge about the structure of the IS. Black-box testing examines whether the class meets with the requirements stated in the specifications (Dennis et al., 2005). According to Tretmans and Belifante (1999) only the outside of the system is tested by the tester. Hailpern and Santhanam (2002) have the same point of view. Instead of understanding the codes used, black box testing analyze the various external specification. In various literatures they describe black box testing is the same as functional testing (Whittaker, 2000; Segue, 2005).
The conclusions can be correct, due to the fact that the system is tested to ensure that the IS is behaving according to its specification looking from a behavioral or functional perspective (Schroeder & Korel, 2000).
Regression testing is a type of testing that is repeated each software version till it is determined that the IS is ready to be implemented (Whittaker, 2000). Myers (2004-p18) defines regression testing as “saving test cases and running them again after changes to other components of the program”. Further, Everett and McLeod (2007-p21) define regression testing as “The technique of reusing test scripts on a subsequent version”. The objective of the regression testing is to check if the modified IS is still behaving correctly and that the changes did not impact the system negatively (Rothermel & Harrold, 1997). Thus the regression testing make sure that the functionality is not affected (Burnstein, 2003). According to Bertolino (2007) these type of testing could have high costs. Not only Bertolino, but also Kim et al. (2005) have the same point of view. For this reason, many project teams decide to automate the test cases to reduce the costs of execution (Bertolino, 2007). Further, Elbaum et al. (2003) has the same point of view that this type of testing is expensive. However, they purpose another strategy to reduce the costs. Not only by automating these, but also prioritize each test case. The most important test cases are executed first.
3.5 Testing in a Software Development Life Cycle
The first phase starts upper left point of the V and last phase is at the upper right end. In this phase the business requirements are defined. The second phase deals with the system specification, which means that the design is executed. The third phase is the Technical Design & Code phase which, deals with programming and building the IS. On the right hand side, it shows the various stages of testing. After finishing the Technical Design & Code phase, the process continues with the component testing and the component integration testing. These two testing phases are part of the unit testing. The other parts, system testing and system integration testing are part of the functional testing. As illustrated in figure 3.7, the first testing stages are executed based on white-box testing. As the testing progress project teams are focused on black box testing.
Figure 3.7: Vee Model (Source: Logica internal documents)
3.6 Conclusion
4 TESTING APPROACH
4.1 The Executing of IS Testing
As described in section 3.4, there are various categories of testing that are executed in various stages of the development of an information system (IS). On the other hand, there are four basic steps that are crucial to guarantee a successful testing phase (TestFrame Research Center, 2006). Figure 4.1 shows the various steps. According to the authors of TestFrame Research Centre (2006), the TestFrame method is not executed through a unique plan. As illustrated with the arrows, steps can be achieved in various sequences. In other words, a test team can decide to start immediately with the analysis phase, then the navigation phase, and finally the execution phase. The authors of TestFrame recommend to execute the several phases in the following order: preparation phase, analysis phase, navigation, and execution.
