Testing the pension administration system IRIS with formal test methods

Testing the pension administration system IRIS with

formal test methods

Testing the actuarial functions in the pension administration system

IRIS with Clean and GAST at the Test Service Center of

Achmea, Centraal Beheer and Avero

MSc Thesis Business & ICT

Tjeerd Bosma

University of Groningen

Faculty of Management and Organization

Researcher:

Tjeerd Bosma S1503960

t.bosma.3@student.rug.nl

Faculty of Management and Organization University of Groningen

Supervisors:

Prof. dr. ir. J.L. Simons

Faculty of Management and Organization University of Groningen

Dr. T.W. de Boer

Faculty of Management and Organization University of Groningen

Dhr. B. van der Zee

Manager Test Service Center Achmea

Dhr. J. van Rooyen

Preface 3

PREFACE

In order to graduate for my Study ‘Business Administration’ for the Masters program in ‘Business & ICT’ at the University of Groningen, I started my graduation assignment in March 2006. I was provided the possibility to do a graduation assignment at Achmea, Centraal Beheer and Avero in Leeuwarden. Special thanks go out to Dick Egberts of the Totality Groep, because he arranged the first contact between Achmea and me that led to this assignment. I am very grateful for his effort.

The assignment took place at the Test Service Center (TSC) within Achmea. I was supervised and supported by the manager of this department, Bert van der Zee, and the principal

consultant of testing of LogicaCMG, Jos van Rooyen. I would like to take this opportunity to thank Bert, Jos, and the employees of the TSC department for their time and support during the graduation process.

For this research a formal test method had been selected. Without the advice and support of the ‘Institute for Computing and Information Sciences (ICIS)’ it would have been impossible for me to make the right choice. I would especially like to thank dr. ir. Jan Tretmans, dr. Pieter W.M. Koopman and drs. John H.G. van Groningen of the ICIS for their advice and support during the research. They realized the connection with the module ‘Actuarieel Formularium’ of the pension administration system IRIS and helped me with making the models in Clean.

Prof. dr. ir. John L. Simons and dr. Thomas W. de Boer provided the input of the University of Groningen for this assignment. I would like to thank John and Thomas for their effort and time they put into this assignment. Their feedback helped me by making a structured and scientific thesis.

I would also like to thank my fellow students (Martijn Huitema and Mark Walstra), my family, and friends who supported me during the assignment from start to finish and throughout the other student years.

Tjeerd Bosma

MANAGEMENT SUMMARY

During the life cycle development of software projects, most faults are originated in

requirements and design phase. The current practice is that most of the test effort is put in the system and/or acceptance phase, which is not really efficient. It is relatively expensive to find and solve the problems in these test phases. To increase the efficiency of testing, the process of exercising software to detect which specified requirements do not satisfy and to detect other errors with the fewest possible tests, the effort of testing must switch to the source of the life cycle: requirements and design phase. Current techniques as inspections, reviews etcetera are strong techniques, but not sufficient enough, because with these techniques it is not possible to simulate calculations or procedures, which is necessary to test a system earlier in the life cycle.

Research has been performed for years to control the costs and raise the quality of the

requirements. From the technical engineering’s view the so-called formal methods are applied to technical software projects under defined conditions. The results of using formal methods in technical software projects are successful. Therefore a case study is started by hand of the pension administration system IRIS, together with Achmea, the university of Groningen and LogicaCMG, to investigate the applicability of formal methods in testing requirements and designs for administrative systems to solve the problem as stated before.

The purpose of the case study was to examine if formal methods can be used to test an administrative system earlier in the life cycle, thus more efficient, than the traditional way of testing. Some questions that have been answered during the research were:

- Can formal test methods be used to test an administrative system?

- Can the formal test method be used for generating test cases automatically?

- In which situations can the formal test method be used?

After literature survey and some conversations with the ‘Institute for Computing and

Management Summary 5 which is an ideal method for testing systems without states. Because the research is pointed at testing the actuarial functions in the system and does not consist of any states, the formal test method Clean is very suitable, in combination with the GAST tool, for testing the module ‘Actuarieel Formularium’ of the pension administration system IRIS.

Figure MS-1: Working of Clean and GAST

The results of testing the actuarial functions with the formal test method Clean and GAST can be divided into two categories:

- Results found during specification analysis: During the analysis of the specification

document, needed for making the models in Clean, four specification faults are found. The first three faults are superfluously input parameters, because they are dependent and based on other input variables. The fourth and last specification fault is the specification on which one of the output parameter must be calculated.

- Results found after testing: Testing the actuarial functions with GAST, based on the

models made in Clean, result in five test results that leads to differences between the values generated by the ‘Actuarieel Formularium’ and the values generated by GAST. Two of these faults are small differences, mostly caused by wrong rounding of the calculations in between. The other three test results are caused by using a wrong

1. Making models based on specification document (input,

functions, and output)

2. Present input parameters to the Actuarieel Formularium

3. Actuarieel Formularium presents output values

4. Compare results with each other and document the test results

Management Summary 7 function. The function described in the specification document is different than the one used in the ‘Actuarieel Formularium’. In the traditional situation the complete process, changing and testing the new actuarial function, will take about sixteen hours. The advantage of the formal test method Clean is that the models can be reused when the functions have been changed. Retesting a separate function with Clean will take only a few minutes. This means that the complete process of changing and testing the new actuarial function will take eight hours instead of sixteen hours.

The financial impact on the business will be determined by the actuaries. Because the

actuaries were still busy to determine the impact on the business on the moment of writing the thesis, the impact cannot be defined in the thesis.

To increase the efficiency of testing, the effort of testing must switch to requirements and/or design phase. This can be realized when formal test methods will be used together with the designers of the system, so faults will be found earlier, through which fewer tests are needed. To realize this situation it is very important for the different departments to work closely together. Research is needed to select the best formal method that can be used for developing and testing the actuarial functions. This must be done because this research only proved that the formal test method Clean can be used for testing administrative systems in a formal way. The models that have been made are based on the specification document and it can therefore be assumed that formal test methods can be used during the design phase. To actual integrate a formal method into the organization in a way that formal methods can be used for testing an administrative system in the design phase, the following questions must be answered:

- What is the impact of the test results of this research on the business?

- Adds formal methods value to the organization, based on the impact of the results on

the business?

- Which formal methods are suitable for which projects within the organization of

Achmea?

- Is there a connection possible between the formal method and the AS400 or other

applications?

TABLE OF CONTENTS

1.1 Achmea and the Test Service Center department ... 11

1.2 The pension administration system IRIS ... 12

1.3 Problem statement ... 12

1.4 Structure of the thesis ... 14

1.5 Research setting and the methods used ... 15

Part 1: Testing in General

2.1 Introduction in testing ... 18

2.2 Life cycle development model ... 19

2.3 Formal test methods ... 20

2.4 Conclusion... 21

Part 2: The Pension Administration System IRIS

3.1 System architecture ... 24

3.2 Actuarieel Formularium ... 25

3.3 Data model ... 25

3.4 Programming languages ... 26

3.5 Conclusion... 26

Chapter 4: Traditional Way of Testing... 27

4.1 Changing an actuarial function ... 27

4.2 Test techniques ... 27

Table of Contents 9

4.4 Conclusion... 29

Part 3: Formal Test Methods

5.1 Classification of formal test methods... 31

5.2 Selection formal test methods ... 32

5.3 Conclusion... 34

Chapter 6: The Formal Models... 35

6.1 Description of the selected part in the Actuarieel Formularium ... 35

6.2 Description of the models used to test the Actuarieel Formularium... 40

6.3 Working of GAST ... 42

6.4 Regression testing with GAST... 44

6.5 Conclusion... 45

Part 4: Test Results

7.1 Test results... 47

7.2 Analysis test results ... 50

7.3 Experiences ... 51

7.4 Conclusion... 53

Part 5: Advice on using Formal Test Methods

8.1 The position of the formal test method ... 56

8.2 The possible implementation areas of the formal test method... 57

8.3 The level of usability of the formal test method ... 58

8.4 The level of education the employees need ... 59

8.5 Conclusion... 59

Chapter 9: Conclusion and Recommendations ... 61

9.1 Answering the main question ... 61

9.2 Recommendations ... 65

LIST OF FIGURES

Figure MS-1: Working of Clean and GAST... 6

Figure 1-1: Structure Thesis... 15

Figure 2-1: The Life Cycle Development Model... 19

Figure 3-1: Deelaanspraak and the ‘Actuarieel Formularium’ ... 25

Figure 6-1: Vormcode 100 – Pension of Old Age ... 36

Figure 6-2: Function 100-1 before Interpolation ... 40

Figure 6-3: Model for NK ... 40

Figure 6-4: Input Parameter Lft1eVerz... 41

Figure 6-5: Testing the Function with GAST ... 42

Figure 6-6: Working of GAST ... 43

Chapter 1: Introduction 11

CHAPTER 1: INTRODUCTION

Testing software is becoming a real profession since a couple of years. “Testing came to maturity in many organisations under the influence of large projects, such as the millennium changeover and the preparations for introducing the euro” (Pinkster et al. 2005). The

evolution of testing starts in the 1970s with intuitive testing. Since a couple of years detailed test approaches like TestFrame and TMap are used for testing software. The latest research in testing is automatic testing and finding ways to detect faults earlier in the life cycle

development than with the current test approaches.

The thesis contains the research on using formal test methods on the pension administration system IRIS within Achmea, Centraal Beheer and Avero for the graduation project of Tjeerd Bosma. The research aims to examine if formal test methods can be used to test the IRIS system earlier in the life cycle to realize more efficient tests and less issues during the production phase.

This introduction chapter will first start with a general description of Achmea and the Test Service Center (TSC) department. The second paragraph will be used to introduce the pension administration system IRIS, which is the scope of this research. The third paragraph describes the problem statement with the sub-questions. The fourth paragraph of this chapter will be used to describe the structure of the thesis and the final paragraph describes the research setting and the methods used during this research.

1.1 Achmea and the Test Service Center department

The company Achmea is one of the biggest financial service companies in the Benelux. Achmea offer companies, institutions, and consumers a broad package of insurances and banking- and mortgage products and services. The activities of Achmea can be divided into several different business units. See Appendix A for the organization chart of Achmea. The business units are:

- Achmea Pensioenen (Pensions)

- Achmea Bedrijven (Companies)

- Achmea Intermediair (Intermediary)

- Achmea Bank

- Achmea Corporate Accounts

- Achmea Sociale Zekerheid (Social Security)

- Achmea Active (IT)

- Remaining

This research took place within the Test Service Center (TSC) department, which is part of Achmea System Development & Maintenance Center (SDMC). See Appendix C for the organization chart of Achmea SDMC. Achmea SDMC is part of the business unit Achmea Active. See Appendix B for the organization chart of Achmea Active.

1.2 The pension administration system IRIS

The pension administration system IRIS consists of several modules. The module that will be used in this research is the ‘Actuarieel Formularium’. This module calculates a factor, on behalf of premiums and reservations, based on several parameters. At this moment the functionality of the system is tested by the TSC department when changes take place in the system, with exception of the actuarial functions. The actuarial functions are tested by the users (actuaries) themselves. The IRIS system and the traditional way of testing the system will be described in more detail in Chapter 3 and 4.

1.3 Problem statement

Chapter 1: Introduction 13 A formal test method will be selected and used to test the IRIS system in this research. The purpose of this research can be defined as follows:

To examine if formal test methods can be used to test the module ‘Actuarieel Formularium’ of the pension administration system IRIS earlier in the life cycle to realize more efficient tests.

The results of this research will be presented in the thesis. From the description above, the following research question can be defined:

Can formal test methods be used, by the Test Service Center of Achmea, Centraal Beheer and Avero, to test the module ‘Actuarieel Formularium’ of the pension administration system IRIS earlier in the life cycle to realize more efficient tests?

The research question can be divided in several sub-questions:

Testing in general

1) What is (efficient) testing?

2) What is the life cycle development model?

3) What are formal test methods?

The pension administration system IRIS

4) What is the pension administration system IRIS?

5) What is the traditional way of testing the ‘Actuarieel Formularium’?

Formal test methods

6) Which types of formal test methods are available?

7) Which formal test method can be used to test the ‘Actuarieel Formularium’?

8) What will be the models, made with the selected formal test method, to test the

‘Actuarieel Formularium’?

Test results

9) What are the results of using the selected formal test method on the ‘Actuarieel

Advice on using formal test methods

10)What will be the position (in the life cycle development model) of the formal test

method?

11)What are the possible implementation areas in which the formal test method can be

used?

12)What is the level of usability of the formal test method?

13)What must be the level of education the employees must have to use the formal test

method?

1.4 Structure of the thesis

The thesis is divided into five parts. Each part will give an answer on a couple of sub-questions described in the previous paragraph. The first part, ‘Testing in General’, will present a general description of testing, including the life cycle development model and formal test methods (Chapter 2). The second part of the research, ‘The Pension

Administration System IRIS’, will be used to describe the IRIS system (Chapter 3), which will be the scope of the research. This part will also describe the traditional way of testing the module ‘Actuarieel Formularium’ of the IRIS system (Chapter 4). In the third part, ‘Formal Test Methods’, a distinction will be made between the different types of formal test methods and a formal test method will be selected (Chapter 5). The third part will end with a

description of the models made with the selected formal test method (Chapter 6). In the fourth part of the thesis, ‘Test Results’, the models described in chapter 6 will be used to test the ‘Actuarieel Formularium’, and the results of these tests will be described and analyzed

(Chapter 7). The final part of the thesis, ‘Advice on using Formal Test Methods‘, will be used to describe the position of the formal test method in the life cycle development model, the possible implementation areas of the formal test method, the level of usability in which the

formal test method can be used, and the levelof education the employees must have to use the

formal test method (Chapter 8). The last chapter will be used to answer the main question of the thesis and describe the further actions that must be taken to integrate formal methods into the organisation (Chapter 9).

Chapter 1: Introduction 15 The structure for this research, as described above, is depicted in Figure 1-1.

Part 1: Testing in General Chapter 2: Testing in General

Chapter 3: The Pension Administration System IRIS

Part 2: The Pension Administration System IRIS

Chapter 4: Traditional Way of Testing Chapter 5: Description Formal Test Methods

Part 3: Formal Test Methods

Chapter 6: The Formal Models

Part 4: Test Results Chapter 7: Test Results

Chapter 8: Analysis of the use of GAST and Clean

Part 5: Advice on using Formal Test Methods

Chapter 9: Conclusion and Recommendations

Figure 1-1: Structure Thesis

1.5 Research setting and the methods used

The following information concerns the setting for this research:

- The research took place within Achmea’s Test Service Center (TSC) department.

- The research started at 20 March 2006 and ended on 29 September 2006.

- The final deliverables are:

Research Plan

Master thesis

Article

Presentation

- The supervisors of Achmea for this assignment are B. van der Zee and J. van Rooyen

- The supervisors of the University of Groningen for this assignment are prof. dr. ir. J.

Simons and dr. T.W. de Boer.

The methods used during the research are both literature survey and field research.

what does it do and what test techniques are used to test the system. This knowledge has been gathered by exploring the application on the test-server and interviewing the employees of TSC who are testing the IRIS system. On the other hand, external resources have been explored. Which documentation can be found about testing applications used in a financial company. This information has been gathered by searching for scientific documents and/or books that describe test approaches, test techniques or formal test methods in general and/or for financial applications.

Part 1: Testing in General 17

Part 1

CHAPTER 2: TESTING IN GENERAL

This chapter will be used to give an introduction about testing in general, which is necessary before the real research can be introduced. The first three sub-questions, described in the introductory chapter, will be answered in this chapter:

1) What is (efficient) testing?

2) What is the life cycle development model?

3) What are formal test methods?

The first paragraph gives the general definition of testing used within the TSC department. In the second paragraph the life cycle development model, which is used for developing the software within Achmea, will be introduced. The third paragraph will introduce formal test methods and will give a definition of efficient testing.

2.1 Introduction in testing

According to Myers et al. (2004) poor tests begin with a false definition of testing. Some of these wrong definitions of testing are: “testing is the process of demonstrating that errors are not present”, “the purpose of testing is to show that a program performs its intended functions correctly”, or “testing is the process of establishing confidence that a program does what it is supposed to do”. These definitions are false because testing is not to show that a program works, but testing is finding errors in the program: “testing is the process of executing a program with the intent of finding errors” (Myers et al. 2004). The definition of testing used within Achmea adds the component ‘requirements’ to the definition of Myers et al. (2004). The definition of testing (Testmanagement team of LogicaCMG 2003a cited RTCA 1992) used within Achmea and, therefore, also in the thesis is:

Part 1 Chapter 2: Testing in General 19 The test approaches used within the TSC department have been developed based on this definition. The three most used test approaches in the Netherlands are TestFrame (TestFrame Research Center of LogicaCMG 2001), TMap (Pol et al. 2003), and Risk & Requirements Based Testing (RRBT) (Pinkster et al. 2005, Testmanagement team of LogicaCMG 2003a, and Testmanagement team of LogicaCMG 2003b). The test approach used within the TSC department of Achmea is Risk & Requirements Based Testing (RRBT) in combination with TestFrame.

2.2 Life cycle development model

The model used within Achmea to develop software is called the life cycle development model, which is a simplified representation of the Vorgehensmodell made by Bröhl and Dröschl (1995 cited van Moll et al. 2002). The life cycle development model that is used within Achmea, Centraal Beheer and Avero is shown in Figure 2-1.

Figure 2-1: The Life Cycle Development Model

At the left side of the model the phases in which the system is build are shown and at the right side you can find the different test phases used within the TSC department (Pol et al. 2003 and Testmanagement team of LogicaCMG 2003a). These test phases are:

A cc ep ta ti o n p ro ce ss Systemdesign Testdesign Testpreparation Testexecution D es ig n p ro ce ss Control Manuals Process Descriptions (including requirements) Functional Design Technical Design

Realisation Unit Test

- Unit test: In the unit test the different units of the system will be tested. The purpose

of the unit test is to decide if the independent unit works.

- Integration test: In the integration test, the interaction between the units, both

functional and technical, will be tested.

- System test: In the system test, the complete system is tested. The purpose of the

system test is to decide if the program satisfies all formulated requirements. - Acceptance tests: The acceptance tests can be divided into three parts, user’s

acceptance test, application manager’s acceptance test, and production acceptance test.

The user’s acceptance test is used to check if the system satisfies the

specifications with relation to functionality and quality.

The application manager’s acceptance test is used to check if the system

satisfies the management’s specifications.

The production acceptance test tests the system on functioning without

problems and with enough performance. The system may not disturb the other applications in the organization.

The tests described above can be called reactive testing because the tests take place after the program is made. This is also the main reason raised by many critics of the life cycle

development model (Marick 1999 and Pyhäjärvi and Rautiainen 2004). Pyhäjärvi and

Rautiainen (2004) state that the “costs of defects increase significantly the later the defects are found”. Therefore, research has been performed for years to control the costs and raise the quality of the requirements.

2.3 Formal test methods

From the technical engineering’s view the so-called formal methods are applied to technical software projects under defined conditions. The results of using formal methods in software projects are successful. One of the experiences with formal methods is that the systems can be specified with “more precision, more consistency and less ambiguity, because of their

Part 1 Chapter 2: Testing in General 21 properties in a design or specification” (Tretmans and Belinfante 2000); and you can “model and validate high levels of design” (Lindsay 1998). With these characteristics of formal methods tests can be executed earlier in the life cycle, which result in finding faults earlier, through which the tests are more efficient.

As mentioned above, formal methods can be used for developing and testing software. Because formal methods will only be used, during this research, for testing, formal methods will be called formal test methods. During the research a selected formal test method will be used to test the ‘Actuarieel Formularium’, which is a module within the IRIS system. The purpose of this research is to examine if formal test methods can be used to test the IRIS system more efficient than the traditional way of testing. Efficiency can be defined as “getting any given results with the smallest possible inputs, or getting the maximum possible output from given resources” (Black 2002). Based on this definition efficient testing can be defined as:

“The process of exercising software to detect which specified requirements do not satisfy and to detect other errors with the fewest possible tests”

This can be realized when tests can be executed earlier in the process, so fewer tests have to be executed in later phases. The advantage of formal test methods is that they can be used during the complete life cycle (Figure 2-1). When formal test methods can be used during the design and/or requirement phase within the IRIS system, errors will be detected earlier than with the traditional test techniques.

2.4 Conclusion

In this chapter an introduction in testing and the life cycle development model has been given and formal test methods have been introduced. The definition of efficient testing, which is the answer on the first sub-question, is:

Part 2: The Pension Administration System IRIS 23

Part 2

CHAPTER 3: THE PENSION ADMINISTRATION SYSTEM IRIS

This chapter will be used to describe the pension administration system IRIS to answer the fourth sub-question:

4) What is the pension administration system IRIS?

To answer this question, different elements of the IRIS system will be described. The first paragraph will be used to describe the system architecture of the IRIS system. The second paragraph will introduce the module ‘Actuarieel Formularium’, which will be the scope of this research. The third paragraph will describe the data model of the IRIS system and the fourth paragraph describes the programming languages used for IRIS.

3.1 System architecture

The system architecture of the pension system IRIS consists of six areas. Within these areas different parts of the system can be situated. The system architecture is shown in Appendix D. The six areas that can be defined are:

- Markt (Market): the purpose of this part is to register the relevant data of the market,

competitors, and intermediaries.

- Verkoop (Sale): the purpose of this part is to register the data of employers, contracts,

and regulations.

- Product: the purpose of this part is to register all technical data concerning the current

products and possible new products.

- Deelname (Participation): the purpose of this part is to register and maintain the data

of the employer/insured person.

- Financieel (Financial): the purpose of this part is to register the current account data,

to book the current account of collective pension contracts, to pay the persons entitled to relief, and to book to the ledger.

- Analyse en Verslag (Analysis and Report): this part takes care of the reports of the

Part 2 Chapter 3: The Pension Administration System IRIS 25

3.2 Actuarieel Formularium

Each area within the IRIS system consists of several modules. The module ‘Actuarieel Formularium’, which is part of the Product area, will be the scope for this research (see also the circle in Appendix D). The ‘Actuarieel Formularium’ is a tool that calculates sixteen factors based upon 58 parameters. It takes care for all calculations that must be done to calculate the premiums (fixed amount every year), purchase prices (big amount once) and deposit purchase prices (changing amount every year) per unit assured amount of money. There is chosen to use the ‘Actuarieel Formularium’ for this research because it is the most crucial part of the IRIS system and it is hard to test the actuarial functions defined in this part.

3.3 Data model

The data model shows the relations between the different parts of the IRIS system. The global data model of IRIS is shown in Appendix E. The ‘Actuarieel Formularium’ is executed within the ‘Deelaanspraak (DAA)’ (see also the circle in Appendix E). The output values generated by the ‘Actuarieel Formularium’ are processed to the ‘Deelaanspraak (DAA)’ as shown in Figure 3-1.

Figure 3-1: Deelaanspraak and the ‘Actuarieel Formularium’

3.4 Programming languages

The first version of IRIS was built in Powerhouse. To keep up with the current technology the programming language RPG was introduced. RPG is better and easier to use for the batch processes within IRIS than Powerhouse. At this moment Achmea is phasing out Powerhouse to keep RPG as the only programming language. Therefore all new batch processes are built in RPG. The only exception is the ‘Aanspraak Formularium’ and the ‘Actuarieel

Formularium’, which are written in C. This is done because of the big amount of calculations that is needed in these parts of the IRIS system.

3.5 Conclusion

In this chapter the pension administration system IRIS has been described to answer the fourth sub-question. The system architecture, with the six different areas in IRIS has been described in the first paragraph. The second paragraph described the module ‘Actuarieel Formularium’, which is a tool that calculates different factors which will be used to calculate the different premiums. There is chosen for the ‘Actuarieel Formularium’ as the scope of the research, because it is the most essential part of the IRIS system. The ‘Actuarieel

Part 2 Chapter 4: Traditional Way of Testing 27

CHAPTER 4: TRADITIONAL WAY OF TESTING

In this chapter the traditional way of testing the ‘Actuarieel Formularium’ will be described to answer the fifth sub-question:

5) What is the traditional way of testing the ‘Actuarieel Formularium’?

Testing starts when the ‘Actuarieel Formularium’ is changed. The first paragraph will describe the change process in short. The second paragraph will describe the test techniques used within the TSC department to test the changed actuarial function. The third paragraph will be used to analyse these different test techniques.

4.1 Changing an actuarial function

Testing the ‘Actuarieel Formularium’ take place when a new function is added or an existent function is changed. In the last situation, the actuaries describe the new function and the developers must change the function in the ‘Actuarieel Formularium’. Programming the new function will take about eight hours. The second step of the development process is testing the new function. The complete test process (system- and acceptance test) will also take about eight hours. The complete process, changing and testing the new actuarial function, will take about sixteen hours. The test techniques used for testing a changed actuarial function will be described in the next paragraph.

4.2 Test techniques

The most important tests are based on testing the different functionalities of the ‘Actuarieel Formularium’. The functional test techniques used for testing the ‘Actuarieel Formularium’ are:

- Ad hoc testing or Free form testing: ad hoc testing or free form testing is “a creative,

- Algorithm test: the purpose of the algorithm test is to test the structure of the program.

It tests if the possible paths within the system can be taken. (Pol et al. 2003 and Testmanagement team of LogicaCMG 2003a).

- Boundary value testing: boundary value testing “focuses on the boundaries of the

input and output equivalence classes. Focusing testing in these areas increases the probability of detecting errors” (Lewis 2004).

- Semantic test: semantic testing includes testing relations. This can be relations within

one screen; relations between data within superior screens; or relations between input data and data within the database (Pol et al. 2003 and Testmanagement team of LogicaCMG 2003a).

There are many more test techniques described in the literature (Testmanagement team of LogicaCMG 2003a; Lewis 2004; Pol et al. 2003; and Tretmans and Brinksma 2003) about testing software, but because these techniques are not used for testing the ‘Actuarieel Formularium’ and can only be used in the system and/or integration phase these techniques are not relevant for this research.

Testing the actuarial functions, executed by the users of the IRIS system, is done by using the ad hoc testing or free form testing method. The TSC department choose some extreme values to test if some output will be generated. The actuaries test a couple of test cases completely. They just simple check some possible variables and compare them to their own expectations.

4.3 Analysis traditional way of testing

Part 2 Chapter 4: Traditional Way of Testing 29 The above analysis of the traditional way of testing resulted in a case study that is conducted to examine if formal test methods can be used to test the ‘Actuarieel Formularium’ earlier in the life cycle to detect faults earlier. The next chapters will be used to describe the actual research of using those formal test methods on the ‘Actuarieel Formularium’.

4.4 Conclusion

In this chapter the traditional way of testing the ‘Actuarieel Formularium’ has been described and analyzed to answer the fifth sub-question. The ‘Actuarieel Formularium’ will only be tested when changes take place. The first paragraph described, in short, the process of changing and testing a new actuarial function. The complete process will take about sixteen hours. The second and third paragraph described and analyzed the traditional test techniques that are used to test the ‘Actuarieel Formularium’. The TSC department tests the ‘Actuarieel Formularium’ most of the time on its functionality. When the TSC department tests the actuarial functions they use the test technique ad hoc testing or free form testing. The results of the functions are tested by the users themselves.

Part 3

Part 3 Chapter 5: Selection Formal Test Methods 31

CHAPTER 5: SELECTION FORMAL TEST METHODS

Most faults in software development are originated in requirements and design phase. The current practice is that most of the test effort is put in the system and acceptance phase, which is not really efficient. To increase the efficiency of testing, the effort of testing must switch to the source of the life cycle: requirements and/or design phase. The results of using formal test methods for technical projects are very positive, tests can be executed earlier in the life cycle, through which faults are found earlier. Therefore a formal test method will be selected and used to test the ‘Actuarieel Formularium’.

As described in Chapter 2, “Formal methods are mathematical approaches to software and system development which support the rigorous specification, design and verification of computer systems” (Aichernig 2006). “A formal specification is a precise, complete, consistent and unambiguous basis for design and code development as well as for testing” (Tretmans 2002). When formal methods are only used for testing, they are called formal test methods.

In this chapter the sub-questions six and seven will be answered:

6) Which types of formal test methods are available?

7) Which formal test method can be used to test the ‘Actuarieel Formularium’?

The first paragraph will be used to answer the sixth sub-question by classifying the different formal test methods. The second paragraph will make a selection between the different formal test methods that can be used to test the ‘Actuarieel Formularium’.

5.1 Classification of formal test methods

5.1.1 State-based formal test methods:

State-based formal test methods focus on the specification of the behaviour of sequential systems and consist of “a set of states, a set of inputs, a set of outputs, a transition function, an output function and an initial state. A machine starts in its initial state. When an input is received from the environment, the transition function determines the next state, and the output function determines the output returned to the environment” (Frappier 1999). There are two types of state-based formal test methods:

- Labelled transition systems (LTS): This type is mostly based on the so-called

ioco-version. In this version the input and output are two independent actions. “The name ioco, which stands for input/output conformance, refers to the implementation relation (i.e., notion of correctness) on which the theory and the test generation algorithm have been build” (Bijl et al. 2003). More information about labelled transition systems and/or ioco is described in literature written by Tretmans (1996) and Tretmans and Brinksma (2003).

- State machine: In the state machine version the input and output are coupled during

the transition, instead of two independent actions as in labelled transition systems. More information about state machines can be found in the annotated bibliography written by Petrenko (2000).

5.1.2 Machine-based formal test methods:

Machine-based formal test methods are based on first order logic as specification. These specifications can be written in so-called functional programming languages, which are based on the concept of mathematical functions. In machine-based formal test methods the input and output are coupled during the transition as in state machines, but in this version there are no states in the program. Machine-based formal test methods can thus be seen as state machines without states.

5.2 Selection formal test methods

Part 3 Chapter 5: Selection Formal Test Methods 33 (DAA)’ as described in chapter 3. There are no states described during the process executed by the ‘Actuarieel Formularium’. Therefore, it is reasonable to use a machine-based formal test method for testing the ‘Actuarieel Formularium’ within the IRIS system. There are several tools available to test a system based on the models based on first order logic. “One of the best known examples is JUNIT for JAVA-programs, it has been ported to a very large number of other programming languages” (Koopman and Plasmeijer 2005). More information about JUNIT can be found in a book written by Link (2003). “Model based tools like

QuickCheck and Generic Automatic Software Test-system (GAST) are more powerful because they support automatically chosen input-values” (Koopman and Plasmeijer 2005); therefore one of those two tools will be used to test the ‘Actuarieel Formularium’.

- QuickCheck: To test the ‘Actuarieel Formularium’ with the tool QuickCheck

functional models, written in Haskell, must be made. “QuickCheck comes with a simple domain-specific language of testable specifications which the tester uses to define expected properties of the functions under test” (Claessen et al. 2002). QuickCheck then checks if the properties hold in a large number of cases. More information about QuickCheck is written by Claessen and Hughes (2000).

- GAST: To test the ‘Actuarieel Formularium’ with the tool GAST (Generic Automatic

Software Test-system), functional models, written in Clean, must be made. “GAST generates test-data based on the types used in the properties, it executes the test for the generated test-values, and gives an analysis of these test-results” (Koopman et al. 2002). More information about Clean can be found at the website about Clean (Software Technology Research Group 2005) and in the paper written by Koopman and Plasmeijer (2005).

described in the functional descriptions. The ICIS also realized the connection between the ‘Actuarieel Formularium’ and the GAST tool. The models, made in Clean, needed to realise the connection between GAST and the ‘Actuarieel Formularium’, are described in Appendix F.

5.3 Conclusion

This chapter gives an answer on the sixth and seventh sub-question. In the first paragraph the two categories state-based formal test methods and machine-based formal test methods have been described. It can be concluded that the machine-based formal test methods can be used for testing the ‘Actuarieel Formularium’, because there are no states described in the IRIS system. QuickCheck and GAST are the two possible tools for testing the ‘Actuarieel Formularium’. The GAST tool has been chosen because it can generate the test data in a systematic way and the set of logical operators is richer than QuickCheck. The last advantage is that the tool GAST and formal test method Clean is supported by the ‘Institute for

Part 3 Chapter 6: The Formal Models 35

CHAPTER 6: THE FORMAL MODELS

This chapter will be used to describe a selected part in the IRIS system as well as the models made with Clean of that selected part. There is chosen to define the most used pension type to show the working of the formal test method Clean and the GAST tool. The first paragraph will describe the selected actuarial function. The second paragraph will describe the models made with Clean to test the ‘Actuarieel Formularium’, which will be the answer on the eighth sub-question:

8) What will be the models, made with the selected formal test method, to test the

‘Actuarieel Formularium’?

The third paragraph will describe the working of the GAST tool in which the different models, described in the second paragraph will be used. The fourth paragraph describes models that can be used to execute a regression test with the formal test method Clean and the GAST tool.

6.1 Description of the selected part in the Actuarieel Formularium

The IRIS system consists of different types of pensions, which will be calculated by the actuarial functions defined in the ‘Actuarieel Formularium’. The types that are used in this research are the pensions of old age and the incapacitated pensions. Each type can be divided into different pensions, the so-called vormcodes. Vormcode 100, one of the pensions of old age, will be described in this paragraph, because it is frequently used.

The functions that are described within this vormcode are shown in Figure 6-1. Type Phase Vormcode Output 1 2 3 4 5 6 7 90 100 NK 100-1 100-2 100-4 100-4 100-1 100-2 NSK 100-1 100-4 NRK LF 100-1 100-2 100-4 100-4 100-1 100-2 BF 100-3 100-3 RF-1 RF-2

Figure 6-1: Vormcode 100 – Pension of Old Age

The figure presents a matrix which shows which function must be used to calculate a certain output parameter in a certain situation. For example, the output variable NK for phase one can be calculated by using function 100-1. The next part of this paragraph will give more

information about the matrix. This paragraph will end with an explanation of function 100-1.

6.1.1 Phases of ‘Deelaanspraak (DDA)’

The insurer can have different phases, which result in different payments and thus different functions. When no function is defined the output value is equal to zero. The different phases that are possible are:

1. Insurer pays the premiums

2. Running

3. Insurer is exempt from paying

4. Sleeping

5. Premium free before pension date

6. Premium free after pension date

7. Payment third parties

Part 3 Chapter 6: The Formal Models 37

6.1.2 Output parameters

The ‘Actuarieel Formularium’ defines different output parameters, of which the ones shown in Figure 6-1 are calculated with help from the defined functions. The other output variables are calculated based on fixed functions that use one of the output values calculated before. The ‘Actuarieel Formularium’ generates sixteen different output values:

- NK: Netto Koopsom (Purchase price after taxes). This output parameter is always

calculated based upon the described functions.

- BSKEN: Netto Stortingskoopsom (Deposit purchase price after taxes). This output

parameter is based on the parameter NSK, which is a factor that can be used for other calculations. The parameter NSK is based on the described functions.

- NRK: Netto Risicokoopsom (Risk purchase price after taxes). The output parameter

NRK is calculated based on the described functions.

- NRP: Netto Risicopremie (Risk premium after taxes). This parameter will only be

calculated for phase one (insurer pays the premiums) and is based on a fixed function. - LF: Lastenfactor (Charge factor). This output parameter is based upon the described

functions.

- BF: Batenfactor (Benefit factor). This output parameter will be calculated by the

described functions and only for the phases one (insurer pays the premiums) and three (insurer is exempt from paying).

- NP: Netto Premie (Premium after taxes). This output parameter will be calculated

based on the Lastenfactor (LF) and the Batenfactor (BF).

- RF-1: Risico factor 1 (Risk factor 1). This factor will be calculated based on the

described functions.

- RF-2: Risico factor 2 (Risk factor 2). This factor will be calculated based on the

described functions.

- BP: Bruto jaarpremie (Premium before taxes). This output parameter is calculated by

hand of a fixed function, which is based on the Netto Premie (NP) and a couple of input parameters.

- BK: Brutokoopsom (Purchase price after taxes). This output parameter is calculated

- BSK: Bruto Stortingskoopsom (Deposit purchase price before taxes). This output

parameter is based on the Netto Stortingskoopsom (BSKEN) and a couple of input parameters.

- QX: Risk of death for the first insurer. This output parameter is calculated based on

the so-called mortality table.

- QY: Risk of death for the second insurer. This output parameter is calculated based on

the so-called mortality table.

- RX: Rehabilitation chance. This output parameter is calculated based on a

rehabilitation table.

- BPEX: Premium before taxes excluding individual rise. This output factor is not being

used anymore and the output value will therefore be equal to zero.

6.1.3 Function 100-1

As seen in the phases and output parameters described above, a lot of models must be made with Clean to test one complete vormcode. To show the working of the GAST tool, function 100-1 will be explained. This function is used to calculate the ‘Netto Koopsom (NK)’ for vormcode 100 for the phases 1 (insurer pays the premiums) and 5 (premium free before pension date):

is the purchase price after taxes per unit assured amount of money for an assurance for life on the life of a person with age ‘x’, which will continue for the term ‘u’.

is the complex fraction lifelong rising annuity on ‘x+u’, which is continually affordable.

The following variables are used in function 100-1:

- VormCode

Part 3 Chapter 6: The Formal Models 39

- x: Lft1eVerz (age first insurer)

- u: ‘Uitstelduur-tot-ingangsleeftijd-uitkering’ (delay-duration until entrance age of the

payment is reached). This parameter is calculated as follows: u = a - x, where variable ‘a’ is the input parameter ‘IngLftUitk’ (entrance age of the payment).

- i: RekenRente (computing percentage)

- lx: the amount of persons who statistically are alive at the age of the first insurer. This

value is based on the mortality table (table starts with ten million people on age zero).

- r: StijgRente (percentage of rising)

- ω_{: the age at which only one person statistically is alive. This value is based on the}

mortality table (table starts with ten million people on age zero).

The parameters ‘x’ and ‘u’ consists of years and months. Because the function can only be calculated with years and the final answer must be based on months exactly, some other functions, called interpolation functions, are needed to calculate function 100-1. In this situation interpolation is needed on ‘x’ and ‘x+u’. The interpolation functions will not be described in the thesis to keep the explanation of Clean and GAST comprehensible. The notation of function 100-1, as shown above, is used by the actuaries. The two parts of the function can be written down as follows:

As can be seen, it is hard to calculate the functions by hand. It becomes even harder when interpolation must be applied to the functions.

6.2 Description of the models used to test the Actuarieel Formularium

In the previous chapter function 100-1 has been written down. To test this function with the tool GAST, the function must be written in a Clean model (see Figure 6-2). The specification document uses some letters for different parameters, which cannot be done in Clean.

Therefore the letters, defining the input parameters, used in the specification document and the Clean models are different. The duration parameter will be calculated by a separate function called ‘Uitstelduur’, which consists of the input parameters ‘IngLftUitk (e)’ and ‘Lft1eVerz (y)’. All general functions are written down in separate Clean documents, called ‘Gegevens’ (the durations) and ‘Kansen’ (the chances). These models can be imported when needed.

Figure 6-2: Function 100-1 before Interpolation

The models of the different functions (for example the functions 100-1, 100-2, and 100-4) will be imported in another model called ‘Uitvoer_NK’ (see Figure 6-3) in which the different functions will be used to calculate the output parameter ‘Netto Koopsom (NK)’ for the correct vormcode and phase (Figure 6-1). For vormcode 100 the model looks like the one as shown in Figure 6-3.

Part 3 Chapter 6: The Formal Models 41 The different input parameters (58 input parameters in total), that will be used to generate randomized variables, will be generated in an input model (see first part of Figure 6-4). The different input parameters will be set in long string to make it possible to put the data into the ‘Actuarieel Formularium’. More detailed information about the input model can be found in Appendix G. The second part of Figure 6-4 presents the actual values for the input parameter, which will be generated in the final test module. In Figure 6-4 an example is given for the input parameter ‘Lft1eVerz’. The values defined by this Clean model are 20, 30, 40, 50 and 60 for the years and 1 until 11 for the months of the age of the first insurer.

Figure 6-4: Input Parameter Lft1eVerz

Before the models can be tested a connection must be made with the ‘Actuarieel

Formularium’. The connection models, made with Clean, are described in detail in Appendix F.

to the number generated by the ‘Uitvoer_NK’ model. The test will be activated by the definition after ‘Start’. In Figure 6-5 can be seen that the NK will be tested for 900 test cases where 10 test cases may go wrong. When 10 test cases goes wrong the GAST tool stops automatically with testing. The input values that where used to generate the wrong output values can be compared with each other to find the cause of the fault.

Figure 6-5: Testing the Function with GAST

6.3 Working of GAST

With the GAST tool, written in Clean, a connection can be made with the ‘Actuarieel

Formularium’ (see Appendix F). The defined input parameters in Clean (see Figure 6-4) must be set in a long string, which is defined in an input model, and will be presented to the

Part 3 Chapter 6: The Formal Models 43 Figure 6-6 gives an overview of the above described working of GAST. More information about the working of GAST and Clean is written by Koopman et al. (2002), Plasmeijer and van Eekelen (version 2.0), Koopman and Plasmeijer (2005), and on the website about Clean (Software Technology Research Group 2005). The Clean code that is needed to realise the connection between GAST and the ‘Actuarieel Formularium’ is described in Appendix F and the models used to test the ‘Actuarieel Formularium’ is presented in Appendix G.

Figure 6-6: Working of GAST

1. Making models based on specification document (input,

functions, and output)

2. Present input parameters to the Actuarieel Formularium

3. Actuarieel Formularium presents output values

4. Compare results with each other and document the test results

6.4 Regression testing with GAST

An advantage that has been found during this research is that the tool GAST and the formal test method Clean can also be used for another test type, called regression testing. Regression testing is a testing type that tests ‘a system in light of changes made during a development spiral, debugging, maintenance, or the development of a new release’ (Lewis 2004). Within the TSC department, regression testing is rarely executed. With GAST a new release can be compared with the old version to check if unchanged parts of the system in the new release function the same way as in the old version. Regression testing with GAST can be realized by making four Clean models. First, a connection must be made with the old system (see

Appendix F). Next, a connection must be made with the new release of the system (see

Appendix F). The third model that is necessary is the input model (see Appendix G). The final model that is necessary to execute the regression test is the output model (See Figure 6-7). Regression testing, in this way, is different than testing unchanged functions with the Clean models described in the previous paragraphs. Regression testing only makes connection with both versions and compares those two versions.

Part 3 Chapter 6: The Formal Models 45

6.5 Conclusion

In this chapter vormcode 100-1, the Clean models used to test vormcode 100-1, the working of GAST and regression testing with GAST has been described to answer sub-question eight. The steps that are needed to test a complete vormcode are:

- Define input parameters

- Making models of general calculations

- Making models of the different functions

- Making models of each output parameter

- Making the connection module

- Making the test module

Part 4

Part 4 Chapter 7: Test Results 47

CHAPTER 7: TEST RESULTS

The previous chapter described the working of GAST and Clean by hand of vormcode 100. For this research more vormcodes have been modelled and tested. The pension types that have been used in this research are the pensions of old age and the incapacitated pensions. During this research about 15% of the ‘Actuarieel Formularium’ has been modelled and tested. The ‘Actuarieel Formularium’ is less then 1% of the complete IRIS system, but the impact, based on the amount of usage, is about 80% of the complete IRIS system. Although the ‘Actuarieel Formularium’ is a very small part of the overall system, a couple of faults have been found. These faults can be of great impact because the actuarial functions calculate the premiums. The first paragraph will be used to describe these faults, called test results, which will be the answer on sub-question nine:

9) What are the results of using the selected formal test method on the ‘Actuarieel

Formularium’?

The second paragraph of this chapter will analyze the different test results. The third paragraph of this chapter will describe some experiences of the researcher on using GAST and Clean to test the ‘Actuarieel Formularium’.

7.1 Test results

This paragraph presents the test results generated by testing the ‘Actuarieel Formularium’ with formal test methods. For this research the actuarial functions have been tested and regression testing has been conducted. The results of testing the actuarial functions can be divided into two categories:

- Results found during specification analysis

7.1.1 Results found during specification analysis

During the analysis of the specification document, needed for making the models in Clean, four specification faults are found. The first three faults are superfluously input parameters, because they are dependent and based on other input variables:

- IngLftUitk (iu): The input parameter ‘IngLftUitk (iu)’ can be calculated by the input

parameters ‘Lft1eVerz (x)’ and ‘WachttijdTotIngUitk (m)’. The calculation would then be ‘iu = m - x’. Therefore the input parameter ‘IngLftUitk’ is not needed. - Factor_wmk: The input parameter ‘factor_wmk’ is dependent on the input parameter

‘Regl_indv_code’. When ‘Regl_indv_code’ is equal to 2 the ‘factor_wmk’ is 0.95, otherwise ‘factor_wmk’ is 1.10.

- OpslRekenrente: The input parameter ‘OpslRekenrente’ is dependent on the input

parameter ‘RekenRente’. If ‘RekenRente’ is equal to 3 then ‘OpslRekenrente’ is equal to 0 and when ‘RekenRente’ is equal to 4 then ‘OpslRekenrente’ is equal to 6.

The fourth and last specification fault is the specification on which the output parameter ‘Netto Risicopremie (NRP)’ must be calculated. The specification document gives the following calculations: NRP = NRPE and NRP = NRPE * 1000. Purely based on this information it is not clear which function must be used.

7.1.2 Results found after testing

After the analysis of the specification document and making the models, the ‘Actuarieel Formularium’ have been tested, automatically, with the GAST tool. This way an test analysis sheet, used in the traditional way of testing, is not needed anymore. Testing the actuarial functions with GAST, based on the models made in Clean, result in five differences between the values generated by the ‘Actuarieel Formularium’ and the values generated by GAST.

- Climbing functions in the incapacitated pensions: These functions have been changed

almost a year ago. The changes must still be tested by the actuaries. The system uses the new functions, but the specification document still represents the old functions. - Interpolation faults: Some functions present differences between the ‘Actuarieel

Part 4 Chapter 7: Test Results 49 found in the pensions of old age and the incapacitated pensions. Appendix H shows some examples of differences that have occurred during the tests.

- Function 650-3: The function described in the specification document is the

following:

This function was defined in a Clean model and tests were executed. The comparison made by GAST said that all situations are wrong. When the following function was defined in Clean and compared with the output values of the ‘Actuarieel Formularium’ all test situations are correct:

The function that is used within the ‘Actuarieel Formularium’ has been found by trial and error. It is possible in Clean models to deselect a part of the function. This way the function used in the ‘Actuarieel Formularium’ has been found.

- Function 100-3: The values generated by the ‘Actuarieel Formularium’ and the values

generated by GAST consists of differences around the 0.5. It is not clear what the cause of the difference is. Appendix H shows some examples of these differences. - Function 121-1 and function 121-2: These functions consists for a part of the

following function:

7.2 Analysis test results

This paragraph will analyse the different test results found by using the formal test method on the ‘Actuarieel Formularium’.

7.2.1 Analysis of the results found during specification analysis

The test results that have been found during the analysis of the specification document are only written down within the specification document, but will not be changed in the system. This is done, because the costs of changing these parameters are higher than the benefits. For the users of the ‘Actuarieel Formularium’ it is a point of attention to skip these input

parameters. Only the output parameter ‘Netto Risicopremie (NRP)’ can be changed in the specification document to solve the problem. The function NRP = NRPE * 1000 is the right function that is needed to calculate the output parameter ‘Netto Risiscopremie (NRP)’.

7.2.2 Analysis of the results found after testing

The test results that have been found after testing the ‘Actuarieel Formularium’ with the formal test method Clean must be verified by the actuaries.

- Climbing functions in the incapacitated pensions: The climbing functions in the

incapacitated pensions are under construction by the responsible departments within Achmea. When the final tests are done, the specification document will be made up-to-date.

- Interpolation faults: The interpolation faults found after the tests do not have an

Part 4 Chapter 7: Test Results 51 - Function 650-3: A wrong function is used within the ‘Actuarieel Formularium’.

Therefore the function must be changed. In the traditional situation, described in Chapter 4, the complete process, changing and testing the new actuarial function, will take about sixteen hours. The advantage of the formal test method Clean is that the models can be reused when the functions have been changed. Testing a separate function with Clean will take only a few minutes. This means that the complete process of changing and testing the new actuarial function will take eight hours instead of sixteen hours.

- Function 100-3: The cause of the fault must be found and a little change is needed,

which will take a few hours. As described before, the Clean models can be reused to test the changed functions.

- Function 121-1 and function 121-2: The cause of the fault must be found in the

described part of the function. After that, this part must be changed, which will take a couple of hours. As described before, the Clean models can be reused to test the changed functions.

7.2.3 Impact of the test results on the business

The test results have an impact on the business. This impact will be mainly financial. The actuaries must check the amount of usage of the wrong functions and the actual difference of the premiums. When those two questions have been answered the impact of the test results on the business can be calculated. On the moment of writing the final version off the thesis the actuaries were still busy to determine the impact on the business.

7.3 Experiences

7.3.1 Personal experiences

According to the researcher it is very important to know the language Clean well, because making faults in Clean is very easy. Therefore, enough time must be reserved to learn the language. More important than knowing Clean is the knowledge of the system that must be tested with GAST. When the system is not completely understood, it is hard, maybe

impossible, to make good models. In that situation a comparison will be made between the system and wrong models. This will result in differences between the values generated by GAST and the values generated by the system while the system could be correct. Another point of attention is that good understanding of the actuarial functions is unavoidable. Without this knowledge it is not possible to make the models in Clean.

Another personal experience is the difficulty of finding the cause of the differences between the values generated by the ‘Actuarieel Formularium’ and the values generated by the models. The main reason for this is that the ‘Actuarieel Formularium’ does not show the results

halfway the complete calculation. Therefore, the more complex the functions, the more complex it is to find the cause of the faults. Because the functions are complex, the models in Clean will be complex, through which the chance of faults in the models will be high. The question raises, who will test the Clean models. This is an overall problem that consists in every test technique. The only solution is to review the quality of the models before using it for testing the actuarial functions. The advantage of the Clean models is that the models of the functions in Clean are made differently than the ones programmed in the ‘Actuarieel

Formularium’, therefore it is not reasonable that in the Clean models the same mistakes will be made as in the ‘Actuarieel Formularium’. The advantage of the GAST tool is that it uses the specification itself during the tests in stead of specific test cases derived from the specification as done in unit testing.

Part 4 Chapter 7: Test Results 53

7.3.2 Experiences for GAST and Clean

The formal test method Clean and the tool GAST have been used during this research for the first time on a real life system used in an organisation. This means that Clean and GAST is tested during this research as well. This paragraph will therefore represent some experiences gained by using GAST and Clean.

The experience on using GAST is that the tool can execute 900 test cases within 10 minutes when a complete vormcode is tested. When a regression test is executed 1500 test cases can be executed within 10 minutes. When a tester wants to execute more test cases than described above GAST will be closed automatically without executing the test cases.

The experience on using Clean is that the models can be reused. Once a model is made it can be used whenever the tester wants. This is a big advantage when faults are found. The

actuarial function must be changed in the ‘Actuarieel Formularium’, which will take about eight hours. In the traditional situation the retests will take also about eight hours, while with the Clean models the tests will take only a few minutes.

7.4 Conclusion

This chapter described the test results found with the formal test method Clean and the GAST tool to answer the ninth sub-question. Faults have been found during the specification analysis and after the tests. There have been found four specification faults, where only the

specification fault for the output parameter will be changed. The real tests have resulted in five faults, of which three of these faults will result in changing the ‘Actuarieel Formularium’. In the traditional situation this will take about sixteen hours to complete the whole process. With the formal test method Clean the tests can be reused and the complete process will take about eight hours. The financial impact on the business will be determined by the actuaries. Because the actuaries were still busy to determine the impact on the business on the moment of writing the final version off the thesis, the impact cannot be defined.

The experiences of using GAST and Clean have also been described in this chapter. The personal experience of the researcher is that knowledge of the system, the actuarial functions and the Clean language are crucial to use the formal test method Clean for testing the

Part 5: Advice on using Formal Test Methods 55

Part 5