semantics in the generated code
Alex L.E. Kok
alexkok08@gmail.com
October 11, 2017, 78 pages
Research supervisor: Jurgen J. Vinju Host supervisor: Jorryt-Jan Dijkstra Host organisation: ING,http://www.ing.nl
Universiteit van Amsterdam
Faculteit der Natuurwetenschappen, Wiskunde en Informatica Master Software Engineering
Abstract 3
1 Introduction 4
1.1 Problem statement . . . 4
1.2 Solution direction. . . 5
1.3 Research method . . . 6
1.4 Contribution & outline. . . 7
1.5 Related work . . . 7
1.5.1 Axiom-Based Testing . . . 7
1.5.2 Random testing. . . 7
1.5.3 Testing generated systems . . . 8
1.5.4 Product-line testing . . . 8
1.5.5 Model-based testing . . . 9
2 Background and context 11 2.1 Rebel . . . 11
2.1.1 A Rebel example . . . 11
2.1.2 Rebel checkers versus our approach. . . 13
2.2 The Scala/Akka generator . . . 14
2.3 Property-based testing . . . 14
2.4 Terminology. . . 15
3 Properties of Rebel 17 3.1 Determining the properties . . . 17
3.2 Property definitions . . . 18
3.2.1 Field properties. . . 19
3.2.2 Properties of equality and inequality . . . 22
3.2.3 Additional properties of equality and inequality. . . 26
3.3 Conclusion . . . 28
3.4 Threats to validity . . . 29
4 Test mechanics 30 4.1 The test framework. . . 30
4.1.1 Create specification . . . 31
4.1.2 Check & build . . . 32
4.1.3 Generate system . . . 32
4.1.4 Generate test suite . . . 32
4.1.5 Run test suite. . . 33
4.1.6 Test framework evaluation. . . 35
4.2 Conclusion . . . 36
4.3 Threats to validity . . . 37
5 Experiment 1: Using random input 39 5.1 Method . . . 39
5.2.1 First run . . . 39 5.2.2 Second run . . . 40 5.3 Analysis . . . 41 5.4 Evaluation criteria . . . 45 5.5 Conclusion . . . 46 5.6 Threats to validity . . . 46
6 Experiment 2: Improving coverage 48 6.1 Method . . . 48 6.2 Results. . . 50 6.3 Analysis . . . 51 6.4 Evaluation criteria . . . 53 6.5 Conclusion . . . 54 6.6 Threats to validity . . . 54
7 Experiment 3: Automatic value generation 56 7.1 Method . . . 56
7.1.1 Updating property definitions . . . 56
7.1.2 Improving dynamicallity . . . 57 7.2 Results. . . 59 7.3 Analysis . . . 59 7.4 Evaluation criteria . . . 60 7.5 Conclusion . . . 61 7.6 Threats to validity . . . 61 8 Discussion 62 9 Conclusion 66 9.1 Future work . . . 66 Bibliography 69
A Property definitions in Rebel 71
Rebel is a domain specific language focused on the financial industry. Banking products can be speci-fied in Rebel. The tool chain for Rebel can be used to check, simulate and visualize Rebel specifications to reason about the specified banking product. The tool chain for Rebel also provides some genera-tors. These generators can generate a system based on Rebel specifications. These generated systems provide an API to work with the specified banking products.
Although the generated system is based on the Rebel specification(s), the generated code is not being checked against the Rebel specification(s). This means that the generated system is perhaps working as expected, or perhaps not. In this thesis we aim to improve this, by automatically testing the generated code using property-based testing and the Rebel language itself.
First we define the expected properties of Rebel, using the existing axioms of algebra as inspira-tion. Then we write these properties down as “events” in a new Rebel specificainspira-tion. We introduce our test framework, to automatically test this Rebel specification on the generated code. This is done as follows: the test framework uses the existing generator in the Rebel tool chain to generate the system for this Rebel specification. It then generates a test suite containing test cases for each “event” in this Rebel specification. As last, it runs the test suite against the generated system.
With this approach we identified some problems in the generated code that were unknown before. We have found precision errors, overflow/underflow errors and a compilation error. However, the number and the kind of bugs that can be found with this approach depend on the properties that were defined in the first place. We conclude that, by using this approach, the semantics of the generated code can be checked automatically on whether it satisfies the defined properties. The set of defined properties in this thesis is not complete for Rebel. Therefore, it is not the case that all the generated code is tested, but only the generated code concerning the defined properties is being tested.
Introduction
Large systems often suffer from domain knowledge that is implicit, incomplete, out of date or con-tains ambiguous definitions. This is what Rebel aims to solve for the financial domain [1]. The toolchain of Rebel can be used to check, simulate and visualize the specifications, allowing to reason about the final product [2]. Checking is done based on bounded model checking by using the Z3 solver. Generators are being used to generate a software system from the Rebel specifications. The gen-erated system provides an API in order to work with the specified product and handles the database connectivity. However, the implementation of the generated system is not checked against the specifi-cations, meaning that the generated system is perhaps not doing what it is supposed to do according to its specifications. The aim of this project is to improve this, by automatically testing the generated system against a Rebel specification.
1.1
Problem statement
From the Rebel specifications, a system can be generated by the generator. However, neither the gen-erator nor the generated program is being tested against the specification. Thus it could be that the generated system doesn’t work according to what was specified in the Rebel specification. Although the generator should translate everything correctly, we cannot assume that it actually does translate it correctly for each case and that the implementation works as expected.
Currently, there are no tests for the generator or the generated system, during the development of the generator the results are being checked manually. Testing is a major cost factor in software development, with test automation being proposed as one of the solutions to reduce these costs [3]. We aim for an approach such that much of the testing is automated to reduce the time (and costs) needed for testing certain components of the generated system.
The main research question is as follows:
How can the generator in the Rebel toolchain be tested automatically, by using the gen-erated system, to check whether the implementation works as expected?
We investigate the following solution: generating tests based on a Rebel specification and then run these tests against the generated system.
Property-based testing is an approach to validate whether an implementation satisfies its specifi-cation [4,5]. It describes what the program should or should not do. As described in [4]: “Property-based testing validates that the final product is free of specific flaws.”. With property-“Property-based testing, a property is being defined which should hold on the system. Next, the property is being tested for a certain number of tries, using different input values to check whether the property holds. In case the property doesn’t hold, it will result in a failure, reporting that there is a case in which the property
doesn’t hold. This indicates that a bug in the system has been triggered.
Property-based testing has already shown a success in earlier studies [4, 6, 7], by detecting errors in a system that were not known before. In this thesis, we will use property-based testing to check the generator, by using the generated system to check whether the properties hold.
We hypothesize that there are yet unknown bugs in the generator, resulting in that the generated system does not work as expected. By using property-based testing we expect to detect bugs in the generator.
To answer the main research question, we will first answer the following research questions: RQ 1: Which properties are expected to hold on the semantics of the generated code?
RQ 2: How can we test each property as automatically as possible to find bugs in the generator? RQ 3: What kind of bugs can be found using this approach and how many?
The third research question is influenced by the first research question. When the expected properties are known, we can indicate what kind of bugs can be found by using this approach. Namely, bugs that invalidate these properties. These properties should be meaningful for Rebel, we focus on properties of Rebel when using the available operators and types in the Rebel language.
1.2
Solution direction
The expected properties which we would like to test on the generated code first have to be defined. This is the result of the first research question. The generator in the Rebel tool chain requires a Rebel specification as input to generate a Scala system (called “generated system” throughout this thesis). A Rebel specification contains the states, the data fields and the transitions between the states along with the pre- and post conditions of the transitions. The transitions in a Rebel specification are called “events”. These levels of abstraction can be confusing. InSection 2.4we describe the confusing ter-minologies and abstractions in detail to avoid this confusion throughout this thesis. Rebel is described in more detail inSection 2.1.
When the properties are defined, a Rebel specification can be created in which the properties are written down as “events” in the Rebel specification. Each property can be implemented as one “event” in the Rebel specification. Next, a test can be generated from each “event” in this Rebel specification. These tests can then be run against the generated system to check each property on the generated code. When a test fails, a bug has been found. In the next section we will describe this in more detail.
In order to run the test suite, we assume that the generated system can be compiled and that it can be run. But nothing more, such that our approach can be reused between different generators for Rebel. The specification which was used to generate the system should be syntactically and se-mantically correct. Which means that the Rebel type checker should not report any errors about the specification.
The test framework generates a test suite that can be run against the generated system. In case the test suite finishes without errors, it means that it did not find any bugs and that the generator satisfies the properties that were tested. This doesn’t mean that there are no bugs in the generated system, instead, it means that our test suite was not able to find errors in the properties that it checks for. The generated system will probably still contain bugs which are not detected by using the test framework. In this case, improving the test framework might extend the number of bugs that it can find.
1.3
Research method
To check each property on the generated code, we divide this process in different phases. Most of the phases are executed by the test framework such that minimal manual steps are needed to test the defined properties. An overview of our approach is shown inFigure 1.1, each step is explained in detail inChapter 4.
Figure 1.1: Overview of the test framework and the phases
We start off with defining the properties that are expected to hold on the semantics of the generated code. Coincidentally, the Rebel syntax is strong enough such that the defined properties can be writ-ten down in the Rebel syntax. This means that a specification can be created from these properties, in which each property is defined as an “event” in the Rebel specification (Figure 1.1, phase 1). Only this phase is done manually, the next phases are executed by the test framework.
This Rebel specification is checked by the type checker and then being built, resulting in a Con-crete Syntax Tree (CST) (Figure 1.1, phase 2). The Rebel tool chain provides a generator to generate a Scala/Akka system by using the CST of the specification (Figure 1.1, phase 3). The same CST is used to generate the test suite (Figure 1.1, phase 4). The test suite consists of multiple tests based on the events in the CST. Note that, as described earlier in phase 1, the defined properties are written down as events in the Rebel specification.
The test suite is then being run against the generated system (Figure 1.1, phase 5). After run-ning the test suite, we analyse the results of the experiment. When one or more tests are failing, a bug is found. An investigation of the failing case is needed to discover the actual bug is because the cause of the failing case is not clearly visible in the log.
We evaluate the test framework in each experiment using test coverage and the number of bugs found as metrics. Based on the results of an experiment, we improve the test framework and continue to the next experiment. In the next experiment we run the test framework again with the improve-ments, analyse the results and evaluate the test framework again.
The full source of our test framework is available on Github1. However, note that the test framework
makes use of the Scala/Akka generator that is written by ING and is closed-source. Thus in order to run the test framework, access to the code of the generator is also required as the test framework depends on this generator.
1.4
Contribution & outline
InChapter 2we describe the background (Rebel, the generator and property-based testing). In Chap-ter 3we provide definitions of the properties that are expected to hold when using the Rebel language and its toolchain. Allowing to reason about whether the properties are satisfied when using the generator and answering the first research question: Which properties are expected to hold on the semantics of the generated code? There were no existing property definitions of Rebel, so we provide a starting point for this with the focus on important properties of Rebel.
The defined properties are being used to check the generator. In Chapter 4 we describe this pro-cess in detail, which contributes to answering the second research question: How can we test each property as automatically as possible to find bugs in the generator?
In the experiments (Chapter 5, Chapter 6 and Chapter 7), the test framework is run with the set of defined properties. For each experiment we evaluate the results, investigate the failing tests and evaluate the test framework. The results of the experiments contribute to answering the third research question: What kind of bugs can be found using this approach and how many?
As last, we provide a discussion about the findings and answer the research questions inChapter 8. Followed up by the conclusion inChapter 9.
1.5
Related work
1.5.1
Axiom-Based Testing
In axiom-based testing (also known as property-based testing) axioms are defined which describe what the expected behaviour is of the system under test. It was introduced already in the early eighties, with DAISTS [8]. The axioms, which they write themselves, are being used as a basis for unit testing. The set of axioms describe the expected behaviour and are used to determine whether the implementation agrees with the defined axioms. Testing is automated, the user only defines the axioms and writes the implementation for this.
DAISTS focused on algebraic axioms, using the equality operator in the definitions. Axioms can also be specified in object-oriented style. An example of this is ASTOOT [9] which applies the ideas of axiom-based testing. Axioms are defined in an object-oriented style, describing the methods used and the expected outcome. Whereas in algebraic style, the axioms are written more in a functional notation (as is done in DAISTS ). In this thesis, we use the algebraic style and use the equality as well as the inequality operators. Furthermore, we also have to define the expected behaviour of Rebel expression.
In [10] axioms are used to automatically generate tests from these. They use the Concepts in C++ which contain the definitions of the axioms. Based on these Concepts they automatically generate tests for each defined axiom and run these tests to check whether the system under test satisfies these axioms. In this thesis, we define the properties (axioms) that are expected to hold in Rebel and then generate tests based on these definitions by using the generator. The generated tests will then be run against the generated system that can be generated from a Rebel specification. Our approach can be compared to the approach in [10], but we define the axioms in a Rebel specification and then run the generated tests against the system that was generated based on the Rebel specification that we created.
1.5.2
Random testing
Random testing is a technique in which random values are being used as input for the test cases. QuickCheck [6] and Randoop [11] are examples of random testing techniques. These differ in how they automatically test systems and what actually is being tested. QuickCheck is based on
property-based testing, which we also use throughout this thesis. Randoop on the other hand is property-based on feedback-directed random testing.
With feedback-directed random testing, random tests are generated which will immediately be run. The result of earlier test attempts can affect the next test that is being generated, which can be seen as feedback for the next generated test. This allows each test case to ‘learn’ from earlier attempts and to create unique tests.
Randoop is built for Java projects and checks some built-in specifications of Java that can’t be checked by the compiler. The test cases are simple unit tests, consisting of a unique sequence of methods due to the feedback of earlier attempts. The method sequences are unique because it also checks whether the same case has already been checked. Since there can be unlimited sequences of methods to test, the test suite will be terminated after a defined timeout. Next, the result is deter-mined and failing cases are being reported, although when using this approach it cannot determine whether the whole system is correct according to the Java specifications. Instead, it just wasn’t able to find a case for which it fails. This is a useful approach to generate unique tests, but its goal is to check systems built-in Java and thus is not compatible with the semantics of Rebel. It works with calling the Java objects and using methods on those, while the generated system consists of mainly states and events. When using an approach like Randoop with random input values, it cannot be known whether the result of a specific transition was expected to succeed or fail.
Approaches like QuickCheck [6] and Randoop [11] enforce the system under test to be written in a specific language (Haskell for QuickCheck, Java for Randoop). For QuickCheck there are alterna-tive solutions for other languages. In our case, we need to use Rebel when generating test cases, such that we test the generator when generating the tests. Another reason why we can’t use a method like Randoop is that Randoop strictly checks for Java properties, which are not in line with the Rebel language.
1.5.3
Testing generated systems
Using domain-specific languages leads to the creation of code generators to automatically generate executable code [12]. This is also the case for Rebel, the toolchain provides some generators which use a Rebel specification as input. These generators and the resulting generated systems can be au-tomatically tested too. In [12] they focus on testing non-functional properties, namely testing the performance of the generated code. In [13] the authors present a regression testing automation frame-work for a domain-specific language. They detect regressions in the generated systems by generating and executing test cases.
In this thesis, we also focus on automatically testing the generator (by using the generated system) but we check more basic functionalities that are specific to Rebel and its supported types.
1.5.4
Product-line testing
In [14] the authors describe a way to efficiently test a software product line. A software product line consists of a set of software systems that share a common set of features. However, many different products can be created from such a software product line, adding a challenge to test each combination of those products. In [14] the authors introduce IncLing, which efficiently tests combinations of these products and take already executed combinations into account when selecting a new combination. To determine the possible combinations they use the IPCL algorithm introduced in [15], an algorithm that can be applied to determine the combinations of software product lines at the size and complexity found in projects in the industry.
IncLing executes its tests sequentially by selecting a combination of features that were not covered in the previous test. This is a different way of testing each possible combination and ensuring unique test cases. In this thesis, we basically define the possible combinations by defining the properties
(axioms) that we expect to hold in Rebel. Additionally, the input values are determined random, allowing different combinations as in values. However, we do not test combinations of the properties combined, but test each individually. In our approach there could be some overlap over the various defined properties.
1.5.5
Model-based testing
Model-based testing is an evolving technique to generate tests based on a model [16]. The model is used to describe the requirements to which the system under test. Essentially, the model is a “spec-ification” of the inputs to the system under test. To avoid confusion with our thesis, we stick with “requirements” in this section.
From the data model, tests are being generated and run against the system under test. The ex-pected outcomes of the test cases are known beforehand based on the requirements. Thus, after running the tests against the system under test, the actual results are compared to the expected results. When the results are different compared to each other, it indicates that there is an error. In Rebel a banking product can be specified. Which is essentially the abstract model of the product. When using the model-based testing approach one could check a generated system with the Rebel specification that was used to generate that system. Another master’s thesis was done in parallel on the same topic [17]. They use model-based testing to test the generated system and uses the Rebel specification that was used to generate the system as data model. They use the existing solver that is available in the Rebel tool chain to generate tests. Next, they run each test against the generated system.
Although the topics are similar, the approach and the results are different. We will describe the differences in between these approaches in detail shortly. The results are different in that the bugs found are different. For example, they found distribution errors while we found precision errors. It is also important to note that we have only used one generator throughout our thesis, while [17] uses multiple generators.
We expect that some bugs found in the other generators can be detected with our approach too, such as the incorrect implementations for certain operators. When the properties have been defined for those, we should be able to detect those. However, we did not find such errors in the generator that we used when testing the properties that we defined. However, the vice-versa is also possible, their approach could probably find the bug we have found. Although they are depended on the traces that are being generated by the solver.
We discuss the differences in input, the test cases and the number of tests between these approaches: Input specification(s)
The first difference are the input specifications that are being used to test the generated system. In our approach we create a new Rebel specification based on the defined properties. This can be done due to the fact that the Rebel syntax is strong enough to write the properties down in Rebel. Then we use this specification to generate tests for the generated system.
With the model-based testing approach in [17], existing Rebel specifications are used as model to test the generated system. The Rebel specifications for the model are the same specifications as those that were used to generate the system.
From this, we can conclude that: in our approach we test the generated system on certain prop-erties. Exactly these properties should then be tested for all Rebel specifications. Thus we test things that should hold for all Rebel specifications. While with the model-based testing approach only spe-cific Rebel spespe-cifications are being tested on the generated system. Allowing to reason about spespe-cific Rebel specifications, but requires test runs for each specification. While with our approach, we run
the test framework once and reason about all Rebel specifications. Tests
Another difference is the way how the tests are being determined and generated. In our approach we generate a test case for each “event” in the Rebel specification. Remember that the properties are being written down as “events” in the Rebel specification that we create. Thus we know exactly what we test. A test case consists of checking the property 100 times with random input values. These test cases can be considered “unit tests” as each test checks one and only one property definition. We first generate all the test cases and then run the test suite against the generated system.
In [17] they use the SMT solver to generate a test case. Note that they use existing Rebel speci-fications as input, which already define the possible transitions for a specific banking product. Due to the SMT solver that they use, the actual test cases are unknown as these are determined by the SMT solver. The test cases can differ in each run and thus cannot exactly know the contents of each test. However, due to the solver they assume that the test case is valid and thus finds bugs when a test fails. Thus we generate unit tests and run the whole test suite against the generated system. While in [17] they use the SMT solver to generate the tests at runtime, requiring to have both the solver and the generated system to be running available while the test framework is running.
Number of tests
The number of tests that are done also differ. Since we generate unit tests for each property, many tests are being generated. Additionally, each test consists of checking a property 100 times with random input values. Despite of the many generated tests, the execution time of the whole test suite is considerably fast since each test is considerably small. Although we did not measure the efficiency in our experiments specifically.
In [17] they generate a test case by using the SMT solver. The test is then run against the gen-erated system, checks the result and continuous to the next test by repeating this process. Executing such a test case can be considered slower compared to our test cases. Although we test each property 100 times in a single test case, increasing this amount would affect the execution time too. However, we can reason about properties on all Rebel specifications while they can reason about a existing Rebel specifications.
Background and context
2.1
Rebel
Rebel is a domain specific language that focuses on the banking industry [1]. Banking products can be specified in the language, with the use of Rebel types like Money, IBAN and Percentage. In Sub-section 2.1.1an example of a Rebel specification is described.
The tool chain of Rebel allows to check, visualize and simulate the specified banking product. For checking and simulation an efficient, state-of-the-art SMT solver is being used called Z3, which is developed by Microsoft [18]. Rebel is written in Rascal and is developed by the ING in corporation with the CWI. Currently, the tool chain of Rebel is also written in Rascal.
Checking a Rebel specification is based on Bounded Model Checking [1]. The Rebel specification is being translated to SMT constraints, next, the Z3 solver is being used to check whether the spec-ification is consistent. An inconsistent specspec-ification means that a counter example has been found (meaning that a trace is found for which an invariant doesn’t hold). It is bounded since it only checks if a counter example can be found within a predefined number of steps. Besides checking the spec-ification, the specification can also be simulated. For simulation, the SMT solver is also being used to determine whether a transition can happen. After successfully checking the specification, meaning that no counter examples could be found, the result is still that the specification ‘might’ be valid. As the checking method is bounded, it stops at a certain point (which, in the Rebel toolchain, is defined as the maximum depth of the traces that can be used for checking). This means that there can still be a long or untested trace for which the invariant doesn’t hold.
From the Rebel specification, a system can be generated by using a generator which is developed by ING. The generators are also written in Rascal. This requires a specification that is consistent and that does not trigger errors by the type checker.
2.1.1
A Rebel example
InListing 2.1an example of a simple bank account specification is shown in Rebel. In this specification, an account can be opened and money can be deposited to an opened account.
1 modulesimple account.Account 2 importsimple account.Library 3
4 specificationAccount { 5 fields {
6 accountNumber: IBAN@key
7 balance: Money 8 } 9 events{ 10 openAccount[] 11 deposit [] 12 } 13 lifeCycle{
14 initial init −> opened: openAccount 15 opened −> opened: deposit
16 final opened 17 }
18 }
Listing 2.1: A simple account specification in Rebel.
The module keyword describes the module of this specification. The import statement is used to import logic from another file, which we will cover when describing the events block. The specification of the account itself is defined in the specification block, which consists of the following three blocks: Fields
The fields block describes which data is available in this bank account. In this case, accountNumber and balance which are of type IBAN and Money respectively. The accountNumber is used as identifier for this account specification, because of the @key notation behind the type.
Life cycle
The lifeCycle block describes the life cycle, it can be seen as the life cycle of a finite state machine. It describes the states and the transitions between them. It also defines the initial and final state. Note that the transitions between these states are the events that have been defined in the specification. Events
In the events block the events for this specification are described. The optional parameters of the event definition can be overloaded inside the brackets. Note that the implementation of the events are not described here. This is defined in another file called Library, shown inListing 2.2.
1 modulesimple account.Library 2 importsimple account.Account 3
4 eventopenAccount[minimalDeposit: Money = EUR 0.00](initialDeposit: Money) { 5 preconditions{
6 initialDeposit >= minimalDeposit;
7 }
8 postconditions{
9 new this.balance == initialDeposit;
10 }
11 }
12 eventdeposit(amount: Money) { 13 preconditions{
14 amount > EUR 0.00;
15 }
16 postconditions{
17 new this.balance ==this.balance + amount;
18 }
19 }
Listing 2.2: The library for the simple account example in Rebel.
An event definition can consist of preconditions, postconditions and a sync block. Each of those can only occur once in an event definition. The sync block defines that each operation inside that block should execute synchronously, without the interaction of other events in the meanwhile. However, in this simple specification the sync block is not used.
The preconditions and postconditions blocks contain expressions and can use functions. These func-tions can be defined in Rebel too. The new keyword in the postcondifunc-tions block describe that the value of the field should be updated when the event transition takes place.
2.1.2
Rebel checkers versus our approach
A Rebel specification can be checked by the Rebel tool chain, while we are checking properties when using the generator. Additionally, the Rebel tool chain also has a type checker. In this section we aim to solve this confusion by describing both in more detail.
The Rebel tool chain contains a checker which checks a Rebel specification using bounded model checking. It is limited in that it only checks a Rebel specification, by trying to find cases in which the invariant doesn’t hold. It does not do anything with Rebel specifications without an invariant. Thus, it only checks a Rebel specification and is not able to check or test the implementation of the generator or the generated system. The input is a set of Rebel specifications, only these Rebel specifications are being checked, nothing more. It only supports checking a Rebel specification. Due to this it cannot be used on the generated system, as this is written in Scala.
The type checker in the Rebel tool chain is used to determine whether certain expressions are al-lowed in Rebel. Operations like +, −, ∗ and / can be used in Rebel, but these are not compatible with each and every type. For example, a Money value cannot be multiplied by another Money value. Such errors should be prevented by the type checker. The type checker will also be taken into account when defining the properties inChapter 3.
In our property-based testing approach we check whether certain properties hold when using the generator. These properties describe the expected working of the Rebel language. Due to the syntax of Rebel, we can define a Rebel specification containing these properties. The type checker will prevent type-related errors in the property definitions. Our test framework uses this Rebel specification to check whether these properties hold when using the generator. Thus, all defined properties are being checked by only running the test framework once.
This means that, in our approach, we test whether the generated code works as expected for ev-ery Rebel specification that can be created. While the existing checker in the Rebel tool chain, that uses bounded model checking, only checks whether a Rebel specification is consistent. This existing checker is not able to check whether the generated system is working as expected. In our approach we check the generated system, but we only check parts of the whole implementation. Namely, the parts which are covered by the defined properties.
2.2
The Scala/Akka generator
There are multiple generators developed within ING. The generators are different in that the result-ing product is written in a different language or uses a different implementation, like database or messaging layer. Each generator is written in Rascal.
The Scala/Akka generator is one of these generators. It generates a system that is written in Scala, uses Akka [19] as actor system and uses Cassandra [20] as database. This generator is often used within experiments inside ING and is considered the most mature generator among the currently developed generators. Because of this, it’s interesting if we encounter yet unknown problems in the generator itself or the resulting system. We will only make use of this generator throughout this thesis. A specification can be defined in terms of a Labeled Transition System [1], containing the states, the data fields and the transitions between the states along with the pre- and post conditions of the transitions. The generated system is based on these states and transitions defined in Rebel, resulting in a system that works like an Extended Finite State Machine. Thus the generated program also implements it like states and transitions between them. An instance of a banking product can have fields and is in a specific state. In order to go to another state, a transition can be done which might have pre- or post conditions. In case there are pre- or post conditions, these have to be satisfied in order to successfully complete a transition.
Rebel introduces custom types, such as IBAN, A library or own implementation is used for the custom types available in Rebel. An example is the Money type, which is available in the Squants [21] library. The generated system uses this library to deal with the Money type and its operations. Another example is the Percentage type, which is simply translated by calling a method percentage() . In order to conclude that the generated program is doing something incorrectly, we have to specify what the expected properties are in Rebel.
2.3
Property-based testing
With property-based testing properties are defined and being tested. It uses random values as input and checks whether the defined property holds. After a certain number of succeeding cases the test succeeds and the next property is being checked.
A well-known tool which is based on property-based testing is QuickCheck, which is written for Haskell [6]. It tests the properties automatically by using random input values. For each property QuickCheck tries to find counter examples, which are a set of values for which the desired property does not hold. If 100 test cases are succeeding in a row, it goes on to the next property. In case it found a counter example, it will try to minimize the values to try to report the edge case of the failure. However, one might have properties that only hold under certain conditions. For this QuickCheck allows using preconditions. Although, this doesn’t work well for every case as QuickCheck will just generate new pairs of values in case the precondition didn’t hold for the generated set of values. An example of this is where 2 of the input values have to be equal, the chance that this happens with random values is rather low. QuickCheck will try to generate new values each time, with a maximum of 1000 tries by default. In case this maximum is reached, it reports the case as “untested” and
continues to the next property. Note that these values of 100 and 1000 are the default values, these can be adjusted when needed.
Due to the effectiveness of QuickCheck, many ports for other languages were written. Most ports implement the basics of QuickCheck, additionally, each port could have added extra features. Exam-ples of some ports are FortressCheck (for Fortran) [22] and ScalaCheck (for Scala) [23]. FortressCheck supports polymorphic types and, unlike QuickCheck, heavily uses reflection for its value generation to solve certain problems with polymorphic constructs. Although Scala supports polymorphism, ScalaCheck does not test this [22].
There is no QuickCheck implementation for Rebel. This is not interesting for Rebel because it is a declarative language, it does not execute. However, when testing the generated code it is interest-ing, which is what we do in this thesis. Since the generated system is written in Scala, ScalaCheck might be applicable for us. However, this would result in using ScalaCheck as a black box, imple-menting the random functionality ourselves makes sure we know what is going on. Thus resulting in a white box implementation for our test framework.We can also modify it to our needs when we want to improve our test suite. One of the things that we want to improve is to generate values under certain conditions. For example, if a property only holds under a certain condition, the chance that random values satisfy the condition can be very low. Resulting in a test case that wouldn’t do anything most of the time. An example of such a property is Symmetry ( x = y =⇒ y = x ). Additionally, we might also need to slightly interact with other components, such as the messaging layer, that the generated system uses. This could make the implementation more complicated when using ScalaCheck due to the format of a property test when using ScalaCheck.
2.4
Terminology
In this thesis there are some levels of abstraction, the terminology used throughout this theses can cause confusion. Words like “specification”, “properties” and “tests” generally can have a different meaning depending on the context. In this section, we describe the confusing terminologies and abstractions in detail.
Specification
In this thesis we use the word “specification” exclusively to indicate banking products described in the Rebel language. This includes the state machines (life cycle), pre and post-conditions and logical invariants.
Properties
We use the word “property” to describe semantic properties of the Rebel language. The set of properties we introduce in this thesis can be seen as a partial “specification” of the semantics of Rebel, but we do not use this word to avoid confusion. We stick with “properties”.
Implicative property
A “property” that uses the implication ( =⇒ ) operator in its definition. Generator(s)
The generator(s) that can be used to generate a system based on a “specification”. When using the term “the generator”, we refer to the Scala/Akka generator which is the only generator that we use throughout this thesis.
Test framework
The test framework that was developed during this thesis. Which builds the specification, generates a system from the specification by using the generator, generates the test suite and runs the test suite against the generated system.
Tests
The generated tests by the “test framework”, intended to check whether a certain “property” holds.
Test suite
The collection of generated tests by the “test framework”, along with its configurations which can be run to test the generated system. Note that the test suite initially doesn’t exist. Instead, it is being generated and added to “the generated system” when we run the “test framework”. Events
The event definitions in a “specification”, these define the transitions between the states. Scala Build Tool (SBT)
The tool that is used to compile and run the “generated system” and the “test suite”. Concrete Syntax Tree (CST)
A “specification” can be checked and built. This results in a “CST” containing the data of the specification. This “CST” is being used by the toolchain of Rebel and by the “test framework”.
Properties of Rebel
Rebel introduces custom types, like IBAN, Percentage and Money and allows operations on those [1]. But what are the expected properties of the generator? In this chapter, we will try to answer the first research question:
RQ 1: Which properties are expected to hold on the semantics of the generated code?
To answer this question, we first describe a way how we can determine the properties. Followed by the property definitions that will be used throughout this thesis, with a motivation why these properties are expected to hold. The properties that are defined in this chapter will be used in our experiments (Chapter 5, Chapter 6andChapter 7).
3.1
Determining the properties
Currently, there are no definitions available of which properties are expected in Rebel. Due to the missing definitions of these types, it means that we first have to define what the expected properties on these types are and substantiate these. Only then we can determine whether the generator is working as expected with these properties.
There are many operations available among the available types in Rebel. We are not able to de-fine all the properties that exist in the Rebel language, because this is not effective for the scope of this thesis. Our goal is to show that this approach works. Next, one can write all the relevant properties down and use the test framework to test these properties automatically. During this thesis we will focus on the Money type, considering this is the most important type for a bank and has the highest priority to be implemented correctly.
For types like Integer, the axioms of algebra can be used as inspiration to determine whether the implementation is correct. An Integer in Rebel is most likely translated to an Integer in the gener-ated system too, with perhaps the expectation that these have the same properties in Scala. However, it is not possible to rely on the Integer definition of a specific language. Because another generator might generate a system in another language or might implement it differently in the same language. Would that mean that the properties of that other language should now hold on the Integer type? Well, as this is not defined for Rebel, this is unknown. In this chapter, we will define properties that are expected to hold when using the Money type in Rebel. The properties that we define are based on the known axioms in algebra [24,25,26]. We provide an explanation of why a certain property is expected in Rebel.
The Money type can be seen as a currency with an amount value. The amount of a Money value can have multiple decimals depending on the currency. Thus, the amount can be seen as a floating number. Does this mean that it inherits the computational properties of Floating-Point Arithmetic, as defined in the IEEE standards 754 or 854? Since the Rebel is intended to be a formal specification
language for banking products, we don’t expect that the described problems with this arithmetic are intended to exist on the Money type. Considering high volume flows within a bank in terms of Money, using the Floating-Point Arithmetic properties can result in the known precision, overflow and underflow errors as described in [27]. Such errors could be avoided by using the Money type. The author of [28] also describes that the intention of the Money type is to avoid this:
“You should absolutely avoid any kind of floating point type, as that will introduce the kind of rounding problems that Money is intended to avoid.” – Martin Fowler [28]
In [28] the operations that can be done with the Money type are described, which are: +, -, *, allocate, <, >, ≤, ≥ and =. Where the allocate method is used instead of the division (/) operation. This is due to the division problem, requiring a number to be rounded off at a certain time. For example, when splitting 1 EUR by 3, each part would be 33 cents with the consensus of two decimals. However, what is done with the last cent that is left? This is the problem that is being solved by the allocate method, representing the ratio in which the rest amount should be allocated (where the last cent would go to in this case) [28]. The allocate method is a thing that Rebel does not have, instead, it just allows the use of the division operator. Because of this, we expect the division problem to occur while running the test framework.
The amount of a Money value is often rounded when it is being represented to the user, as it could have many decimals. The representation of the Money value is up to the business on how this is done, as there are multiple factors influencing this. Instead, we only focus on the internal value that is used when operating with the Money type.
It is unsupported to use operations on Money values when using values that are of different cur-rencies. This could be done by taking the exchange rates into account, as described in [28]. However, this is not implemented in Rebel yet and thus it is unsupported to use operations on it when using different currencies.
When using equality both the currency and the amount are taken into account, which should be equal for both variables. In case of different currencies, it is not considered equal, even if the amount would be equal.
We say that the amount of a Money value in Rebel should hold the exact value as if we would calculate the same expression by ourselves. The representation of the Money value to the customer is up to the business.
3.2
Property definitions
The properties that we define are based on the known axioms in algebra, although not every property is relevant or holds for each type in Rebel.
For example, it isn’t possible to multiply two Money types with each other in order to support the multiplicative property. It is unknown to which dimension “Money squared” belongs, this does not occur in the financial domain. Because of this, detecting this prevents such mistakes. When this is accidentally defined in the Rebel specification, the type checker would detect and report this as an error. Division has a similar case, as something cannot be divided a Money type.
Instead we can use these operations with Money in combination with other types, such as Inte-ger and Percentage. The type checker for Rebel will be used to determine these combinations. Due to this, one property definition can lead to different possible combinations which can be used. This results in multiple property definitions when using different types, for each definition a unique name is being defined to identify each separate property definition.
equal-ity and inequalequal-ity”. In the following sections we describe these categories in detail along with the properties in the category. These property definitions will be used throughout the thesis. Addition-ally we will describe some property definitions that were added to this list of properties in the third experiment (Chapter 7).
3.2.1
Field properties
The field properties define the behaviour of addition, multiplication, subtraction and division on rational and real numbers1. As described in [26], such operations can be used to determine the sum
and the product of such values:
“Along with the set R of real numbers we assume the existence of two operations called addition and multiplication, such that for every pair of real numbers x and y we can form the sum of x and y, which is another real number denoted by x + y, and the product of x and y, denoted by xy or by x . y.” – Tom M. Apostol [26]
These operations are also available when using the Money type in Rebel and are expected to apply these operations on the amount value2. However, not every combination is possible in Rebel when it comes to using these operations. This is because Rebel does not support certain combinations. We define the following properties in this category:
Associativity
Formula Property name Variable (Type)
(x + y) + z = x + (y + z) associativeAddition x : Money y : Money z : Money (x · y) · z = x · (y · z) associativeMultiplicationInteger1 x : Integer y : Integer z : Money (x · y) · z = x · (y · z) associativeMultiplicationInteger2 x : Money y : Integer z : Integer (x · y) · z = x · (y · z) associativeMultiplicationPercentage1 x : Money y : Percentage z : Integer (x · y) · z = x · (y · z) associativeMultiplicationpercentage2 x : Integer y : Money z : Percentage
Table 3.1: Associativity when using Money
The law of associativity is known on addition and multiplication [24]. It defines that the order in which certain operations are done does not affect the result of the whole expression. InTable 3.1we define possible combinations for this property, which are accepted by the type checker of Rebel.
1https://en.wikipedia.org/wiki/Field_(mathematics)
2We do not claim that the Money type in Rebel is a field. As it does not support each described operation when
Non-associativity
Formula Property name Variable (Type)
(x − y) − z 6= x − (y − z) nonassociativity x : Money y : Money z : Money
Table 3.2: Non-associativity when using Money
In contrast to associativity (Associativity), non-associativity describes that the order of the arguments does affect the result of the whole expression. As we can see in Table 3.2 subtraction is a relation where this property holds. An exception to this would be when each argument is zero.
Commutativity
Formula Property name Variable (Type)
x + y = y + x commutativeAddition x : Money y : Money x · y = y · x commutativeMultiplicationInteger1 x : Integer y : Money x · y = y · x commutativeMultiplicationInteger2 x : Money y : Integer x · y = y · x commutativeMultiplicationPercentage1 x : Percentage y : Money x · y = y · x commutativeMultiplicationpercentage2 x : Money y : Percentage
Table 3.3: Commutativity when using Money
These properties are based on the commutative law [24]. The result of an addition or multiplication does not vary when swapping the input variables. Because of the Money type, we can only do addition on Money values with other Money values. As described in earlier, it is not possible to multiply two Money variables with each other, but it is possible to multiply it by an Integer or Percentage. Also in this case, the order shouldn’t matter if we would put the Money value as the first input parameter to multiplication or the other way around.
Anticommutativity
Formula Property name Variable (Type) x − y = −(y − x) anticommutativity x : Money
y : Money
Table 3.4: Anticommutativity when using Money
Commutativity described the commutative properties. Note that the properties in Commutativity
only use addition and multiplication. Subtraction is an operation that is anticommutative as swapping the order of the two arguments is negates the result. The anticommutative property thus negates the
result of swapping the two arguments, intending to result in the actual value again, as shown in
Table 3.4.
Identity
Formula Property name Variable (Type) x + 0 = x additiveIdentity1 x : Money 0 + x = x additiveIdentity2 x : Money x · 1 = x multiplicativeIdentity1 x : Money 1 · x = x multiplicativeIdentity2 x : Money
Table 3.5: Identity when using Money
The identity relation describes a function that returns the same value as the value that was given as input. For additivity, this entails the addition of zero to the input value and for multiplicativity, this entails multiplying the value by 1. Also, the Commutativityproperty holds here, as the order does not matter in which this function is applied. Since it is not possible to just add 0 to a Money value, the 0 showed inTable 3.5must be defined in a Money format. Thus it must have the same currency as the parameter, with the amount of 0. The Integer type can be used for multiplication.
Inverse
Formula Property name Variable (Type) x + (−x) = 0 additiveInverse1 x : Money (−x) + x = 0 additiveInverse2 x : Money
Table 3.6: Inverse when using Money
The inverse relation describes (for additivity) that using addition with the input parameter and the negative of the input parameter, results in the value zero. Note that the operation is used on the Money type, so the expected value is 0 with the same currency as the currency of the input parameter. Although the inverse relation could also be used with multiplication and division (defined as x · (1/x) = 0 ), it is not possible to use this definition in our case. As we cannot divide something by a Money type, which is why we only define the inverse relation using addition.
Distributivity
Formula Property name Variable (Type)
x · (y + z) = x · y + x · z distributiveInteger1 x : Money y : Integer z : Integer (y + z) · x = y · x + z · x distributiveInteger2 x : Integer y : Money z : Money x · (y + z) = x · y + x · z distributivePercentage1 x : Percentage y : Money z : Money (y + z) · x = y · x + z · x distributivePercentage2 x : Percentage y : Money z : Money
Table 3.7: Distributivity when using Money
The law of distributivity is another well-known law [24]. UnlikeAssociativity, the order does matter here when using different operations. These operations can be used on Money and since we can see Money as a number, this property is also expected on this type. Remember that it is not possible to multiply Money types with each other. Thus the variable types are an important part of these properties, inTable 3.7property definitions are shown based on this property.
3.2.2
Properties of equality and inequality
The properties in this category are mainly using the equality (=, 6=) and inequality (<, >, ≤, ≥) operators in their definitions. For the property names we abbreviate these operations with the naming LT, GT, LET and GET respectively. The following property definitions belong to this category:
Property of Zero
Formula Property name Variable (Type) x · 0 = 0 multiplicativeZeroProperty1 x : Money 0 · x = 0 multiplicativeZeroProperty2 x : Money
Table 3.8: Property of Zero when using Money
The property of zero on multiplication states that if something is multiplied by zero, the result will always be zero. Since Rebel allows the use of multiplication on the Money type, it’s possible to multiply it by 0. The order in which this happens shouldn’t matter. Since the value of a Money variable is based on a decimal number, this property states that the value will be exactly 0 (or 0.00 in the representation of a Money value). It should not contain any decimals that would make the amount bigger than zero.
Reflexivity
Formula Property name Variable (Type) x = x reflexiveEquality x : Money x ≤ x reflexiveInequalityLET x : Money x ≥ x reflexiveInequalityGET x : Money
Table 3.9: Reflexivity when using Money
The reflexive property is defined as a relation of a type with itself [25]. An instance of type Money should be equal to itself. The inequality relations smaller or equal to and greater or equal to should hold too, as we can compare Money variables.
Symmetry
Formula Property name Variable (Type) x = y =⇒ y = x symmetric x : Money
y : Money
Table 3.10: Symmetry when using Money
Reflexivitydescribed relations on the same variable of the Money type. When two different variables are used, the order should not matter and thus it should work in both ways. Which is known as the symmetric property [25].
Antisymmetry
Formula Property name Variable (Type)
x ≤ y ∧ y ≤ x =⇒ x = y antisymmetryLET x : Money y : Money x ≥ y ∧ y ≥ x =⇒ x = y antisymmetryGET x : Money y : Money
Table 3.11: Antisymmetry when using Money
The antisymmetric relation describes that whenever there is a relation from x to y and a relation from y to x, then x and y should be equal. The lower or equal than (≤) and greater or equal than (≥) relations fit in this category, as shown inTable 3.11. This antisymmetric relation is also expected to hold, as the Money type supports these operations.
Transitivity
Formula Property name Variable (Type)
x = y ∧ y = z =⇒ x = z transitiveEquality x : Money y : Money z : Money x < y ∧ y < z =⇒ x < z transitiveInequalityLT x : Money y : Money z : Money x > y ∧ y > z =⇒ x > z transitiveInequalityGT x : Money y : Money z : Money x ≤ y ∧ y ≤ z =⇒ x ≤ z transitiveInequalityLET x : Money y : Money z : Money x ≥ y ∧ y ≥ z =⇒ x ≥ z transitiveInequalityGET x : Money y : Money z : Money
Table 3.12: Transitivity when using Money
Operations can be done on the Money types. The transitive properties [25] on the (in)equality operators should still hold on the Money type as we can compare the Money values.
Additivity
Formula Property name Variable (Type)
x = y =⇒ x + z = y + z additiveEquality x : Money y : Money z : Money x = y ∧ z = a =⇒ x + z = y + a additiveEquality4params x : Money y : Money z : Money a : Money
Table 3.13: Additivity when using Money
Addition was earlier mentioned for theCommutativityandAssociativity properties. The properties mentioned here extend these by defining properties that are true when the input values are equal. When using the addition operator such that the resulting values on both sides remain the same, as shown inTable 3.10, it should not break the equality property on the resulting values.
Division
Formula Property name Variable (Type)
x · y = z =⇒ x = z/y division1 x : Money y : Integer z : Money x = z · y =⇒ x/y = z division2 x : Money y : Integer z : Money
Table 3.14: Division when using Money
When using division with the Money type, it is not possible to use a Money value as the denominator. A Money type can be dived by an Integer, thus we can define the division properties by using both the Money and Integer type. Note that the denominator cannot be zero, as division by zero is not possible.
Trichotomy
Formula Property name Variable (Type)
x < y ∨ x = y ∨ x > y trichotomy x : Money y : Money
Table 3.15: Trichotomy when using Money
The law of trichotomy defines that for every pair of arbitrary real numbers, exactly one of the relations <, =, > holds. We can define a property for this when using Money, as shown inTable 3.15.
3.2.3
Additional properties of equality and inequality
For the third experiment (Chapter 7), the property definitions for division have been updated and additional properties have been added to the list of defined properties. These extra properties can be put in the category of equality and inequality properties and are using the implication ( =⇒ ) sign in its definitions.
Division
Formula Property name Variable (Type)
round(x · y) = round(z) =⇒ round(x) = round(z/y) divisionEquality1 x : Money y : Integer z : Money round(x) = round(z · y) =⇒ round(x/y) = round(z) divisionEquality2 x : Money y : Integer z : Money x = y ∧ z 6= 0 =⇒ x/z = y/z divisionEquality3 x : Money y : Money z : Integer x < y ∧ z > 0 =⇒ x/z < y/z divisionInequalityLT1 x : Money y : Money z : Integer x < y ∧ z < 0 =⇒ x/z > y/z divisionInequalityLT2 x : Money y : Money z : Integer x > y ∧ z > 0 =⇒ x/z > y/z divisionInequalityGT1 x : Money y : Money z : Integer x > y ∧ z < 0 =⇒ x/z < y/z divisionInequalityGT2 x : Money y : Money z : Integer
Table 3.16: Updated definition of Division when using Money
The initial property definitions for Division did not take the division problem into account and thus had to be updated. The updated definitions are using the round() method to prevent the occurrence of the division problem. The round() method rounds the value to 4 decimals and uses the “HALF UP” rounding technique, this is described in more detail inChapter 7. In addition to this modification, more property definitions have been added for division, as shown inTable 3.16.
Additivity
Formula Property name Variable (Type)
x = y =⇒ x + z = y + z additiveEquality x : Money y : Money z : Money x = y ∧ z = a =⇒ x + z = y + a additiveEquality4params x : Money y : Money z : Money a : Money x < y =⇒ x + z < y + z additiveInequalityLT x : Money y : Money z : Money x > y =⇒ x + z > y + z additiveInequalityGT x : Money y : Money z : Money
Table 3.17: Additivity when using Money
For additivity, additional property definitions are added that use inequality, as shown inTable 3.17. The existing ones (described in the first 2 rows) only used equality in its definitions.
Subtraction
Formula Property name Variable (Type)
x = y =⇒ x − z = y − z subtractiveEquality x : Money y : Money z : Money x < y =⇒ x − z < y − z subtractiveInequalityLT x : Money y : Money z : Money x > y =⇒ x − z > y − z subtractiveInequalityGT x : Money y : Money z : Money
Table 3.18: Subtraction when using Money
The property definitions concerning subtraction are kind of similar to those of Additivity. It uses subtraction instead of addition in its definition. Nevertheless, to check whether the properties holds for subtraction we defined these properties as shown inTable 3.18.
Multiplication
Formula Property name Variable (Type)
x = y =⇒ x · z = y · z multiplicativeEquality x : Money y : Money z : Integer x < y ∧ z > 0 =⇒ x · z < y · z multiplicativeInequalityLT1 x : Money y : Money z : Integer x < y ∧ z < 0 =⇒ x · z > y · z multiplicativeInequalityLT2 x : Money y : Money z : Integer x > y ∧ z > 0 =⇒ x · z > y · z multiplicativeInequalityGT1 x : Money y : Money z : Integer x > y ∧ z < 0 =⇒ x · z < y · z multiplicativeInequalityGT2 x : Money y : Money z : Integer
Table 3.19: Multiplication when using Money
The property definitions for multiplication are looking similar to those of Division. The property definitions described in Table 3.19 are different in that these use the multiplication operator along with the inequality symbols.
3.3
Conclusion
In this chapter, we focused on the properties when using the Money type in Rebel. The research question lead as follows: Which properties are expected to hold on the semantics of the generated code?
Since there was no definition available of the types in Rebel, it was required to define the proper-ties in detail. Describing what the expected behaviour is when operating with the types and which operations are allowed on each type. In this chapter, we have defined a set of properties that are expected to hold with Rebel. These property definitions will be used in the experiments to test the generator.
The properties are based on the axioms in algebra, but the requirements of Rebel have been taken into account when it comes to using certain operations with certain types. The defined properties are separated into 2 categories: “Field properties” and “Properties of equality and inequality”.
The properties that are defined in this chapter will be used to test the generator. However, it is not true that there are no bugs in the generator in case every property holds. It does mean that there were no errors found in the properties that are being checked. Many properties can be defined within the Rebel language, the properties defined in this chapter are certainly not all the properties that exist in Rebel.
3.4
Threats to validity
Lack of propertiesWe defined a set of properties that are expected to hold in Rebel. This can be seen as an incomplete set, as many more properties can hold and can be expected when using Rebel. The set of properties we defined are aimed to cover many operations when using the Money type, as this is considered the most important type for a bank.
We have used the axioms of algebra as inspiration for the property definitions in this chapter. We did not specify Rebel, as we have not defined all the operators and all the important properties of it. This leads to a threat of validity for the next chapters in that this affects the answer to our third research question: What kind of bugs can be found using this approach and how many? A complete list of property definitions in Rebel could result in more bugs, and thus affect the answer to this research question. For this, a profound study should be done on how Rebel works exactly, describing all the expected properties in Rebel.
Invalid definitions
We defined some properties that are expected to hold on Rebel types, specifically when using the Money type. We already discussed the lack of properties, but it could be the case that some of our definitions are invalid. We also assume that these properties do hold on Rebel types.
Some might disagree with certain properties that we have defined, which leaves this as a threat to the first research question: Which properties are expected to hold on the semantics of the gen-erated code? But when this is the case, the properties can be updated when needed. When we encounter a bug in our experiments by using the properties in the following chapters, the designers or implementers of Rebel might disagree with the fact that it’s a bug (a false-positive). However, this issue is then getting known by using this approach, which does not mark this as a major threat. Minimum number properties
We defined many properties in this chapter. This might lead to questions like: is this the minimal (or optimal) number of property definitions in order to detect the bugs found in the experiments? Because having a minimal (or optimal) set of properties would mean that there are less tests needed to test the same thing, resulting in that less time is required to run the test framework. The answer to this question is no, this is out of scope for this thesis. This leads to a threat to answering our third research question: What kind of bugs can be found using this approach and how many?
Some of the defined properties might be overlapping each other. It might be possible to lower the number of properties that have to be defined. However, the purpose of each property on what it tests is different. So one should be careful when determining some property unnecessary. For this thesis, we used known axioms of algebra as inspiration and defined some properties for Rebel.
Test mechanics
In order to check whether the defined properties inChapter 3hold when using the generator, we need to determine how the test framework should work. In this chapter we will try to answer the following research question:
RQ 2: How can we test each property as automatically as possible to find bugs in the generator? We use property-based testing as testing technique. The aim of this project is to test the imple-mentation of the generator and trying to find, yet unknown, bugs in it. Unfortunately, we cannot test the properties right away on the generator, but we aim to test the properties as automatic as possible. To check the generator, we use the system that it generates. In order to do that, a valid Rebel specification is required. In this chapter, we describe how the test framework is setup such that it can automatically check whether the defined properties hold when using the generator.
4.1
The test framework
A Rebel specification can be created with the property definitions. Which can then be used to generate the test cases. The collection of resulting test cases is the content of the test suite, which we can be run against the generated system. We can divide this process into different phases. The goal of the test framework is to combine most of the required phases such that each defined property is being checked as automatic as possible. An overview of the phases and the test framework is shown in
Figure 4.1.
Figure 4.1: Overview of the test framework and the phases The phases are defined as follows:
2. Check & build 3. Generate system 4. Generate test suite 5. Run test suite
We will describe each phase in detail in the next sections. Additionally, we will define some evaluation criteria which will be used to evaluate the test framework. The Reflexitivity property will be used to demonstrate each phase. More specifically: the case of Reflexivity when using equality, called ReflexiveEquality throughout this thesis. The definition of ReflexiveEquality is shown inTable 4.1.
Formula Variable Type
x = x x Money
Table 4.1: Property definition of ReflexiveEquality
4.1.1
Create specification
The generator requires a consistent Rebel specification in order to generate a system. Coincidentally, the Rebel syntax is strong enough to write down the defined properties in Rebel. This opens the possibility to create a Rebel specification containing the property definitions and to reuse the gener-ator when generating the tests. This way, we can be sure that we check exactly the same code as which would be used when using a real Rebel specification. A real specification means, for example, a specific bank account specification written in Rebel. This is a convenient choice, because the generator is being checked this way and it opens the ability to use the same approach with different generators. We can use the event blocks in Rebel to write down the properties. An event describes a transi-tion from one state to another and accepts parameters. Additransi-tionally, it can have pre- and post conditions, where the post conditions state what happens when the transaction is being executed. In
Listing 4.3the event definition for the ReflexiveEquality property written in Rebel is shown.
1 eventreflexiveEquality(x: Money) { 2 postconditions{
3 new this.result == ( x == x );
4 }
5 }
Listing 4.3: The event definition for the ReflexiveEquality property.
The event name and the parameters are used to generate a test case from this event definition. To check whether the property was fulfilled given a certain set of parameters, we store the result in a data field called result1. The test suite uses the value of this field, to determine the result. In case
the result value is false during testing, a bug has been found.
Besides the event definition, we need to write the actual Rebel specification to be able to gener-ate a system from it. The specification describes the fields, the events it uses and the life cycle of the state machine. Since we are only interested in testing the events, we can hold the specification itself to a minimum. The life cycle consists of 2 states, the initial and final state. The transition between these states is the event we defined, ReflexiveEquality. InListing 4.4a specification used for one property is shown. In the case of multiple properties, we can add these to the events block. In the life cycle, we can comma separate the transitions.
