Evaluation of Test Results

The evaluation of the results greatly depends on the support provided by the tool. Spec Explorer gives a detailed trace of all the actions that was executed as a part of the test, which makes the analysis and the follow-up activities easier. In this phase we look for any false negatives and false positives (how to find it is described in Chapter 3). If they are found, we correct them in the model. In case of true negative, we make a problem report for the SUT.

Chapter 3

Approach

In this chapter, we discuss the methodology adopted in solving the problems described in Section 1.1.

Initially all subproblems were carried out independently. We subsequently combine the basic model (model without data) of the subsystem with its data model. To develop the basic model and the data model of the subsystem, we follow an iterative and incremental approach. Figure 3.1 shows an iterative and incremental model development process. To develop the generalized tool, that support test case generation for applications having dependencies among parameters, we follow an incremental approach.

Figure 3.1: Iterative & incremental development model

We initially consider a basic set of requirements in the specification, design a model to capture those requirements, generate a test suite from the model and evaluate the results after executing the test suite. This cycle continues untill all the requirements have been covered correctly, i.e., all requirements are reflected in the model. A detailed description of the steps involved are given below:

• Construction of the model:

14 Chapter 3. Approach

Construction of the model is the most important and crucial step of MBT. A clear understanding of the SUT and its specification is crucial to designing a correct model. Incorrect modeling may lead to false positive or false negative (described later).

• Analysis of the model:

The SUT’s expected behavior can be viewed in the form of a graph. The graph can be used to analyze the correctness of the model. If any inconsistency with respect to the specification is detected in the graph, the model needs to be corrected before we can generate test cases from it.

• Generate test suite from the model:

Once we are able to construct the model, the tool can generate a test suite of test cases from it.

• Execute and evaluate tests: The generated test suite can automatically run against the implementation and a verdict is generated together with test run de-tails. Based on the correct or the incorrect implementation of both the model and the SUT, the test results can fall into four categories [11]:

– A false negative: The result of incomplete or incorrect modeling. Since, the implementation has correctly implemented the requirements. This give rises to a contradiction. Hence, it is easy to detect the modeling error.

– A false positive: The result of both incorrect modeling and incorrect imple-mentation of requirements by SUT. There is no contradiction. These errors are hard to find out.

– A true negative: The modeling is correct. However, there is a fault in the implementation. This is a desired outcome of the testing process.

– A true positive: Both model and implementation have correctly imple-mented the requirements. Test case passes. This is a desired outcome of the testing process

During the evaluation phase, our task is to look for every false negative (this is not hard to find and can be easily found out by scrutinizing the trace of failed test cases) and try to be careful with false positives. One of the recommendations to deal with false positives is to verify the model generated in the process of MBT with the requirement engineer. As the requirement engineer has deep insight of the system’s requirements, he can easily catch any inconsistency in the model which may lead to a false positive. We, start from a small and less complex model to make our task easier. Only when we have resolved all false negatives and verified the model for false positives, we should go for adding new requirements to the

15

model.

For the construction of data model, we will go for an incremental approach, adding new functionalities on the top of the old and verified model.

Chapter 4

Basic Model and Adapter

This chapter gives an introduction to the state behavior of the SUT and how we can model it within our MBT tool. We discuss the level of abstraction used for modeling and the motivation behind it. We also mention the advantages of using an adaptor (especially with respect to raising the abstraction level) within the current MBT framework. Finally we show how to generate and execute test cases from the model.

4.1 Introduction

The basic model aims to capture all the states and transitions of the IDS. The subsystem contains a finite set of commands, which provides a means to interact with the it. The subsystem may reside in a number of states during its execution.

Only a certain number of commands (actions) can be executed from a particular state, which may or may not cause a transition to another state according to its specification. However, the subsystem is expected to issue an appropriate message for any command requested from any state.

The infrastructure for MBT uses instance-based methods, i.e., there can be sev-eral instances of the infrastructure at a time. Therefore, while making the model we have to make sure that the entire model executes on one instance of the in-frastructure, otherwise the results will be unexpected as different instances of the subsystem can be in different states. The graph obtained as the result of explo-ration of a model in Spec Explorer (SE) reflects this phenomenon by explicitly enumerating each instance of the adapter. Each action in the graph is preceded