Amir Cohen 10686835 10686835@student.uva.nl June 13, 2016 Internal supervisor: Paul Griffioen, CWI External supervisor: Femke van Raamsdonk, VU
Contents
Abstract 1. Introduction 1.1. Background 1.2. The predicate switching mechanism 2. Research question 3. Method 4. The Original Studies 4.1. Contribution of the Original Study 4.2. Implementation 4.3 Limitations 4.4 Experimentation and Evaluation of the Results 5. The Replication Study 5.1 The Predicate Switching tool 5.1.1. The Role of the Rascal Language 5.1.2 Design of the Predicate Switching Tool 5.1.3. Implementation 5.2. Comparison between our Rascal tool and the tool built by Zhang et al. 6. Experimentation 6.1 The testing suite 6.2. Faults injection 7. Results 8. Related Approaches 9. Threats to validity 10. Future Work 11. Conclusions 12. Bibliography
Fault locating by predicate switching in Rascal
Abstract
The predicate switching method is a heuristic way for locating faults, which was presented by Zhang et. al. In this method the outcome of executed branching predicate instances is automatically being switched during reexecutions of a failing program, looking for a single critical predicate instance whose switching results in a successful execution. By examining this predicate instance, the cause of the bug can then be identified. In this work we analyse the effectiveness of predicate switching. To this end we build a predicate switching tool in Rascal, usings Rascal’s M3 meta model for sourcecode analysis. We then run our tool on a testing suite of 18 Java programs (in comparison to Zhang et. al who used C programs) with injected faults that we have composed, and we present an analysis of the observations. We found that predicate switching is unable to locate faults if multiple instances of executed predicates are infected by them. We provide an analysis of the effect of predicate switching in these cases in order to explain why it fails to locate them. Our work serves also as a demonstration of using Rascal to implement dynamic analysis tools. As Rascal’s M3 model is language independent, the implementation could be adjusted to support other languages than Java or other language elements.
Keywords: fault localisation, program slice , branching predicate, predicate trace, predicate instance, critical predicate, predicate switching, Rascal
1. Introduction
1.1. Background
During the process of software development, faults are likely to occur. Some faults can directly be identified with static analysis by, for example, the parser or the compiler. Other faults that can not be found with static analysis, might only be identified with dynamic analysis during the program’s execution.
Such faults that cannot be identified in a static way, can only become visible at a certain point of the execution. The cause of the observed fault however might arise at a point in the execution earlier to the point at which it becomes observable. The propagation of the fault to the rest of the program is referred to as the infection chain [4]. Once a fault is observed at some point, it is the programmer’s mission to trace back in the execution to find the origin of the infection chain. This process is often done by manual interventions during the program´s execution, and examining different hypotheses in order to explore and isolate the possible cause of a fault. This process can be very time consuming.
Different search based approaches have been proposed during the years [410] that aim at automating and accelerating this process. These approaches, however, are often based on exhaustive exploration of enormous search spaces of all combinations of possible states. This makes it often impossible to compute both sound and reasonably precise (i.e., nearly complete) analysis for such problems.
Considering these limitations of the two approaches to debugging manual versus automated, the option of combining the search power of the computer with the acquaintance of the programmer with the system could be interesting to investigate. Zhang et. al [1] propose in their work a heuristical approach for locating faults which is based on automated predicate switching. They argue that this approach is effective at efficiently locating the predicate in a program which is related to the fault and thus may save time in finding the origin of the fault. They present results that support this claim, and also report some cases in which the predicate switching mechanism fails. However, they do not provide an analysis to explain what makes it fail in such cases.
1.2. The predicate switching mechanism
A branching predicate, (or shortly predicate), is a boolean expression whose evaluation is the condition for execution of a block of instructions. The basic idea of the predicate switching method is that switching a single predicate might in specific cases remove the influence of some earlier emerging fault and thus result in a successful execution. The executed branching predicates are systematically switched separately during reexecutions of a given failing program. Once a successful execution is generated, the programmer can directly examine the predicate whose switch caused the successful execution. This can spare debugging time by helping to locate the origin of the fault faster while benefitting from the programme’s contextual knowledge.
The predicate switching mechanism starts by extracting a predicate trace of a failing program execution. The predicate trace of an execution is the list of all the predicates that have been visited during a specific run of a program, in the order of execution.
In the sequel, for each predicate instance (a specific iteration of a predicate in a specific location in the execution trace of a program) a corresponding execution instance of the program is generated with the only difference that this particular predicate instance is switched. The rest of the program is executed just like the original failing execution instance.
The order of the predicate instances to be switched is from the last predicate instance visited to the first. This process is being iterated until, ideally, the initially observed faulty behavior disappears.
The predicate instance whose switching prevents the manifestation of the fault and thus results in a successful execution of the program is called the critical predicate instance. The critical predicate instance is seen as an indication for the location related to the fault and, once identified, it is reported to the user and the search terminates.
Example 1 illustrates the dynamics of the predicate switching mechanism. Consider the hypothetical function from this example which we use to illustrate a fault that can be located by predicate switching. This example is characterised by the following features:
Its control flow has multiple branching predicates (lines 4, 5 and 9). A computation that influences the control flow (lines 23).
An assertion (line 12) that checks if the specifications are satisfied (in this example the returned integer x is expected to have a nonnegative value).
A fault seeded (line 2) that makes the assertion fail (namely, we changed the assignment statement from z = y*x to z = yx)
Assume that the specification that this program has to meet is that, given any integer input, the value it returns has to be a nonnegative integer value. When this specification is met the program is considered to be correct. The assertion at line 12 tests that this specification is met by controlling that right before the exit point of the program the value of the variable x that is about to be returned is indeed not negative.
In the correct version of this program (namely when at line 2 the integer variable z gets the value of x*y assigned), given any integer input, the assertion is evaluated to true, which means that the program always returns a nonnegative value. Take for instance the input x=(5) and y=10. Without the fault at line 2, the branching predicates at lines 9 and 10 should have been evaluated to true (z<=0, u>0), which would have led to a true evaluation of the assertion (x = 50). The fault in line 2 however (z = yx) makes the function execute lines 5 and 6 instead of 9 and 10 (z>0, u>0). As a result, the assertion fails (x=(10).
Traditionally, a programmer will review and debug the code manually to determine the first place where a suspicious statement related to the assertion does something undesired. In this example manually reviewing the code to detect the location of the fault is not necessarily an extremely tedious task. But in real life settings where programs are bigger this could be very time consuming. This is the
place where an automated or
semiautomated search approach could come to hand as a useful way to reduce debugging time. The predicate switching approach suggests automatically switching the outcome of each predicate that is 1 public int returnNonNegative(int x, int y){ 2 int z = yx; // ←BUG! should be z=y*x 3 int u = (x); 4 if (z>0) { 5 if (u>0) {x = (y);} 6 else {x = x+1;} 7 } 8 else { 9 if (u>0) {x = x*(10);} 10 else {x = y;} 11 } 12 assert(x>=0); 13 return x; 14 } Figure 1 an example of a faulty program to be located by predicate switching
visited in a given run, one predicate at a time, and in this manner forcibly making the execution take a different path.
Back to the example in figure 1, the predicate instance trace of the faulty execution of the code given the input x = (5) and y = 10 is {4,5}. The numbers 4 and 5 stand for the lines of the visited branching predicate instances (z>0 at line 4 and u>0 at line 5). These predicate instances are being switched, starting from the last predicate instance visited. Specifically in this case, switching the outcome of the predicate at line 5 (z>0) as if it was (!(z>0)) makes the program continue at that point from line 4 to line 8, instead of to line 5. As u is assigned the value 5, the program execution continues to line 9, which assigns to the variable x the value 50, which is a nonnegative number.
Note that switching the predicate at line 4 returns the execution to the path it should have taken if the fault at line 2 had not existed. This results in a correct program with respect to the specification that the returned value should be nonnegative. The correctness of the program is then controlled by the assertion at line 12, which becomes correct, as x indeed has a positive value at that point.
In this case, the predicate switching method helped the execution in this case to “recover” or avoid the fault and this way makes it possible to identify the origin of the fault as one that is related to the predicate at line 5 (z>0), and thus specifically to the variable z in this case.
2. Research question
In this work we implement a tool for locating faults through automated predicate switching. For our implementation we used Rascal, which is a meta programming language for analysis and transformation of programs. Our work serves as a demonstration of using Rascal for building dynamic analysis tools and automated debugging techniques.
Zhang et al. report that in some situations faults may not be found by the predicate switching tool. However, an analysis of the reasons why in some situations faults can be located and in others not, was missing.
As a following step, we had initially explored different factors that can determine the effectiveness of predicate switching. We restricted our attention to faults that only influence the condition of ifstatements (the branching predicates). This restriction makes it possible to completely compensate the fault’s infection on the program by switching a predicate outcome. On the other hand, faults whose influence exceeds to parts of the program beyond only branching predicates can have effect on the entire program, which would be irreversible by predicate switching. However, Zhang et al. show that the code that influences branching predicates is usually only a small part of entire programs’ code. That means that predicate switching can only be effective in such restricted settings.
After further investigating the related literature [2,3] and analysing the predicate switching method, it appeared that a main factor that determines the effectiveness of the predicate switching tool is whether a fault influences merely a single predicate instance or multiple predicate instances. If a fault influences more than a single predicate instance it seems that it cannot located by predicate switching. We are interested in examining the influence of predicate switching in such cases by addressing the research question:
Why does the
predicate switching technique fail at locating faults whose infection chain infects
multiple predicate instances?
3. Method
In order to explain why predicate switching fails at locating faults whose infection chain diverges to multiple predicate instances we were interested in investigating the infection chain of the faults in such programs in comparison with programs in which faults are found by predicate switching.
To this end, we replicated the implementation of the predicate switching tool as described by Zhang et al (see section 5). Once we implemented the tool, we composed a testing suite that we used to experiment with the predicate switching tool on. The testing suite consists of 18 Java programs that we seeded with faults (see details in section 6.2). To fit the scope of our research the faults were seeded such that their influence is restricted merely to branching predicates. Besides, we were looking merely at predicates of ifstatements and left whileloops outside the scope of this work.
In our experiment we run the predicate switching tool on each program in our testing suite and then report the results. The next step in our research is analysing the effect of the predicate switching tool on these programs. We track the effect of the tool on each of these programs and use these observations to explain why it fails at locating faults whose infectionchain diverges to multiple predicates.
4. The Original Studies
In this section we describe the work that Zhang et. al performed. We briefly describe their contribution, then we present their implementation or their predicate switching tool. We then present a summary of their experiments and their results, followed by their conclusions.
4.1. Contribution of the Original Study
In their work, Zhang et. al. show how automated predicate switching helps finding the location of faults efficiently. They distinguish between what they refer to as “data part” as opposed to “selection part”. In short data part is the code slice that takes part in producing the output. Selection part, on the other hand, is the code slice that influences the choice of control flow (through branching predicates). They show that the size of the search space is being considerably decreased when limited to merely predicate instances.
4.2. Implementation
In order to implement their predicate switching tool, Zhang et. al used Valgrind, an instrumentation framework that works at binary level and then maps back to source code [14]. Valgrind’s kernel is a dynamic instrumenter which dynamically calls the instrumentation function provided by the user for a given basic block and executes the instrumented basic block instead of original one. In their implementation, a program is being instrumented in such a way that it surrenders the control to the tool at the moment that the execution reaches the predicate instance at hand, and the outcome of the predicate instance is being reversed by switching the two branch targets.
4.3 Limitations
Zhang et. al mention also some limitations of predicate switching. For instance, the report that it cannot locate a fault which is quite complex. They explain that overcoming the problems created by faulty code may not be possible by switching one predicate instance at a time and may require switching multiple predicate instances. They report also that predicate switching is highly unlikely to locate very significant faults such as, for instance, when some functionality is missing. However, they do not present any analysis of the reasons that cause faults to be overcome or not in these cases. They also did not describe examples that manifest this situation.
4.4 Experimentation and Evaluation of the Results
In their experiment Zhang et. al tested their implementation on 20 cases, out of which 15 pointed at a predicate instance that, once switched, made the program produce correct output or eliminated the cause of the program crash. In 10 out of these 15 cases in which a fault was eliminated by switching a predicate, the most recent instance of a predicate (namely, the first predicate checked which appeared last in the predicate trace) was the critical instance. In 4 other cases up to 10 predicates were examined until the critical one was found, and in one single case 143 predicates were switched before finding the critical predicate, and only the second predicate switch that produced correct output was actually related to the fault.
4.5 Conclusions
In their work Zhang et. al presented the idea of critical predicates and an efficient automated algorithm for locating critical predicates. They show that critical predicates very often are located in many real reported faulty programs, and provide valuable clues to the cause of the
failure and hence assist in fault location. They demonstrated that the mechanism of automated predicate switching can be effective for reducing the effort required for fault location. They show how dynamic program slicing can use the results of the predicate switching technique in order to point to the exact origin of a fault.
5. The Replication Study
In this part we describe the research we have conducted to identify factors that influence the effectiveness of the predicate switching approach to locate faults. To this end we have built a predicate switching tool using the Rascal framework [15]. Rascal is suitable for this goal as it enables doing both structural analysis and transformation of a program (see section 6.2 for a comparison of our implementation and the implementation of Zhang et al.).
We created a testing suite consisting of 18 Java programs. In each program we seeded one single fault. As testing all possible errors and combinations is not possible, we were limited to conducting qualitative experiments. We do that by running the predicate switching tool we built on all the programs in the testing suite.
5.1 The Predicate Switching tool
5.1.1. The Role of the Rascal Language
One of the features of Rascal that we used is the M3 model, which is a language independent meta model for source code analysis. That model includes a symbolic representation for abstract syntax trees of programming languages. Using it made it possible to automatically identify the locations of all ifstatements in the programs with which we experimented. These locations were then used to perform crosscutting transformations on the programs in our testing suite in order to implement the program tracing process and the predicate switching process.
5.1.2 Design of the Predicate Switching Tool
We first define the main steps of our implementation as shown in Figure 2. The input of the system is a program in a specific failing setting. A tracing step then follows in which a trace of the branching predicates visited during the specific failing execution is being created by
running an instrumented version of the failing program. In the next step, the program is transformed to facilitate the function of dynamic instrumentation of the program. The last step is then iterative execution of the input program in which using dynamic instrumentation for each branching predicate from the tracing list a corresponding variation of the execution is being carried out. The function of the tool is divided into the following steps: Figure 2 The general function of the predicate switching tool
5.1.3. Implementation
Our implementation of the tool with Rascal has the following functions:Extracting location of predicates with Rascal The tool starts with the location of the original faulty program as an argument (step 1 in Figure 3). This location is then used by Rascal’s M3 model to extract the exact locations (file location plus the exact begin and end point in the file) of the branching predicates of all ifstatements from all the methods of the program. We limited the tool to ifstatements, while the original work done by Zhang et. al also considered whileloops.
Create a selftracing program This part of the tool uses the ifstatements’ locations
extracted in the previous step and wraps the predicate with a call to another boolean function (step 2 in Figure 3). That function returns the value of that predicate and registers the predicate’s location in memory. The result of this step is a program which is functionally identical to the original program, but also creates a trace of the predicate instances which are visited during the execution of the program.
Create predicate trace In this step (step 3 in Figure 3) the selftracing program is being run, which results in a list of all the predicate instances in the sequence of their execution for the program in the given faultinducing context.
Instrument the program to facilitate predicate switching In this step (step 4 in Figure 3) each predicate that appears in the predicate trace is again being wrapped in the original program in another boolean function which increments a counter and returns the value of the predicate’s evaluation. Once the program is run at a later stage, when the counter reaches the serial number which corresponds to a specific predicate instance, the predicate switching
function returns the negation of the predicate’s evaluation, and in fact causes the desired switching of the predicate (Figure 4). Figure 4 The method added to a program by the tool to facilitate predicate switching
Iteratively execute the program while switching predicates In this step (step 5 in Figure 3) the program is executed once for each predicate instance from the predicates trace list. The result is a list that maps each execution instance (corresponding to a specific switched predicate instance) of the program to a boolean value representing success or failure of the execution (resulting from the evaluation of the assertion that is used to check correct behavior). The first encountered predicate instances that are reported as the critical predicate instances are being examined until an actual critical predicate instance is found, if at all. Figure 3 The different intermediate phases in the function of the predicate switching tool
Reflection on the implementation process
In order to control the implementation, we made sure that the tool identifies correctly all the possible variations of ifstatements (including ifstatements and ifthenelse statements, nested ifstatements, ifstatements with with curly brackets or without). Additionally, the tool is robust against nonfunctional text (such as comments). We saw that the tool properly creates instrumented versions (both for tracing and for predicate switching) of the programs at the right location, without exceptional cases. However, at the point of adding the calls to the tracing function we could not distinguish between branching predicates that were visited in the program execution and those who were not. This resulted in adding calls to tracing functions that were not used.
We also controlled that the predicate instance trace was registered in the correct sequence of execution. Similarly, the predicate instances are being switched in the correct order. The number of the variations of each program corresponded, as expected, to the size of the predicate instances trace. We then controlled that every true result coming from the tool indeed corresponded to the location in the source code of a predicate instance that influences the program’s output. We did this for each program on which we tested our tool and observed that the tool works seamlessly.
5.2. Comparison between our Rascal tool and the tool built by Zhang et al.
The implementation of the predicate switching tool works similarly to the tool that Zhang et. al present in their paper in the following way:
Both tools create a predicate instance trace of a failing execution of a program. Once the predicate trace is created, both tools start at the end of the predicate trace, according to the Last Executed First Switched (LEFS) ordering principle, and generate for each predicate instance in the trace an instrumented variant of the original failing program with that predicate instance switched. In the sequel the result of each execution is controlled, and the first critical predicate instance, that is, the predicate instance whose switching leads the program to a successful execution ends the search process.
The main difference in the way we built the tool and the way Zhang et. al built their tool is the instrumentation method. While Zhang et. al perform program instrumentation on the binary level during runtime, we instrument the programs on the sourcecode level. While Zhang et. al use Valgrind [14] for their implementation, with which they forcibly take over the execution of predicates and manipulate the program’s control flow, our implementation executes the program fed with a reference to the specific predicate instance we want to switch, and the program switches that predicate instance itself.
We chose for this method of instrumentation as it is suitable for program analysis and transformation of sourcecode which served in building our tool. We used the M3 model provided in Rascal, which is a language independent metamodel for facts about source code. Considering this, the implementation can easily be adapted to support any language we want to target the tool at. Besides that, the M3 model is also extensible to include new languagespecific modeling elements.
Another difference between this work and the work presented by Zhang et al. is that in the implementation of Zhang et. al they refine the ordering of the predicate instances trace by prioritizing branching predicate instances based on their dependence on faulty code and on the
dynamic dependence distance from the erroneous output value. They show that doing this can further decrease the search space. We did not implement this refinement as it is merely an optimization feature rather than a crucial functional feature that influences the results.
6. Experimentation
In order to be able to analyse why predicate switching fails at locating faults whose infection chain diverges to multiple predicate instances, we experimented with the predicate switching tool we built on a testing suite that we composed especially for this study. In each program in our testing suite we injected a fault and added an assertion that is expected to fail when the fault manifests during execution.
We used the predicate switching tool to repeatedly execute each of these programs. Correspondingly to the length of the predicate instance trace, the number of iterations on each program ranges from a single iteration (as in the programs “Three Card Poker” and “Tic Solver”) to 889 iterations (as in the program “Morse Tree”).
For each program we looked at whether the fault was located by the predicate switching tool. We then analysed each of these programs and checked whether we indeed observed a relation between the infection of multiple predicate instances and the success of the tool in locating the fault.
We then investigated the effect of the predicate switching tool on each of these programs, and looked for different patterns of the faults’ infection chains through multiple predicate instances in different programs. Our insights are then presented using an example.
Note that the experiments cannot reflect all possible errors in real life, but they still help us gain valuable insights for specific debugging scenarios.
6.1 The testing suite
In order to compose the testing suite, we searched the internet for Java programs that meet different conditions which would make them suitable for our research. Finding such suitable programs appeared to be quite tedious and took three weeks.
We were looking for programs in which the control flow contains branching predicates. Besides, we were interested in programs in which some of the branching predicates depend on some computation, rather than only on a single boolean object. The reason for this is that we
wanted to examine programs with faults that can cause some form of infection chain. Eventually we selected 18 programs, with sizes varying from 38 to 446 lines of code [15].
The number of files or classes involved in the manifestation of the faults was not investigated as a factor that was expected to influence the results. Hence, in order to make the experiments more overseeable, we limited the research to programs that span over a single file. For the same reason we limited the search to standalone programs that do not depend on external libraries.
The testing suite we used is different than the one used by Zhang et al. and by other studies in this field, namely the Siemens testing suite which consists of C programs, containing documented faults [13]. We could not use this testing suite as Rascal, at the moment that this study is being conducted, does not support C programs.
6.2. Faults injection
In order to answer our research question we have seeded a single fault in each program from the testing suite. Each of these faults then leads to some visible undesired behavior.
The injected faults include switching an operator in an equality or in a comparison expression, changing a value at the left hand side or at the right hand side of an equality expression, negation of a logical expression or of a boolean object, and switching a literal in an equality expression.
For the sake of reproducibility, we present the exact faults we have seeded next to the corresponding correct code in table 1.
Detecting when the undesired behavior is being manifested is done by an assertion added at a later point that checks that specific conditions are met, which are expected in a successful execution of the program. It could happen, however, that instead of resulting in undesired output, a program crashes at a certain point as a result of the fault seeded in it.
Note that the type of faults we examined was not taken as a factor in our experimentation, as our focus is more on the influence of the infectionchain caused by a fault on the effectiveness of predicate switching to locate it rather than the influence of the type of fault.
Program Code fragment with fault Code fragment without fault
SudokuGenerator sudoku[y][x] != toCheck sudoku[y][x] == toCheck
Maze if (grid [row][col] == 0) if (grid [row][col] == 1)
Sudoku1 || ! this.row[row][number] || ! this.column[column][number] || ! this.box[getBoxNumber(row, column)][number]); && ! this.row[row][number] && ! this.column[column][number] && ! this.box[getBoxNumber(row, column)][number]);
AVLTree (findClosestNode(rootNode, val).value != val) (findClosestNode(rootNode, val).value == val)
Date year < b.year || ((year == b.year) && ((mm > b.mm) || ((mm ==
b.mm) && (dd <= b.dd))))
year < b.year || ((year == b.year) && ((mm < b.mm) || ((mm
== b.mm) && (dd <= b.dd))))
Agenda if (timetable[day].equals("freee") == true) if (timetable[day].equals("free") == true)
NQueens boolean sameDiagonal = ((c1 r1) == (r2 c2)); boolean sameDiagonal = ((r1 c1) == (r2 c2));
TheoryTimer return !(elapsed > duration); return !(elapsed < duration);
CyclicWords if (a.charAt((i+j)%len) == b.charAt(j)) {// fault ! if (a.charAt((i+j)%len) != b.charAt(j)) {
MineSweeper if (r >= mines.length || r > 0 && c < 0 || c >= mines[r].length) if (r >= mines.length || r > 0 || c < 0 || c >= mines[r].length)
Sudoku2 int boxRowOffset = (i / 3);
int boxColOffset = (j / 3);
int boxRowOffset = (i / 3)*3; int boxColOffset = (j / 3)*3;
LinkedList if ((current==header || current==ender)) {return false;}//fault if ((current==header || current==ender)) {return false;}//fault
ThreeCardPoker return (getRank(hand[0]) == 0 && getRank(hand[1]) == 1
&& getRank(hand[2]) == 12) && //ace low
(getRank(hand[1]) == getRank(hand[0]) + 1 && getRank(hand[2]) == getRank(hand[1]) + 1); return (getRank(hand[0]) == 0 && getRank(hand[1]) == 1 && getRank(hand[2]) == 12) || //ace low (getRank(hand[1]) == getRank(hand[0]) + 1 && getRank(hand[2]) == getRank(hand[1]) + 1);
TicSolver (!won) (won)
MorseTree return (leftChild != null); return (rightChild != null);
BinarySearchTree if(t2!=null) { if(t2==null) {
BMICalculator1 (checkLength < 2 && checkLength > 6) (checkLength > 2 && checkLength < 6)
BMICalculator2 (checkLength < 2 && checkLength > 6) (checkLength > 2 && checkLength < 6)
Table 1 The exact faults seeded in the testing suite
7. Results
We want to understand why predicate switching fails at locating faults whose infection chain spreads to multiple branching predicates. To this end we analyse the behavior of the method by running it on programs from the testing suite that we composed.
In Table 2 the results of the experiments are presented. The table shows for each program we used whether it succeeded or failed at finding the critical predicate that helps locating the fault. For each of the programs the total lines of code is presented, which stands for the maximum amount of code that has to be reviewed in order to locate the fault. Another column presents
the amount of lines of code that is guarded by that critical predicate that was found. The ration between these two columns can be used to estimate the contribution of predicate switching to reducing debugging time. The last column shows in which program false positives were found before successfully locating the actual critical predicate instance. The table also presents the size of the predicate trace which is the number of predicate instances that were visited during the faulty execution. Program Critical predicate instance found Trace Size LOC to check without the tool (total LOC) LOC to check after the tool (% of Total LOC) Number of false positives SudokuGenerator No 25 118 Maze No 196 61 Sudoku1 No 875 109 AVLTree Yes 624 364 9 (2,47%) Date Yes 25 219 3 (1,37%) Agenda Yes 28 82 7 (8,54%) NQueens No 433 65 TheoryTimer Yes 17 98 4 (4,1%) CyclicWords Yes 13 38 15 (39,5%) 1 MineSweeper No 759 255 Sudoku2 No 488 68 LinkedList Yes 20 131 4 (3,1%) ThreeCardPoker Yes 1 86 7 (8,14%) TicSolver Yes 99 92 31 (33,7%) 4 MorseTree No 889 215 BinarySearchTree Yes 98 446 1 (0.2%) 1 BMICalculator1 Yes 2 86 13 BMICalculator2 No 2 85 Table 2 Results
As can be seen in table 2, in 10 programs the seeded fault was located and in the other 8 it was not.
underlying patterns and show why predicate switching is limited to locating faults whose infection chain does not diverge to multiple predicate instances.
The programs in which the faults were located
From the results it seems that in 10 of the programs from the testing suite, the fault was located. This means that switching a certain predicate instance results in a program execution with a correct behavior. From analysing the programs in which the faults were located, it appeared that in all these programs the infection chain of the fault was indeed limited to a single branching predicate instance. Figure 4 illustrates such pattern, of an ifstatement, where the black node stands for a single infected predicate.
The programs in which the faults were not located
In the other 8 programs, the predicate switching was not able to locate the fault. In these cases the fault is being
Figure 4 A general example for program control flow in which only a single branching predicate is infected
manifested, but is not located by predicate switching. We analyzed these programs in order to understand why. From the analysis of these programs we saw that multiple predicate instances were indeed infected by the faults. We observed two patterns in which infection of multiple branching predicate instances happens: when multiple predicates depend on faulty code, and when a single predicate depends on faulty code but is part of an iterative procedure. We will explain these two situations in more detail in the next sections.
In order to explain these observations, we tracked the behavior of the predicate switching tool and analyzed its effect by controlling manually the switching of each predicate instance that we know is related to the fault. The next sections present the findings of these observations.
Multiple predicates depend on faulty code
In such cases where a fault has diverging influence on multiple predicates, it is impossible to repair the fault by merely switching a single predicate instance. Figure 5 gives a simple example for such a pattern. We will explain this pattern by illustrating the effect of the predicate switching tool when multiple predicates are infected. We take as an example the BMICalculator program from our testing suite. This program happens to be written in a way that it contains two functionally identical code fragments, each influencing a different branching predicate (see Figures 6.a).
The code in lines 24 to 37 is responsible for checking whether the input value that is given for the height is a number within a specific range. The predicate at line 38 is influenced by the result of the procedure we just described. The code in lines 52 to 65 does the same checking procedure for the weight, which influences the control flow of the program at another branching predicate, namely at line 66. The fault in this program is in the comparison expression that controls that number of characters in the user input (line 26 in Figure 6.a). Instead of controlling that the input length is within a certain range, the code now controls that it is outside that range.
For comparison we create a variant of the BMICalculator program by simply replacing the duplicated code (lines 2437 and 5258 in Figure 6.a) in a single boolean function CheckValidity (1328 in Figure 6.b) which is similar to these two procedures and is being called at different moments. The same
Figure 5 the infection of different multiple predicates
fault is also kept in this variant of the program, and is now part of the CheckValidity function. (line 16 in Figure 6.b).
When we run the predicate switching tool on each of these two variants of the program, we see that it succeeds at locating the fault in the program depicted in Figure 6.a, whereas when it is run on the program depicted in Figure 6.b, it fails.
A simplified representation of the control flow graphs corresponding to the code in Figures 6.a and 6.b is given in Figures 7.a and 7.b. These figures depict more clearly the difference in the infection chain of the control flow between the two programs (the black nodes represent infected branching predicates, whereas the white nodes represent uninfected ones). It can be clearly seen that changing branching predicate 1 (Figure 7.a) results in a faultfree execution of the program, while in the other case (Figure 7.b), changing either branching predicate 1 or branching predicate 2 does not completely repair the fault’s effect on the execution.
From the analysis of the effect of predicate switching on each of these programs, it seems the explanation for the difference in results between these two functionallyidentical (when correct) versions of this BMI Calculation program is related to the difference in the
24. int checkLength = checkInput.length(); 25. boolean valid = true; 26. if (checkLength < 2 && checkLength > 6) { … 36. } else { valid = false; } … 38. if (valid) { 39. int inputValue = Integer.parseInt(checkInput); 40. if (inputValue == 0) valid = false; 41. else height = inputValue; 42. } 43. else { 44. System.out.println("!ERROR"); 45. System.exit(0); 46. } … 52. int checkWeight = checkInput.weight(); 53. boolean valid = true; 54. if (...) { … 64. } else { valid = false; } … 66. if (valid) { 67. int inputValue = Integer.parseInt(checkInput); 68. if (inputValue == 0) valid = false; 69. else weight = inputValue; 70. } 71. else { 72. System.out.println("!ERROR"); 73. System.exit(0); 74. } Figure 6.a A version of the BMICalculator in which the fault is located by predicate switching 13. public static boolean checkValidity(String input){ 14. int checkInput = input.length(); 15. boolean valid = true; 16. if (checkLength < 2 && checkLength > 6) { ... 26. } else { valid = false; } 27. return valid; 28. } … 50. if (checkValidity(input)) { 51. int inputValue = Integer.parseInt(checkInput); 52. if (inputValue == 0) valid = false; 53. else height = inputValue; 54. } 55. else { 56. System.out.println("!ERROR"); 57. System.exit(0); 58. } ... 59. if (checkValidity(input)) { 60. int inputValue = Integer.parseInt(checkInput); 61. if (inputValue == 0) valid = false; 62. else weight = inputValue; 63. } 64. else { 65. System.out.println("!ERROR"); 66. System.exit(0); 67. } Figure 6.b A version of the BMICalculator in which the fault is not located by predicate switching
infection chains of the fault in each of these programs. In the program depicted in Figure 6.a, the infection chain of the fault includes only a single branching predicate (the one from line 38). In the other variant however, in which the fault is not being located, the infection chain of the fault diverges to multiple predicates (namely those at lines 50 and 59).
Single predicate depends on faulty code, but is part of an iterative or recursive procedure
In this section we describe another infection chain pattern that makes our predicate switching tool fail at locating faults. Namely, such in which faults have influence on a single predicate that is a part of an iterative procedure such as a forloop, a whileloop or a recursive procedure. (a) (b)
Figure 7 The difference in the control flow between single and multiple infected predicate(s)
In these cases (for instance, in the Sudoku Generator, Maze and Mine Sweeper programs in our testing suite), the infected branching predicate is being executed multiple times, which results in an effect similar to the case in which different multiple branching predicates are infected by the fault. As the predicate switching tool causes a single switch of a specific branching predicate instance, it does not manage to repair the effect of the fault completely, and hence it is not able to generate a successful execution instance.
An illustration of the effect of predicate switching on faults that cause infection chains of that sort can be seen in Figure 8. This figure illustrates clearly that switching a branching predicate instance n (presented as white node at the branching point) fails at repairing other branching predicate instances n1 and n+1 (the black nodes at the corresponding branching points).
There are some exceptions however. A fault whose infection chain diverges to multiple branching predicates might still be found by predicate switching in cases where all these branching predicates are themselves controlled by another branching predicate whose switching still compensates the diverging infection caused in the iteration or recursion. Another exception is when a fault is in an iterative procedure, but a certain evaluation of the predicate exits the iteration. In such a case, exiting the iterative procedure might limit the infection to one predicate instance which can be repaired when switched.
predicate instance n1 predicate instance n predicate instance n+1 Figure 8 The effect of predicate switching on the infection of a branching predicate in a forloop
8. Related Approaches
There are multiple publications about methods that use predicate switching . The characteristic that they all share is that they reduce the search space compared to approaches that are based on arbitrary state changes. While the predicate switching technique presented by Zhang et al. is not able to locate a fault if it infects multiple predicate instances, some works suggest ways that could locate faults also in these situations. Liu et al. [3] suggest performing multiple predicate switches at the same time. Whereas Wang et al. [2] propose automatically generating multiple successful executions of a faulty program and selecting one that has a predicate trace most similar to the predicate trace generated from the original failing execution. The sequence of branch instances evaluated differently in these runs is then returned as a bug report.
Although these methods may be able to locate faults that infect multiple predicate instances, they both suffer from another problem, which is a sharp increase in complexity, as the number of possible combinations of predicate instances to switch increases very quickly. Liu et. al propose to fix this problem by bounding the number of iterations of each loop, while Wang et. al have simply excluded them from their analysis for feasibility reasons.
Other examples for automated debugging techniques that could be an alternative for using automated predicate switching are the following: Delta debugging, proposed by Zeller et. al
[7], is a technique where one examines the differences in executions that result from gradually removing parts of the input. This approach, however, is only suitable for a limited range of program types using specific types and structures of input. Another related approach is program slicing [11], and especially dynamic slicing [12]. In this approach, debugging is facilitated by ruling out statements that do not affect the observable error via control and/or data flow.
Predicate switching, and automated debugging in general (including the approaches mentioned in this section) differ from automated testing in the sense that while automated testing aims at detecting faulty program behavior, automated debugging aims at locating the faults themselves. While automated testing is done by generating and testing large amounts of test cases, predicate switching is done by generating and testing large amount of program variations.
9. Threats to validity
One threat to validity in our study is that it uses a testing suite of limited size. This is rather a qualitative exploratory study, and not a quantitative study. Even though the programs we did investigate can provide valuable insights for our exploratory study, we still cannot perform an exhaustive statistical examination of all possible errors and scenarios. Another remarkable threat to validity is that the scope of this work is limited to faults whose influence is restricted on branching predicates. That means that we are unable to generalize the conclusions to faults that violate these restrictions. We were not able to compare our findings with those of Zhang et al, as they did not provide analysis of the negative results they achieved, and due to our use of a different testing suite in a different programming language, as at the time we conduct this study Rascal supports Java and not C.
10. Future Work
The implementation presented here is a demonstration of how Rascal can be used for building dynamic analysis tools. This work could be an inspiration for developing additional dynamic analysis or debugging tools using Rascal. Such tools could be designed to automatically detect erroneous events by tracing and analyzing specific aspects of programs execution which are related to specific language structures (for instance iterations and recursions that could cause excessive use of working memory). Another interesting followup direction could be using predicate switching combined with automated testcase generation. This could be used for locating faults whose occurrence spans over multiple executions, given ranges of inputs.
11. Conclusions
In this replication study we experiment with the predicate switching method presented in Zhang et al. in order to get insights into the reasons why predicate switching fails at locating faults when their infection chain diverges to multiple predicate instances.
To this end we build a predicate switching tool using Rascal, which is a framework that is built for static analysis and program transformation. Our implementation is based on Rascal’s M3 model, which is a languageindependent extensible metamodel for source code analysis , which
makes it suitable to support new languages and language specific modeling elements.
We show that in different patterns of infection chain that diverge to multiple predicate instances, switching a single predicate cannot completely remove the effect of the fault. Such patterns can occur when a fault infects a single predicate which is inside an iterative procedure, or when the fault infects multiple different predicates. In these cases, the infected predicate instances that are not repaired by the predicate switching can still cause undesired behavior of the program.
12. Bibliography
1. X. Zhang, N. Gupta and R. Gupta. Locating Faults Through Automated Predicate Switching. Proceedings of the 28t International Conference on Software Engineering, Pages 272281, 2006.
2. T. Wang and A. Roychoudhury. Automated Path Generation for Software Fault Localization. Proceedings of the 20th IEEE International Conference on Automated Software Engineering, Pages 347351, 2005.
3. Y. Liu and B. Li. Automated Program Debugging via Multiple Predicate Switching . Proceedings of the TwentyFourth AAAI Conference on Artificial Intelligence, Pages 327332, 2010.
4. H. Cleve and A. Zeller. Locating Causes of Program Failures. proceedings of the 27th IEEE International Conference on Software Engineering (ICSE), Pages 342351, 2005.
5. A. Zeller. Isolating Causeeffect Chains from Computer Programs. SIGSOFT Symposium on Foundations of Software Engineering, Charleston, Pages 110, 2002.
6. A. Zeller. “Yesterday, My Program Worked. Today, It Does Not. Why?,” 7th European Software Engineering Conference/ 7th SIGSOFT Symposium on Foundations of Software Engineering, pages 253267, Sept. 1999.
7. A. Zeller and R. Hildebrandt. “Simplifying and Isolating Failure Inducing Input”, IEEE Transactions on Software Engineering, 183200, 2002.
8. M. Renieris and S. Reiss. Fault Localization with Nearest Neighbor Queries. Proceedings of the 18th International Conference on Automated Software Engineering, 3039, 2003.
9. Groce, A. Error Explanation with Distance Metrics. In Proceedings of the 10th International Conference on Tools and Algorithms for the Construction and Analysis of Systems, 108–122, 2004.
10. S. Chaki, A. Groce, and O. Strichman. Explaining Abstract Counterexamples. SIGSOFT Symposium on the Foundations of Software Engineering, 7382, 2004.
11. M. Weiser. Program Slicing. IEEE Transactions on Software Engineering, 352357, 1984.
12. H. Agrawal and J. Horgan. Dynamic Program Slicing. SIGPLAN Conference on Programming Language Design and Implementation, 246256, 1990. 13. The Siemens Testing Suite http://sir.unl.edu/portal/bios/tcas.php, [05022016]. 14. The Valgrind Instrumentation Framework http://valgrind.org/, [05102015].
15. The Code Repository that Contains the Source Code of the Tool and the Testing Suite https://zenodo.org/badge/latestdoi/22542/amirico/Predicate_Switching, [07062016], DOI: 10.5281/zenodo.55059
