Finding null dereferences using simple and effective static analysis

University Of Amsterdam

Finding null dereferences using simple

and eective static analysis

Author:

Danny Koppel

Supervisors: Dr. Alexander Serebrenik Ir. Pier Janssen

Publication status: openbaar 17-07-2014

Abstract

Many techniques have been investigated in recent years to automatically de-tect bugs in software. One of these techniques is applying static analysis to detect bug patterns. The paper "Finding Bugs is Easy" by David Hovemeyer et al. applies static analysis to detect bug patterns, and this thesis's research replicates that paper's approach. This research diers from the original in that this research targets C# instead of Java. An existing framework, Microsoft Visual Studio's Code Analysis, has been used as the basis for the bug pattern detectors instead of creating a new framework.

The focus was particularly on null dereferences. Null dereferences occur often but can be easily detected by using static code analysis bug pattern detec-tors. A proof of concept was implemented to detect null dereferences and to answer the main research question: "can simple analysis techniques eectively nd null dereferences in the object-oriented programming language: C#?". The main research question was divided into three sub-questions to measure whether a sucient number of null dereferences is detected, the acceptance by the developers, and the eectiveness.

The rst sub-question was to measure if null dereference detetors nd a su-cient number of null null dereference bugs. The proof of concept was applied to current Subversion (SVN) revisions and to past revisions. Null dereferences that were already solved were used to determine the suciency of the null dereference pattern detecting. Sixteen solved null dereferences were analyzed to determine the pattern. Fourteen of the sixteen solved null dereferences were found by the proof of concept resulting in a detection rate of 87.5 %.

The second sub-question was to measure acceptance for the null dereference pattern detectors. To determine the acceptability of the bug pattern detec-tors, the developers of Snakeware used the tool for two weeks. After these two weeks the developers were interviewed. The interviewed developers all stated that the false positive rate is acceptable. They also stated that they will use the proof of concept in the future for null dereference detection.

The third sub-question was to measure the eectiveness of the proof of con-cept. The eectiveness was measured by analyzing the SVN log to determine the lifetime of null deferences in production code at Snakeware. Null derefer-ences have a median lifetime of 84 days at Snakeware. All null dereferderefer-ences would have been found and xed within a single day if the proof of concept were in place at the moment the null dereference was introduced in the source code.

Based on the three sub-questions we can condently answer the main research question. This research conrms for C# the ndings of Hovemeyer's paper: bug pattern detection using static analysis is an eective technique for nding null dereferences in software.

Contents

1 Introduction 1

1.1 Research questions . . . 1

1.2 Organization of this thesis . . . 2

2 Background and context 3 2.1 Snakeware . . . 3

2.1.1 Libraries . . . 4

2.2 What is a bug? . . . 5

2.3 Static analysis . . . 5

2.3.1 Static analysis: bug pattern detection . . . 5

2.3.2 Static analysis: Visual Studio's Code Analysis . . . 6

2.3.2.1 Challenges in creating bug pattern detectors using the Code Analysis framework . . . 7

2.4 What the null dereference bug pattern detectors do not detect . 7 2.5 Implementing new bug patterns detector rules . . . 8

2.6 False positives . . . 9

2.7 False negatives . . . 9

3 Problem analysis 10 3.1 Snakeware . . . 10

3.1.1 Libraries Snakeware made available for research . . . 11

3.2 Using SVN for mining frequent bug-x code changes . . . 11

3.3 Validation . . . 12 4 Research method 13 4.1 Approach . . . 13 4.2 Research sub-questions . . . 14 4.2.1 SQ1 . . . 14 4.2.2 SQ2 . . . 14 4.2.3 SQ3 . . . 14 5 Proof of concept 15 5.1 Data-ow analysis . . . 15

5.1.1 Control ow graph . . . 15

5.1.1.1 Basic block . . . 15

5.1.1.2 Ordered nodes . . . 16

5.1.2 Algorithm . . . 16

5.2 Null dereference . . . 19

5.2.1 What is a null dereference? . . . 19

5.2.2 Situations where null dereferences are symptoms and not the real cause . . . 20

5.3 Null dereference bug pattern detector rule . . . 20

5.3.1 Data-ow values . . . 21

6 Results 23

6.1 Applying bug pattern detectors to the current revision . . . 23

6.1.1 ReSharper . . . 24

6.2 Detect null dereference bug patterns in past revisions . . . 24

6.2.1 Null dereference bug pattern detector . . . 24

6.3 Period a null dereference is live in production . . . 25

6.4 Acceptance of false positive rate for null dereference pattern detectors by developers of Snakeware . . . 26

7 Discussion 27 7.1 Applying null dereference bug pattern detectors to the current revision . . . 27

7.2 Detect null dereference bug patterns in past revisions . . . 28

7.2.1 SQ1 . . . 28

7.3 Acceptance false positive rate for null dereference pattern de-tectors by developers of Snakeware . . . 28

7.3.1 Low number of participants . . . 28

7.3.2 Developer "3" . . . 28

7.3.3 Observations . . . 29

7.3.4 SQ2 . . . 29

7.4 Dierences between null dereference bug pattern detector rules and ReSharper . . . 29

7.5 The period a null dereference bug is live in production . . . 30

7.5.1 SQ3 . . . 31

7.6 Answering of the main question . . . 31

7.7 Replication . . . 31

7.8 Threats to validity . . . 32

7.8.1 Sub-question one . . . 32

7.8.2 Sub-question two . . . 32

7.8.3 Sub-question three . . . 32

1 Introduction

Much research has been done in recent years to develop techniques to automat-ically nd bugs in software [3, 16]. Most of these techniques use sophisticated methods [8, 16]. People tend to believe that bugs are dicult to nd because bugs are unique, sophisticated and complex[16]. However, most bugs are not unique, sophisticated, dicult or "deep" [16]. They are common mistakes made by all kinds of developers. Knowing which errors developers commonly occur, makes it possible to search for patterns. Finding these patterns is not very dicult and could lead the discover of most bugs.

Research done by Hovemeyer et al. [16] demonstrates that one easy solution is to nd bugs using simple static analysis in combination with so called, "bug patterns" [16] for the programming language Java. In this thesis the research of Hovemeyer et al. [16] is replicated to determine whether simple static code analysis can be applied to the C# language using an existing environment for null dereference pattern detection rules: Microsoft's Visual Studio Code Analysis, for null dereference bug-nding. This research is an independent replication [23].

One of the major conditions in the research of Hovemeyer et al. [16] is a new built framework for detecting bug patterns. In this report, Microsoft's Visual Studio Code Analysis is used as framework for the implementation of custom null dereference bug pattern detector rules.

We examine whether it is possible to eectively detect null dereference pat-terns using simple static analysis techniques while using an existing framework: Microsoft's Visual Studio Code Analysis.

1.1 Research questions

In this study the main research question was as follows:

Can simple analysis techniques eectively nd null dereferences in the object-oriented programming language: C#?

To answer the main research question, three sub-questions were rst addressed. • SQ1: Can null dereference pattern detectors nd a sucient number of

null dereference bugs?

 In other words, will the null dereference pattern implementations detect more than 50% of the null dereference bugs? This number comes from [16].

• SQ2: Will the frequency of false positives be acceptable to developers? • SQ3: Is null dereference pattern detecting using static analysis an

 In other words, can developers nd null dereferences more eec-tively, and eciently, when they are using static analysis to nd bugs.

1.2 Organization of this thesis

The following chapter (Chapter 2) explains the background and context of this research; the hosting company Snakeware, static analysis, and bug detection are introduced. Chapter 3 contains the analysis of the problem. In Chapter 4, the research method is explained. In Chapter 5, the proof of concept for bug detection is explained. In Chapter 6 the results of this research can be found and Chapter 7 gives an analysis of these results. Chapter 8 contains this study's conclusions and future work.

2 Background and context

2.1 Snakeware

Snakeware is a high-end full-service internet agency: every facet for devel-opment for the company's products is done in-house: back-end develdevel-opment, front-end development, information analysis, requirements engineering, and graphic design of user interfaces for online media.

Snakeware has created their own CMS (Content Management System): Snake-ware CMS. Information is managed centrally and information can be sent on request through dierent information channels, like website(s), intranet, nar-rowcasting screens, social media, prints, Ibeacons and mobiles. Narnar-rowcasting is the spreading (advertising) messages to a select group or audience. One of the customers of Snakeware uses narrowcasting in the supermarket to inform customers about commercials and cooking sessions. The Snakeware CMS is central in all products Snakeware delivers. In Figure 1 an overview is given of the products that Snakeware oers. Snakeware also oers online marketing and strategy `services' like search engine optimization, search engine advertis-ing, social media strategy, website analysis, and (website) usability.

2.1.1 Libraries

Snakeware has developed multiple libraries. The names of the libraries use "Snakeware" as a prex. Currently, Snakeware has around thirty libraries. The largest or the most used libraries are listed in Table 1.

Library C# les LoC

Snakeware.A 4 164 Snakeware.B 17 626 Snakeware.C 449 17.885 Snakeware.D 158 23.287 Snakeware.E 13 288 Snakeware.F 4 10 Snakeware.G 603 16.283 Snakeware.H 255 9.419 Snakeware.I 249 6.671 Snakeware.J 52 1.133 Snakeware.K 11 345 Snakeware.L 19 420 Snakeware.M 30 308 Snakeware.N 20 685 Snakeware.O 78 2.420 Snakeware.P 13 283 Snakeware.Q 720 36.525 Snakeware.R 29 44.237 Snakeware.S 255 9.607 Snakeware.T 33 258 Snakeware.U 12 182 Snakeware.V 41 12 Snakeware.W 7 41

Table 1: Snakeware's libraries. The numbers in Table 1 are measured as follows:

• LoC (Lines of Code): Visual Studio 2013 Professional's Code Metrics. • C# les: running `dir /S /b|nd /c ".cs"' in a batch script.

In all of the above calculations only C# les are used. Other les are omit-ted. Snakeware also develops third party libraries like Magick.NET. Most of these libraries are individual initiatives from developers of Snakeware that are supported by Snakeware. These libraries are also omitted from Table 1. The libraries with the Snakeware prex are all proprietary software. Other libraries like Magick.NET are open-source.

2.2 What is a bug?

In order to nd null derference bugs using simple and eective static analysis, the term "bugs" must be dened. A software bug is a defect in the software: a failure of the software to work as intended [15]. This denition poses a problem for the bug-nding technique under consideration, as this denition requires information about what the software is intended. Depending on the input of tools that apply static analysis, these tools may not know what people intend. Despite this, there are two kinds of bugs that can be found without information on intent [15]:

• software behavior that may be considered erroneous no matter what the software is intended to do;

• inconsistencies in the code.

There is no guarantee that found warnings are real bugs. Trade-os can be used to keep the number of false warnings low. However, making the wrong trade-os harms the potential value of the static analysis: being too conservative keeps the false warnings ratio low but also prevents the analysis from nding real issues. See Section 2.6 for an introduction to false positives.

2.3 Static analysis

Static analysis analyzes source code without executing it and such analysis can be used for dierent purposes e.g., program verication, compiler optimization, program visualization, software maintenance and bug detection[19].

Static analysis has proven to be a worthwhile endeavor for software companies since it allows them to automatically discover defects with minimal overhead[9]. 2.3.1 Static analysis: bug pattern detection

Many techniques for nding bugs in software exist and much research has been devoted to developing techniques to automatically nd bugs [6, 8, 12]. Dynamic techniques, such as assertions, rely on the runtime behavior of soft-ware.

One technique that can be used to detect bugs is bug pattern detecting using static analysis.

Hovemeyer et al. used this technique in their research [16] which shows that simple static analysis could be used to detect bug patterns, code idioms that are often errors [16].

The bugs that are already solved can be learned from, since investigating the bugs shows how these patterns represent a bug. Finding these bugs in the code means that the bug searched for, probably exists in the source-code.

A few advantages of static code analysis for bug-nding in the case of Snake-ware are as follows:

• common errors of the hosting company can be detected with relative ease, through implementing custom bug pattern detector rules;

• full code coverage;

• incomplete programs can be analyzed;

• static analysis does not depend on the compiler used;

• static analysis can be done while writing code, making it cheaper to repair errors because the earlier in the process, the cheaper the x [13, 24].

However, a disadvantage of static analysis remains:

• bug-nding static analysis can not be complete and sound [15]:  complete means "report only real bugs";

 sound means "nds every real bug".

The disadvantage that "static analysis can not be complete and sound" is a commonly accepted trade-o for static analysis [4]. The other two disadvan-tages are discussed in Section 2.6 and 2.7.

2.3.2 Static analysis: Visual Studio's Code Analysis

In 2002 Microsoft built FxCop1_{, a standalone program to analyze managed}

code assemblies and report warnings for potential bugs found. Microsoft later decided to integrate FxCop into Visual Studio and renamed the version Mi-crosoft uses in Visual Studio into: Code Analysis.

For the rest of this thesis, Visual Studio's Code Analysis will be used.

In our research we applied static code analysis to the compiled assemblies. Visual Studio's Code Analysis is used to implement custom rules. VS Code Analysis does the analysis of the meta-data and the Microsoft Intermediate Language (MSIL) code. Metadata is "data about data": information about one or more aspects of the data. The metadata for a method contains the signature of the method: the return type, parameters and the class containing the method. The metadata is stored in the assembly where the MSIL is stored. The MSIL is the intermediate language in which the source code is translated. In our case MSIL and IL are interchangeable and for the rest of this thesis MSIL will be used. The MSIL code will be compiled by the Just In Time (JIT) compiler.

Currently Microsoft's implemented rules, of Visual Studio Code Analysis are based on [11]. In 2003, Microsoft introduced a new (data-ow analysis enabled) engine: Phoenix2_{. Currently the Phoenix project appears dead. There has}

1_{http://msdn.microsoft.com/en-us/library/bb429476(v=vs.80).aspx} 2

been no information about Phoenix from Microsoft since 2008, and neither is there an ocial statement from Microsoft about this subject.

2.3.2.1 Challenges in creating bug pattern detectors using the Code Analysis framework

No ocial documentation from Microsoft on implementing custom bug detec-tion rules exists. Jason Kresowaty gives the most comprehensive introducdetec-tion to, creating custom bug detector rules, in FxCop and Code Analysis: Writing Your Own Custom Rules available on http://www.binarycoder.net. Web-site accessed on 01-06-2014.

Some time is required to learn about Visual Studio's Code Analysis SDK and a .NET decompiler is required to investigate the Code Analysis SDK from Mi-crosoft. Microsoft uses the namespace MiMi-crosoft.FxCop.Sdk and scatters the namespace over a number of dll les.

Developers can nd a starting point by combining the knowledge they gain from investigating the Code Analysis SDK and the actual implementation of rules from Microsoft. Microsoft made all the classes "internal sealed", except the classes from one dll. "Internal" is an access modier for types and type members. Internal types or members are accessible only within les in the same assembly. `Sealed' prevents other classes from inheriting from a class. Ocially, Microsoft does not support creating custom bug pattern detector rules. As a consequence of making all classes "internal sealed" developers can-not initialize classes from the SDK. Microsoft does oer one dll that is open for use for developers. This dll le, FxCopSdk.dll, contains the abstract class BaseIntrospectionRule. The abstract class BaseIntrospectionRule must be ex-tended to implement custom rules.

If developers need to access "internal sealed" classes, they can apply reec-tion. Reection is the process where the program can observe and modify its own structure and behavior at runtime. In our case reection is used to im-plement unit testing. Unit testing is used in our research to test bug pattern detector rules, data-ow analysis and the control ow graph implementations.

2.4 What the null dereference bug pattern detectors do not detect

The null dereference bug pattern detectors do not detect and report potential bugs that more powerful tools do [5, 8]. There are two reasons for this [5]:

1. keep the analysis relatively simple;

2. avoid generating too many warnings that don't correspond to true de-fects.

One such case is nding null pointer dereferences that only exists in a single execution path that will never occur [5]. The problem is that the analysis does not know whether the path is feasible: some execution paths will be used simply because of the logic of a programmer. The static analysis does not know what the developer intended and will check all possible execution paths. The null dereferences detector pattern rule looks for statements and branches that guarantee a null dereference [5, 16].

2.5 Implementing new bug patterns detector rules

Adding new bug pattern detectors means creating a new set of rules in Visual Studio's Code Analysis. In the research from Hovemeyer et al. [16] stated in their research that implementing new rules is relatively easy and could be done by people with an undergraduate level education in computer science. Implementing new rules in Visual Studio's Code Analysis requires program-ming in C#. It is to simple to state that implementing new rules is relative easy and could be done by undergraduate students although this might have been the case for FindBugs. There are variables at play that inuence the level required of the developer to implement bug pattern detector rules. The level diers depending on the bug pattern detector rule. Also Hovemeyer et al. do not state what exactly they mean by implementing new rules. Do they refer to the implementation itself or do they also refer to the research done prior the implementation to determine what the bug is about. It is reasonable to expect that they mean the implementation eort only. There are simple bug patterns and (more) complex bug patterns. Investigating the bugs reported by developers is a complex and time-consuming task. Complete understanding of a bug that has occurred and been xed is required to implement a bug pattern detector rule.

Adding a new pattern is possible after the developer gains enough knowledge about implementing custom rules in Visual Studio's Code Analysis. Presently, there is no good documentation available for writing custom rules in Visual Studio's Code Analysis, so signicant time is required to learn enough to start writing custom rules. There is no ocial support or documentation from Mi-crosoft for writing custom rules, making it harder for us to create custom rules than it was for Hovemeyer et al. See section 2.3.2.1 for more information about this subject. This research required novel implementation of the null derefer-ence bug patterns.

The steps before implementation are much more dicult. Deep understanding of the language or framework is required to create bug pattern detectors. De-velopers have to determine what the bug, that they want to detect, means by researching current bugs: they must determine what the bug pattern is, so it can be implemented as a custom rule in Visual Studio's Code Analysis. A bug pattern has to be implemented precisely to keep the number of false positives low.

2.6 False positives

One disadvantage of static code analysis for bug-nding is the high number of false postives and false negatives. So called "false positives" include both patterns identied as bugs and patterns that probably never occur. Research indicates that when the false positive rate is too high, more than 30%, devel-opers will not use the tool[7]. Research from B. Johnson et al.[17] shows that static code analysis for bug-nding is underused because of the high number of false positives.

One way of keeping the false positive rate low is to notice that each bug pattern rule, Microsoft's rules and custom created rules, can be marked by developers with an annotation in Visual Studio. If a bug pattern found is a false positive, marking it as false positive through annotation will prevent it from being reported again. A potential issue with annotations, however, is that if the code is changed later but the developer forgets or ignores the an-notation, there is the potential that a new bug of the same kind, will be ignored. To keep the false positive rate acceptable for developers, trade-os have to be made in the implementation of bug pattern detector rules. Such trade-os have a noteworthy consequence, however potential bugs are missed by the bug pattern detector rules.

2.7 False negatives

False negatives are a very dicult matter because it is very dicult to prove that software is bug free. When static analysis nishes without nding bugs, this does not guarantee that the software is bug free.

Testing for false negatives can be done by formal proof[13, 16, 25]. How-ever, even formal proofs make simplied assumptions that could result in false negatives. Another method to detect false negatives is to compare the results from the static analysis with the bug tracker corresponding to the software. If bugs are reported, in the bug tracker, but not found during the static analysis, this is a false negative. Snakeware does not use bug trackers, so version con-trol systems have to be used to manually search for bug xes. False negatives are a real risk for static code analysis but due to project size limitations false negatives are left out of the scope of this research.

3 Problem analysis

Many developers think that bugs are dicult, sophisticated, unique and "deep" [16, 21]. However, some bugs are common mistakes that can be located by searching for bug patterns using simple static code analysis[16]. If bug patterns are implemented to detect common bugs, nding bugs could be easier than people assume.

3.1 Snakeware

At the moment the developers of Snakeware use Code Analysis in Visual Studio with a set of default rules, from Microsoft, for bug detection. Found warnings may not even be bugs because the default rules in Visual Studio's Code Anal-ysis are based on the book: Framework Design Guidelines [11]. These rules are guidelines only, and the number of warnings that is found is high. After checking for multiple possible bugs, developers of Snakeware nd a lot of the potential bugs are false positives, will probably never occur or such that the developers of Snakeware disagree with the rule. However due to the high num-ber of results, it is too time consuming to check all results.

Consider an example: the default ruleset "All Rules" from Microsoft is ap-plied to the library Snakeware.Q. This ruleset yielded: 5457 warnings. Among these 5457 warnings the warning "CA1704 - Identiers should be spelled cor-rectly" occurred 4012 times. As a result, this rule is disabled. Other rules are also disabled because for dierent reasons. Dirk Lemstra, one of the developers of Snakeware, provides some reasons:

• "I sometimes disagree with Microsoft about the argumentation for the implementation of a specic rule".

 For the rule "CA1000: Do not declare static members on generic types" Lemstra disagrees with Microsoft's argument that it is not exactly clear what happens in code: "I see immediately what is going on and what happens in the code!"

• "Some rules ag a warning when the element (library or namespace) is just created. While the library or namespace will grow in the future, it will be agged by the rules in the actual situation, which is unwanted behaviour.". To illustrate Lemstra refers to the rule "CA1020: Avoid namespaces with few types"

• "For some rules I lack understanding of the rule". Lemstra acknowledges that he does not know exactly what some rules check for. This is not a tool issue but a problem with lack of knowledge about implementation details of the rule from the side of the developer. However, because of the high number of available rules, the developers of Snakeware do not have thoroughgoing knowledge of all available rules from Microsoft.

Lemstra stated that he would use all the rules if he could create a new public API. These ndings correspond with the research of B. Johnson et al. [17] which indicates multiple reasons for underuse of static code analysis:

• the number of warnings produced by a tool can be high; • false positives can outweigh true positives;

• developers do not always understand the warning that tries to explain the reported issue, these warnings could be claried if there were larger and more complex examples available.

These ndings underline the importance of a very low false positive rate for new custom rules if we want the developers to use the new bug pattern detector rules.

3.1.1 Libraries Snakeware made available for research Snakeware made three libraries available for this research:

• Snakeware.H • Snakeware.I • Snakeware.Q

These three libraries are used for database handling, mail, imaging, xml, html processing, CSS processing, Javascript processing, XSL, payment, weather in-formation, and search and location. The reason Snakeware chose these three libraries is that these libraries are used in almost all projects. Bugs in these libraries impact almost all of Snakeware's projects.

3.2 Using SVN for mining frequent bug-x code changes

Software evolution is inevitable [22] and mining previous bug xes can pro-duce reliable knowledge about why bugs happen and how they are xed [22]. Unfortunately Snakeware has no bugtracking software, so the SVN log had to be analyzed to determine the bug-x history. The bug-x history of the three libraries Snakeware made available for this research were manually analyzed to determine the most common kinds of bugs. The null dereference bug was the most frequently reported bug in the three libraries.

Not all commits are bug-xes making it necessary to determine whether a commit is a bugx commit. Multiple search strings have been used: "bug -debug" and "x". The addition -debug means that strings containing the part "-debug" are excluded from the results. The results from the string "x" were too broad: all kind of xes were accounted for, including loading the correct javascript les, or getting the correct URL. Loading incorrect javascript les and getting the wrong URL are also bugs. However this research did not aim for clerical errors, like typos. Commits containing bug-xes for these errors

# Commits Bug -debug Fix First commit Last commit

H 625 11 109 02-10-2011 04-06-2014

I 298 18 47 19-11-2010 26-05-2014

Q 1982 69 323 02-11-2010 06-06-2014

Table 2: Overview of the three Snakeware libraries

were therefore omitted. In consequence manual investigation of all the results was required.

The hits for `x' were scanned through, to select the real bug xes.

3.3 Validation

After null dereference bug pattern detectors were implemented, detected warn-ings have to be validated.

False negatives were kept out of the scope of this research but we would have known if the implemented bug pattern detector rules found a acceptable num-ber of warnings or potential bugs. See Section 2.7 for more information about false negatives.

To ease the validation process a bit, a baseline was made. The baseline was a starting point with which the results from our null dereference bug pattern detector rules could be compared. One tool was used to build the baseline: ReSharper (8.1). ReSharper, like Microsoft Visual Studio's Code Analysis can detect potential bugs and not only guidelines. Microsoft's Visual Studio Code Analysis is based on [11]. After the results were gathered, results had to be manually inspect to conrm that each warning found is a real bug. Warnings that were not real bugs, are removed from the results. The warnings found by Resharper are used as checkpoint only. Results found by the custom rules that developed for this research were compared to the baseline.

Not all rules that are known in ReSharper were implemented as custom bug pattern detector rules. So the baseline only contained warnings found for null dereference bug pattern detector rules that were also implemented in the re-search. Other found warnings were omitted from the baseline.

4 Research method

This research is constructive in the sense that work was carried out toward the establishment of a practical solution: null dereference bug pattern detec-tors implemented in MS Visual Studio's: Code Analysis, that can detect bug patterns.

4.1 Approach

To begin with, a literature survey is conducted to investigate the current state of static analysis and "bug nding". Starting point is the research of Hove-meyer et al. [16].

Next the bug x history of the Snakeware libraries through the SVN log is investigated. Code prior to and after a bug x commit is compared to learn what the bug means and how it had been xed. A summary is created to describe the bug, and it included: the problem, the solution, and how to (pos-sibly) detect this problem. Concluded was that the null dereference bug was the issue that occurred most frequently at Snakeware. A baseline for the po-tential current bugs is also created, to facilitate validation at a later point. See Section 3.3 for more information about the baseline of the current situation. A theory review is conducted to learn about dataow analysis, control ow graphs and required algorithms [1, 2, 10, 18].

Next the proof of concept is implemented. Before a null dereference bug pat-tern detector is implemented, a dataow analysis framework for bug patpat-tern detection has to be implemented. The dataow analysis framework is made generic, so other bug pattern detector rules could also use the framework, if required. The data-ow analysis framework is targeted to bug pattern detec-tors only. The null dereference bug pattern detector rules are applied to the libraries of Snakeware to test the functionality of the rule.

The next step is to apply the null dereference bug pattern detectors to the source code, both: prior to the x and after the x. The null dereference bug pattern detector should detect the xed bug in the source code prior to the bug-x commit. The null dereference bug pattern detectors are also applied to the current version: the most recent commit.

The nal step is to analyze and validate the results that are gathered dur-ing the previous step, applydur-ing the null dereference bug pattern detectors to the source code. For validation of data gathered while applying the null deref-erence bug pattern detectors, a baseline is created in earlier steps to facilitate the validation process. For more information about the validation and the baseline see Section 3.3.

4.2 Research sub-questions

The multiple sub-questions are introduced in Section 1.1. 4.2.1 SQ1

Sub-question 1: "Can null dereference pattern detectors nd a sucient num-ber of null dereference bugs?".

To determine if the null dereference bug pattern detectors rules could nd a sucient number of null dereference bugs, the SVN log is manually analyzed to determine the bug xes. The "before" revision is checked out when the code contains the bug. The bug pattern detectors are applied to the revision to determine if the bug is detected.

4.2.2 SQ2

Sub-question 2: "Will the frequency of false positives be acceptable to devel-opers?".

The false positive rate need t be low otherwise developers would not likely use the tool. To determine if the false positive rate is acceptable, the develop-ers of Snakeware are asked to integrate the proof of concept, see Section 5, in their (build) environment: Visual Studio.

After two weeks, the developers are interviewed to determine if the false posi-tive rate was acceptable.

4.2.3 SQ3

Sub-question 3: "Is null dereference pattern detecting using static analysis an eective technique for nding null dereferences?".

To determine if bug pattern detecting using static analysis is an eective tech-nique for nding bugs we rst have to explain what "eective" means in this context. Are the null dereference bug pattern detector rules, implemented in Visual Studio's Code Analysis, eective when a developers tries to nd bugs: can the developer nd the bug in less time with the bug pattern detector rules. To determine if (potential) null dereference bugs can be found more eec-tively, the SVN log is analyzed to determine how long null dereference bugs were exposed and used in production code.

5 Proof of concept

After analyzing the SVN logle, null dereferences appeared to be the bug-x that occurred most frequently. The literature survey [1, 5, 16] showed that a dataow analysis could be used for detecting null dereferences.

5.1 Data-ow analysis

Data-ow analysis refers to a body of techniques that derive information about the ow of data along program execution paths [1].

The algorithm described in Section 5.1.2 was used for the data-ow analysis. Problems that require a forward or backward data-ow analysis are supported in the built framework.

The data-ow analysis framework was made generic, so the developers of Snakeware could implement new bug pattern detector rules, if required. In this research the data-ow analysis was specically implemented for the null dereference bug pattern detector.

5.1.1 Control ow graph

Once MSIL is partitioned into basic blocks, the ow of control is represented by a ow graph [1].

The control ow graph is used in the data-ow analysis: a control ow graph for each method is generated.

5.1.1.1 Basic block

Each node in the control ow graph is a basic block. A basic block is a set of instructions that is a straight line sequence of code with only one entry and one exit point. The codes in the basic block are MSIL instructions. To determine which MSIL instructions belong to which basic block, the leaders of the instructions had to be determined. The leader is the rst instruction in a basic block. A basic block contains the instructions from a leader till the next leader. At the next leader, a new basic block starts. To determine the leader, the following rules have been applied:

1. the rst statement is always a leader; [1]

2. any statement that is the target of a conditional or unconditional jump is a leader;[1]

3. any instruction that immediately follows a conditional or unconditional jump is a leader;[1]

4. the rst statement of a try block is a leader; [2] 5. the rst statement of a catch block is a leader; [2] 6. the rst statement of a nally block is a leader. [2]

5.1.1.2 Ordered nodes

The control ow graph oers a property to retrieve the ordered nodes (basic blocks). The order of these nodes is determined by a depth-rst algorithm. The algorithm used is a simple implementation that determines the sequence that is used to walk through the basic blocks in the dataow analysis for a meet operation. In any dataow schema, the meet operator is the one that is used to create a summary of the contributions from dierent paths at the conuence of those paths [1]. The meet operation in Listing 1 is Line 8:

IN [B] = ∧_{p is a predecessor of B}OU T [P ] (1) 5.1.2 Algorithm

The round robin iterative algorithm was chosen because of its robustness in behavior. The theory behind the algorithm shows that, for a broad class of problems, it terminates and produces correct results[10]. In the proof of con-cept forward and backward data-ow problems can be solved using the round robin iterative algorithm. The round robin iterative algorithm for the forward and backward data-ow problem was implemented based on the algorithm in [1], page 627 and [10]. An iterative approach is a common way for data ow equations [1, 10, 18].

1 OUT [ ENTRY ] = VEN T RY

2 for ( each basic block B other than ENTRY ) 3 OUT [B] = >

4 while ( changes to any OUT occur ) 5 {

6 for ( each basic block B other than ENTRY ) 7 {

8 IN[B] = ∧p is a predecessor of BOU T [P ]

9 OUT [B] = fB(IN[B ]);

10 } 11 }

Listing 1: Iterative algorithm for a forward data-ow problem.

The algorithm from Listing 1 is thoroughly described in [1, 10]. The following components are used in the algorithm [1]:

1. a data-ow graph, with specially labeled ENTRY and EXIT nodes; 2. a set of values V;

3. a meet operator ∧ ;

4. a set of functions F, where fB in F is the transfer function for block B;

5. a constant value VEN T RY or VEXIT in V, representing the boundary

The data-ow graph, mentioned in the list above, is the control ow graph discussed in Section 5.1.1. The ENTRY and EXIT nodes are basic blocks that represent the entry and exit points of the ow graph.

IN and OUT are used to store information about the state of a basic block. IN has state information before the basic block and OUT stores the state after the basic block. The state of locals, class elds, arguments and the stack are stored. The state can be: null, non-null, not used or unknown.

If we look closely at the level of statements we nd IN[s] and OUT[s] where s is the statement, the state of IN[s] is prior the statement, while OUT[s] is the state after the statement. OUT[s] is the same as IN[s2] where s2 is the next statement of s.

Transfer functions come in two avors: forward and backward. The trans-fer function takes the value V of a statement s from IN[s] and produces the value V in OUT[s] in a forward scenario. That is: OUT [s] = fs(IN [s]). The

backward scenario is the other way around: it produces a value V for the statement s IN[s] from OUT[s]. That is: IN[s] = fs(OU T [s]). The meet

operator merges the values of the instructions from the basic blocks between predecessors basic blocks as the starting point for the current processed basic block. For the null dereference detector a forward reaching denition transfer function has been used.

As can be seen in Listing 1, in every iteration of the for loop, the rst action is the meet operation. The meet operation is executed on each predecessor, except the entry block. The entry block is an empty basic block with a prop-erty to state it is the only entry point, for the control ow graph. Internally, the meet operation is nothing more than a merge of the predecessors values V. The values V represent values for the current state of stack, elds, locals, and parameters. The value for each local, parameter, eld and stack is merged between the predecessors. If the merged value of a eld was "NonNull" but in the current block is set to "Null", the new value will become "Null". In Table 4 the look-up table is shown that is used to determine the new value V for a eld, local, argument or the stack. Table 4 should been read as follows, the row represents the new value and the column the old value V, so if the value for a local was Null but has a changed value: Non Null, the value will be Non Null. See Section 5.3.1 for an explanation of the values Null, Non Null, Not Used, and Unknown.

After the meet operation is executed on all predecessors, except the entry block, the transfer function is called. In the transfer function all MSIL instruc-tions, for the basic block the transfer function is executed on, are processed. In the transfer function all MSIL opcodes are processed to determine if there is a possible null dereference. The MSIL instructions come from the compiled assembly being analyzed.

In Listing 2 the implementation of the transfer function is written down in pseudo-code. The exit node, the opposite of the entry node, is not processed. It does not have any instructions and it does not change anything. The next step is to process each instruction. The instruction contains an opcode that represents the meaning of the instruction, an opcode species the operation to be performed. Consider an example, if the opcode Ldnull (load null) is exe-cuted, a null reference is pushed onto the evaluation stack. If the next opcode is Std (store eld), the null reference is stored in the eld, the std belongs to. Now the transfer function knows that the given eld is null, and that if the method call is done, it will cause a null dereference.

The scope of the current opcode is determined, and the value is stored in the basic blocks for the corresponding scope only. Listing 3 shows a scope example. The local "str" is initialized with null (opcode: LdNull) and because of the if / else statement the analysis knows the value cannot be null inside the if part while it will be null in the else part. Based on this information the cor-rect value is processed in the basic block for that particular scope. The scope cannot be determined from the opcodes itself, because there is no information in the opcode instruction itself. The scope of the basic blocks is determined by the control ow graph, see Section 5.1.1 for the control ow graph.

For all opcodes described in Appendix A the value V is determined, if possible for the current opcode. The value V is stored for the corresponding element, for instance if the opcode "Stloc_0" is executed, the value is stored in the local variable list at index 0. Each opcode acts like the action the opcode would execute during runtime, so a "store" opcode like Std, Stloc, Starg, and so on, stores the value V in the corresponding location while a "Ldloc_0" would load the variable at the local variable list at index 0 on the evaluation stack. In the transfer function, all opcodes in Appendix A behaves the same as a opcode would do during runtime. However, if an opcode is executed that will be per-formed on an other element, the value of the other element will be checked for null. For instance, in Listing 3 the local variable "str" is used as an object to call a method on. First the "str" would be loaded with the opcode "Ldloc_0" and the next opcode will be "Call", the call to the method ToString(). While analyzing the opcode "Call", the corresponding element is checked for null, in this case the element is the local variable "str". All operations that are executed on other elements are checked for null. These "other elements" are objects like "str" from Listing 3, objects that are used to execute anything on, like calls to methods, elds, or properties.

In Appendix A all opcodes used in the proof of concept are listed and ex-plained.

Unknown Null Non Null Not Used

Unknown Unknown Null Unknown Unknown

Null Null Null Null Null

Non Null Non Null Non Null Non Null Non Null

Not Used Unknown Null Non Null Not Used

Table 3: Merge value table

1 public void Transfer ( BasicBlock pBlock ) 2 {

3 if pBlock is exit block 4 return ;

5

6 foreach instruction in pBlock

7 process the instruction based on opcode ; 8 }

Listing 2: Pseudo-code Transfer function.

1 public void DemoCode () 2 { 3 String str = null ; 4 if( str != null ) 5 { 6 // No null dereference 7 str . ToString (); 8 } else { 9 // Null dereference 10 str . ToString (); 11 } 12 }

Listing 3: Scope example

5.2 Null dereference

5.2.1 What is a null dereference?

A null dereference is a runtime error that occurs when source code tries to access anything on a null pointer. This could for instance be a method call, a property or a public variable. An example is given in Listing 4.

public String GetMyName () {

String myName = null ;

return myName . ToString (); // Null dereference }

Listing 4: A null dereference example

In the example all method calls, after the assignment of null, that will be done on the variable myName will cause a null dereference because of the explicit

null assignment. In this particular case a null dereference could easily have been prevented. Two solutions would be:

1. assigning a empty string to myName instead of null; 2. applying a explicit guard (if check).

5.2.2 Situations where null dereferences are symptoms and not the real cause

In some cases, technically speaking a null dereference, is merely a null deref-erence while the cause of the null derefderef-erence is the real issue. For instance if the wrong equality check is applied in an explicit guard (if check) a null dereference could occur. For these situations dierent rules must be written to detect the warning and notify the developer.

A good example for this kind of situations is the bug pattern detector rule: CheckForWrongEqualityOperator.

// Wrong code : String name = null if( name == null )

return name . ToString () // Correct code :

String name = null if( name != null )

return name . ToString ();

Listing 5: CheckForWrongEqualityOperator example Listing 5 outlines two situations:

1. "Wrong" code: the developer probably intended to write the correct code;

2. "Correct" code: the developer wrote the correct equality operator. Technically the "wrong" code will cause a null dereference. The issue is, how-ever not the null dereference but the use of a wrong equality operator, an error of the developer's logic. These null dereferences should be detected but it is the developer's responsibility to recognize the real cause of the null dereference.

5.3 Null dereference bug pattern detector rule

The null derefence bug pattern detector rule was implemented using the data-ow analysis framework that was built during this research. See Section 5.1 for more information about the data-ow analysis framework.

The null dereference bug pattern detector is a forward analysis on method level. The starting point of the analysis are the methods. This a consequence of using Microsoft's Visual Studio Code Analysis. There was a limited number

of entry points for an analysis. In this research the "method" entry point was chosen as starting point for the analysis. The data-ow values were C# stack frames containing slots representing method parameters, local variables, class elds, and stack operands [16]. Each slot contained a value indicating whether the value contained in the slot is: null, not null, not used, or unknown. To keep the false positive rate as low as possible only in situations where the analysis is reasonably settled about a null dereference, the possible null dereference will be agged as null. This had two potential consequences:

1. The false positive rate would be kept low; 2. Real bugs would be potentially missed.

A trade-o was made to reduce the number of false positives. 5.3.1 Data-ow values

There are four possible values for elements being analyzed. The possibilities are: Null, Non Null, Not Used and Unknown. For instance, if a local has the value "Not Used", the local is only declared in a method but has never been used at the current point of the analysis.

Value Description

Null If a instruction represents a null value, there will be a null dereference. Non Null The analysis is sure a value is non null. This will denitely not cause a

null dereference.

Not Used The element that has the value "Not Used" has not been used until this moment. For instance, if a local has the value "Not Used", it is only declared but never written to, or read from.

Unknown If a element is represented by the value Unknown, there is not enough information available at the current point of the analysis to determine another value like Null, Non Null or Not Used. The value Unknown is only used at the initialization of the default values as described in the next section.

Table 4: Values 5.3.2 Default data-ow values

When the rule is initialized, all slots mentioned in Section 5.3 will be lled with the following default values:

• elds are default Unknown; • arguments are default Unknown;

• reference to receiver object (this) is default NonNull; • locals are default NotUsed;

• stack items are default NotUsed;

These default values were chosen to keep the false positive rate as low as possible. The default values were based on [1, 14, 15, 16]. These values are not VEN T RY, they are initial values chosen to keep a low false positive rate.

6 Results

To determine if simple and eective null dereference bug pattern detectors are feasible, tests were executed to answer the research (sub)questions. See Sec-tion 4.2 for an explanaSec-tion of these tests.

These tests were executed on a machine with the following specications: Intel Core i3-4130, 4 GB memory. Operating system: Microsoft Windows 8.1.1 Pro-fessional 64 bit. Integrated development environment: Microsoft Visual Studio 2013 with ReSharper 8.1. Microsoft .NET version: 4.5.

This chapter only contains factual results. The analysis and discussion of the results can be found in Section 7.

6.1 Applying bug pattern detectors to the current revision

The results of applying the null dereference bug pattern detectors to the current revisions of the libraries of Snakeware are shown in Table 5. The numbers in

Library Revision # Total warnings Real issues False positives

Snakeware.H 626 6 6 0

Snakeware.I 298 2 2 0

Snakeware.Q 1997 165 155 10

Total - 173 163 10

Table 5: Issues found on current revisions

Table 5 show a very low false positive rate. The average of 94.2% real issues on the three Snakeware libraries is good. The null dereference rule of ReSharper was also applied to the three Snakeware libraries. See Table 7. ReSharper had an average of 73% real issues on the three Snakeware libraries.

There was a big dierence in the number of found null dereferences in Snake-ware.H, Snakeware.I and Snakeware.Q. Snakeware.Q had the most reports and also the most lines of code. However size is not the only issue. Snakeware.Q had more complex source code. In Table 6 the cyclomatic complexity [20] of the projects is shown. Cyclomatic complexity is a indicator for the complexity of a program. Snakeware.Q has a cyclomatic complexity of 18.500. The cyclo-matic complexity is calculated in Microsoft Visual Studio. Combining these two observations, results in more reported null dereferences bugs.

Library Cyclomatic Complexity Snakeware.H 4.841

Snakeware.I 3.500 Snakeware.Q 18.500

Table 6: CC per project 6.1.1 ReSharper

An observation made with ReSharper was that ReSharper ags parameters as possibly null when they are not guarded with an if statement, while the null dereference bug pattern detector implemented in the proof of concept sets a default value of "Unknown" to parameters. The trade-o made in the implementation of the proof of concept to choose "Unknown" as the value prevents false positives. Snakeware implemented its own null check that was used. At the beginning of a method, the function Throw.IfNull(parameter) checks if a parameter is null. The Throw.IfNull check is done at runtime. The Throw.IfNull is an ordinary if statement that throws an exception if the parameter value is null. With this knowledge the code following the IfNull check can never be null, unless a new assignment has been done. However, Snakeware does not assign new values to parameters in a method. Snakeware implemented a custom Code Analysis rule that prohibits assigning values to parameters. Parameters were made immutable this way.

The results are discussed in Section 7.

Library Revision # Total warnings Real issues False positives

Snakeware.H 626 40 27 13

Snakeware.I 298 13 6 7

Snakeware.Q 1997 177 135 42

Total - 230 168 62

Table 7: Issues found by ReSharper 8.1 on current revisions

6.2 Detect null dereference bug patterns in past revisions

The null dereference bug pattern detector rules were applied to multiple revi-sions to detect the bug xed in the revision. The revision prior to the bug-x was checked out so the null dereference bug pattern detector rules could be executed on the revision.

6.2.1 Null dereference bug pattern detector

There were sixteen real null dereference bugs xed in the commit history of the three Snakeware libraries: Snakeware.I, Snakeware.H and Snakeware.Q. Fourteen bugs were detected with the bug pattern detector rule. So 87.5% of the solved bugs were found by the bug pattern detector rules making the tool powerful enough for daily use as the two missed null dereferences cannot

occur in new source because of the Throw.IfNull and immutable parameters described in the previous section. The results are discussed in Section 7.2.

Library Null deref bugs Found Not Found

Snakeware.H 6 4 2

Snakeware.I 1 1 0

Snakeware.Q 9 9 0

Totals 16 14 2

Table 8: Issues found on previous revisions

6.3 Period a null dereference is live in production

The sixteen null dereferences bugs that were xed in the past revisions, were analyzed to determine how long the bugs were active in production code. The sixteen bugs have a median of 84 days being live in production code. Three bugs were solved within a day. One bug was live for 541 days. The two null dereferences bugs that the null dereference bug pattern detector did not nd, see Section 6.2.1, were both found and solved within a day by the developers of Snakeware.

Running the null dereferences bug pattern detector took only seconds for each library, based on the size of the library.

In Figure 2 a representation of the number of days the bugs described in Sec-tion 6.2.1 were live in producSec-tion code is shown.

6.4 Acceptance of false positive rate for null dereference pat-tern detectors by developers of Snakeware

Three developers of Snakeware were interviewed to determine if the false posi-tive rate was acceptable for the developers of Snakeware. The developers used the null dereference bug pattern detector for two weeks. After the two weeks, the developers were interviewed.

All three developers applied the bug pattern detectors to at least two li-braries. The libraries the developers applied the null dereference bug pattern detectors to are as follow:

• Snakeware.I; • Snakeware.H; • Snakeware.Q; • Snakeware.S; • Snakeware.C; • Snakeware.G;

All three developers did nd false positives. All developers of Snakeware were instructed about how to use the bug pattern detectors in Microsoft Visual Stu-dio's Code Analysis. One of the instructions was to build the source code in release mode. The MSIL generated diers in release and debug mode. The bug pattern detectors were targeted to the release mode build. One of the three developers ignored this specic instruction, and as a result he found more false positives, see "Dev #3" from Table 9.

Library Dev #1 Dev #2 Dev #3

Snakeware.H 1 1 -Snakeware.I - 1 -Snakeware.Q 0 - -Snakeware.S - - 6 Snakeware.C - - 35 Snakeware.G 1 -

-Table 9: False positives found by developers of Snakeware

All developers responded positively about the false positive rate and stated that they nd the false positive rate more than acceptable. All three developers also stated that they will use the tool in the future.

7 Discussion

7.1 Applying null dereference bug pattern detectors to the current revision

In Table 5 and Table 7 it is shown that there are still null dereferences in the current libraries of Snakeware. However, they did not occur yet because otherwise they would have been xed. Both the null dereference bug pattern detector implementation and ReSharper 8.1 did nd a signicant number of potential null dereferences. The null dereference bug pattern detector imple-mentation found a total of 173 warnings while ReSharper found 230 warnings. ReShaper found more warnings but has more false positives: ReSharper had 27% false positives while the bug pattern detector implementation had only 6%. As such the null dereference bug pattern detector implementation has a better score. The null dereference bug pattern detector diers from ReSharper in the handling of parameters. See Section 6.1 for the dierence in parameter handling. As a result, the false positive rate is kept low while the null derefer-ence detector implemention is kept generic, except for the parameter handling. ReSharper and the null dereference bug pattern detector rules reported only a few of the same warnings. Snakeware.I had 1 match, Snakeware.H had 0 matches and Snakeware.Q had only 7 matches. These numbers are interesting because apparently the two tools nd dierent potential null dereferences. For possible causes of these dierences see Section 7.4.

Initially ReSharper would have been used as a baseline to compare results of the null dereference bug pattern detector rules against. As we learned, the results do not match making it dicult to compare the results against each other. We can, however, compare the false positive ratio. When we do, the implementation of the null dereference bug pattern detector rules has a better false positive score because of the knowledge that was available about the li-braries to be analyzed.

The results from ReSharper and the null dereference bug pattern detector can be combined, except for these 8 matching bugs, resulting in 323 real null dereference bugs in the current revisions. All reports are manually analyzed to determine if they are real bugs.

Real issues detected

ReSharper 168

ND bug pattern detector 163 Total null dereferences 331

Substract matches 8

Total real null dereferences 323

Table 10: Real null dereferences detected by ReSharper and the ND bug pattern detector

In Section 7.1 it is shown that both tools nd slightly more than 50% of the total null dereferences that is found by combing the results of the two tools.

7.2 Detect null dereference bug patterns in past revisions

In Table 8 it is shown that 14 of the 16 solved bugs would have been found by the bug pattern detector rules.

The two bugs that were not found were of the same type. The value in the analysis was set to Unknown for parameters. These two bugs were caused by a null parameter. However, because of a trade-o made to set parameters values to Unknown, the bugs were missed. These bugs are false negatives.

The other bugs that Snakeware xed in previous revisions were all found by the analysis.

7.2.1 SQ1

The rst sub-question SQ1: "Can null dereference pattern detectors nd a sucient number of null dereference bugs?" can now be answered. The null dereference bug pattern detectors nd more than 50% of the null dereference bugs, nding 87.5% of the null dereference bugs in this study. Therefore, SQ1 can be answered armatively.

However, the number of null dereference bugs to detect was limited. There were only sixteen null dereference bugs to detect by the null dereference bug pattern detector. Future research should apply the null dereference bug pat-tern detectors to a greater number of xed null dereference bugs.

7.3 Acceptance false positive rate for null dereference pattern detectors by developers of Snakeware

In Section 6.4 it was indicated that three developers of Snakeware nd the false positive rate acceptable.

7.3.1 Low number of participants

Only three of the six developers of Snakeware were able to participate in the interview. The other three developers did not have time for an interview. The low number of participants could inuence the acceptance of the false positive rate.

7.3.2 Developer "3"

Developer "3" is "Dev #3" from Table 9. Developer 3 did not apply all in-struction correctly resulting in a higher number of false positives. He applied the bug pattern detector rules to assemblies that were built using debug mode in Visual Studio, resulting in other MSIL. However, he still believed the false positive rate was acceptable. After the interview we applied the bug pattern

detectors to "release mode" built assemblies. The false positive rate was bet-ter. In Snakeware.C the null dereference bug pattern detector found 5 false positives in release mode, as compared to the 35 found in debug mode. The numbers for Snakeware.S also improved from 6 false positives to 2 false posi-tives.

7.3.3 Observations

Snakeware was using the default bug pattern rules from Microsoft prior this research. While using the default rules Microsoft implemented, the develop-ers of Snakeware did not see the forest for the trees. This was caused by a high number of warnings. The developers of Snakeware believe many of these warnings are false positives. Not all warnings generated by the default rules of Microsoft were investigated during our research, so a conclusion can not be drawn about the false positives ratios. The experience of the developers can inuence how they feel about the lower number of false positives found by the null dereference bug pattern detector rules implemented during this research. One of the developers even referred to the high number of false positives for the default rules of Microsoft. Research from B. Johnson et al.[17] shows that static code analysis for bug-nding is underused because of the high number of false positives and high number of warnings reported.

Another observation made by Pier Janssen, one of the developers of Snake-ware is that the false positives of the null dereference bug pattern detector occurred in code with questionable design. So the false positive does also let a developer rethink the implementation itself.

7.3.4 SQ2

The second sub-question SQ2: "Will the frequency of false positives be kept acceptable to developers?" can now be answered. All three interviewed devel-opers of Snakeware stated that the false positive rate is acceptable. Therefore, SQ2 can be answered armatively.

7.4 Dierences between null dereference bug pattern detector rules and ReSharper

ReSharper is a tool built by Jetbrains to analyze code quality, and eliminate errors and code smells. ReSharper is made to work on all available C# source code. While the bug pattern detector implementation is built for the libraries of Snakeware, however, the null dereference bug pattern detector implemented is made generic, just like ReSharper. Except for the parameter handling. See Section 6.1.1 for parameter handling dierences. Implementing these parame-ter assumptions can inuence the false positive rate in a constructive way for Snakeware. The number of false positives is reduced.

pattern implementation does not analyze anything outside the current assem-bly while ReSharper does. Consider an example: if method W from assemassem-bly X is analyzed, method call from W to Y where Y is a method of assembly Z, then the result value is assumed to be "Unknown" in the bug pattern detector, while ReSharper will determine the return value of a method call outside the current assembly. Therefore the bug pattern detector will not nd possible null deferences caused by the result of a method call to a method from another assembly.

7.5 The period a null dereference bug is live in production

In Section 6.3 it was explained that a null dereference bug, at Snakeware, is live in production with a median of 84 days. If the Code Analysis would have been available when these bugs were introduced, the source code would not compile and the developer would instantly knows a null dereference bug pattern was detected. The bug pattern detector rules also tell the developer exactly where the null dereference bug pattern occurred in the source.

The developer knows two important things directly: • that there is a potential null dereference bug;

• the exact location of a potential null dereference bug.

It costs more time for a developer to nd and locate a bug himself, than it takes the null dereference bug pattern detector to nd the potential bug. Ap-plying the null dereference bug pattern detectors takes around a minute for the largest library of Snakeware.

The number of days between the bug introduction commit in SVN and the commit containing the bugx, includes time to x the bug. We make the as-sumption that the required time for a null dereference bug-x is equal in both scenarios, bug-nding by applying the null dereference bug pattern and debug-ging applied by the developer. The time required to x the null dereference bug is therefore not relevant to determine the eectiveness in this particular case.

The bug xes for the sixteen bugs have been investigated to determine if the x is simple or (more) complex. Snakeware xes null dereferences by adding if guards, adding Throw.IfNull or, if possible, changing the method call to an extension method. In the extension method, the null value is handled correctly. In this case the extension method is syntactic sugar. The extension method is a disguised if guard. The Throw.IfNull is also syntactic sugar, as it is also a disguised if guard. All applied bug xes are relatively simple.

Applying these bug xes is not time consuming and it is denitely not the cause for null dereference bugs being live for an average of 91 days. The most time is used locating the null dereference bug. The null dereference bugs can

be found more eectively with the null dereference detector rules than by the developer himself.

7.5.1 SQ3

The third sub-question (SQ3): "Is null dereference pattern detecting using static analysis an eective technique for nding null dereferences?" can now also be answered armatively.

The two parameter null dereference bugs that were missed in Section 6.2.1 should not occur in the future because of the implementation of Throw.IfNull in combination with the immutable parameters. Snakeware forces parameters to be immutable by a custom code analysis rule.

7.6 Answering of the main question

Now to consider the main question: "Can simple analysis techniques eectively nd null null dereferences in the object oriented programming language: C#?". To answer the main question we rst had to determine what "simple" and "eectively" mean. The meaning of "simple" was introduced in Section 2.4. We want a relatively simple analysis to detect possible null dereferences. The meaning of "eectively" was introduced by a sub-question (SQ3). See Section 4.2.3 for a description of "eectively" in the context of this research. Based on the results we gathered throughout this research we can also an-swer the main question armatively. With simple analysis techniques we can eectively nd null dereferences in C#.

7.7 Replication

The replicated study [16], demonstrates that relatively simple techniques can be used to eectively nd bugs. That study's research is based on Java while this research is based on C#. In this new exploration of the research, a new framework is built to implement bug pattern detectors. However, in this re-search the existing framework of Microsoft (Microsoft's Visual Studio Code Analysis) is used as a basis to implement custom bug pattern detectors. In the original research, the researchers built bug pattern detectors for dif-ferent kind of bugs. In this research, we have focused on null dereferences in particular. The available time for this master's thesis research is too limited to investigate all the bug patterns the original research investigated.

7.8 Threats to validity

7.8.1 Sub-question one

To answer sub-question one, only sixteen xed null dereferences issues were used to run the null dereference bug pattern detector against. The null deref-erence detector should be applied to more xed null derefderef-erence issues to in-crease the credibility of the results.

The same dataset has been used to implement the null dereferences and to run the analysis on. If the null dereference pattern slightly changes, the null dereferences can potentially not be found by the null dereference detector rules. However, this is inherent at using patterns to detect bugs.

7.8.2 Sub-question two

Snakeware has four employers who program only. Two other programmers are also project leaders. So a total of six programmers. All six were invited to participate in this research, however, only three developers were able to par-ticipate in the interviews to measure the acceptance of the false positive rate for the null dereference bug pattern detectors.

Another threat to validity for the acceptance of the false positive rate is the experience developers of Snakeware have with default rules from Microsoft. As described in Section 7.3.3 the developers of Snakeware experience a high num-ber of warnings generated when the default rules of Microsoft Visual Studio Code Analysis are applied. The developers are biased by their experience. The null dereference bug pattern detector detects a single issue: null dereferences while the default rules of Microsoft detect a lot of issues resulting in more warnings reported, including false positives. Two of the three interviewed de-velopers even mention the comparing between the high number of reports of the default rules and a much lower number of detected warnings by the null dereference bug pattern detector.

7.8.3 Sub-question three

The bug patterns implemented to detect the null dereferences are based on the null dereferences xed by Snakeware. Each particular null dereference bug is analyzed to determine a pattern. These implemented pattern detectors detect the bug pattern. Cases were there is a dierent pattern still go undetected. These missed patterns are left out of the measurement of eectiveness.

8 Conclusions and future work

The results we gathered support the original research [16]. Null dereference bug patterns detectors are a good and simple technique to detect null derefer-ences. In the future, dierent kinds of bug patterns could be implemented to detect dierent bugs.

We have made a trade-o to keep the false positives rate very low. As a consequence we can miss potentially real null dereferences. A lter for a level of certainty would improve user experience because potentially more null deref-erences can be reported. If detected bug patterns could be rated, a developer can adjust the certainty level to increase or decrease the possible number of warnings generated.

In this research false negatives are left out due project time restrictions. As a consequence we can not draw any conclusions about possible missed bug pat-terns. We can however conclude that applying simple static analysis techniques will detect a high number of null dereferences that otherwise go undetected. The libraries of Snakeware are at work in applications and websites that are heavily used. Still 323 null dereferences did go undetected, showing the con-tinuing need for new and eective methods of bug detection.