Efficient Approximate JavaScript Call Graph Construction

(1)

Efficient Approximate JavaScript Call

Graph Construction

A replication study

Sander Benschop

University of Amsterdam sander.benschop@gmail.com

February - July 2014, 58 pages

Supervisor: Tijs van der Storm

Host organisation: Centrum Wiskunde & Informatica

Universiteit van Amsterdam

Faculteit der Natuurwetenschappen, Wiskunde en Informatica Master Software Engineering

(2)

JavaScript has seen an increase in popularity in the last few years, both in the browser as well as on other platforms such as Node.js. However, the tools to help developers reason about JavaScript code remain fairly barebone in comparison with tooling for static languages such as Java. These tools are difficult to implement for JavaScript because of the dynamic nature of the language.

Feldthaus et al. describe the difficulty of static analysis for JavaScript. They identify efficient construction of call graphs as a key requirement to improve development tools. JavaScript call graphs are computationally expensive to build and existing solutions do not scale enough to be useful in an interactive environment. The authors present a scalable field-based flow analysis for constructing call graphs. This analysis, while in principle unsound, is said to produce highly accurate results in practice. Two different static algorithms are proposed: a pessimistic and an optimistic variant. The pessimistic algorithm only reasons about interprocedural flow in the case of one-shot closures like "(function( ¯x) { ... })( ¯e)". The optimistic algorithm reasons about all interprocedural flow but may resolve callback invocation imprecisely. The accuracy of the static algorithms is measured by comparing the static call graphs with dynamic call graphs recorded by running an instrumented version of the program.

In this master research project these algorithms are replicated in Rascal, which is a metaprogram-ming language developed at the CWI. The goal is to find out if the static call graph algorithms recreated from the specification in the original article produce equally accurate results as the ones from the original research. The algorithms are run on the same source programs as in the original research, though different versions are sometimes used as the versions used in the original research are unknown. Furthermore, two of the programs analyzed in the original research could not be analyzed completely, to compensate for this fact two other programs have been analyzed. The static call graphs are compared to the dynamic call graphs to calculate the accuracy of the algorithms as is done in the original research.

The call graphs created by the replicated algorithms were similarly accurate as the ones from the original paper if the accuracy is measuring by averaging these numbers over call sites. However, calculating the accuracy in this way rather than comparing the entire call graph edges can cloud differences between the static and dynamic call graph and increase precision. When the entire call graph edges are compared rather than call graph edges, the precision is in some cases considerably lower. Imprecision in the call graphs has the same causes as in the original research: most of it is a result of the fact that the algorithm merges all equally named properties into equal call graph vertices. This approach is chosen to speed up the algorithm by reducing the size of the total flow graph but has a negative effect on accuracy. The effect appears smaller when comparing call graph targets with each other rather than call graph edges.

After inspecting the results of the call site averaging comparison it seems they point in favor of the claim made in the original paper that both algorithms produce accurate call graph in practice. When the call graph edges are compared instead however these numbers are often less favorable. Comparing the call graph edges with each other provides a more realistic view of how similar the static and dynamic call graphs are in their entirety. In the original paper they already describe in the future work section that the algorithm’s accuracy could be increased by taking dynamic property access and the tracking of objects other than functions into account. When comparing call graph edges the need for increasing the accuracy mentioned in the original paper becomes even more relevant.

(5)

Preface

This master thesis is the result of my research project for the master Software Engineering at the University of Amsterdam. The research was performed at the Centrum Wiskunde & Informatica.

I would like to thank Tijs van der Storm for supervising me during the course of this master research project. His valuable feedback helped me to improve the quality of the thesis.

Furthermore I would like to thank Jurgen Vinju and Davy Landman for their help and insightful tips throughout the course of the project.

I would also like to thank Jorryt Dijkstra, a fellow master student who also performed a replication of this study, for the pleasant co-operation during the development of the JavaScript grammar as well as the interesting discussion throughout the course of the research project.

Jesse van Assen and Ruud van der Weijde reviewed my thesis draft which helped me to improve the quality of the thesis. Their outside view on the matter helped me to gain insight in what aspects should be explained in more depth. Thank you for this.

Finally I would like to thank my employer 42 BV for allowing me time to work on this master research project.

Sander Benschop Amsterdam, July 2014

(6)

Chapter 1

Introduction

For this master project I replicated a study by Feldthaus et al. [8] titled "Efficient Construction of Approximate Call Graphs for JavaScript IDE Services". In this paper they describe the that though JavaScript has seen a recent rise in popularity both in the browser and other platforms, the tools to help developers reason about JavaScript code remains fairly barebone in comparison with their counterparts for static languages such as Java. These tools are meant to help developers reason about the code they are working on, but the dynamic nature of JavaScript is hampering progress on improving these tools.

They identify the difficulty of constructing call graphs as a key impediment to advanced tooling. From a call graph the functions that a given call may invoke at runtime can be determined. Integrated Development Environments (or IDEs) use such reasoning for basic features like "Jump to Declaration" and also for refactoring or analysis tools that need to reason interprocedurally [8]. To be useful in an IDE however a call graph construction algorithm must be lightweight and scalable. Programmers expect IDE services to be constantly available so the call graph information should be quick to compute. Current solutions however work by using heuristics that unexpectedly fail. More sound solutions do not yet scale to process full framework-based JavaScript programs [8].

In an attempt to mitigate these problems and provide possibilities for static analysis of JavaScript programs, Feldthaus et al. [8] have created two different algorithms that create call graphs from ab-stract sytax trees. The pessimistic variant only reasons about interprocedural flow in the special case of one-shot-closures, which are anonymous functions that are directly applied to some arguments, like this: "(function( ¯x) { ... })( ¯e)". The optimistic variant takes all interprocedural flow into account.

They claim that though both these algorithms create approximations of call graphs, they are sufficiently accurate for applications such as IDE services. In order to measure the precision and recall, dynamic call graphs are created against which the static call graphs are compared.

The goal of this replication was recreating the algorithms from the specification provided in the original paper in, implementing them in the meta-programming language Rascal. Then the precision and recall is calculated for all projects to see if the results match the results from the original article. Though the results are sometimes close to the ones from the original research, the precision is in no case equal. For the recall this is only the case for one source program. In the original study in all cases the precision of the pessimistic call graph is either higher or equal to the optimistic variant. This is not necessarily the case with the results of this replication, where the opposite situation does occur. The reason this happens is because of the way the precision is calculated, by calculating the precision of all call sites present in the dynamic call graph and adding this all together. At the end this accumulated value is divided by the number of call sites in the dynamic call graph. When a call site is not present in the static call graph this will also reduce the precision. This happens more in the pessimistic call graph because it contains fewer call sites as not all interprocedural edges are taken into account. Another disadvantage of the way the original authors chose to cal-culate the precision and recall is that because it is an average, outliers can heavily impact the outcome.

(7)

This can cloud differences in the call graphs. When comparing entire call graph edges are compared this does not happen as it is not an average. The outcome of this comparison can provide a better insight in the similarity of the static and dynamic call graph as a whole.

(8)

Chapter 2

Problem analysis

2.1 JavaScript analysis

JavaScript has seen an increase in popularity in the last few years both in the browser as well as on other platforms such as Node.js. However, the tooling to help developers reason about JavaScript code remains fairly barebone in comparison with tooling for static languages such as Java. These tools are meant to help developers reason about the code they are working on, for example by providing code completion or a possibility to jump to the declaration of a function. Other use cases are automated refactoring support, which helps developers make changes to their code without affecting the semantics of the program, and code analysis tools which need to reason interprocedurally [8]. These tools are difficult to implement for JavaScript because of the dynamic nature of the language. The authors of the original paper identify the difficulty of constructing call graphs as a key impediment for such tooling. These call graphs can show what functions a given call may invoke at runtime. Because the developers use these tools want instant feedback, the call graph construction algorithm must be lightweight and scalable. It should be quick to compute so developers have these services constantly available. Current solutions however work by using heuristics that unexpectedly fail. More information about other solutions that provide alternatives to the algorithms described in this paper can be found in section3.3.

2.2 Replication challenges

There are several challenges with the replication of this project. First of all, a grammar had to be implemented from which a JavaScript parser for Rascal can be generated. This parser needs to be able to parse large JavaScript programs and libraries such as jQuery because these are used in the programs analyzed in the original paper. The original authors built their algorithms in CoffeeScript and used Esprima to parse the programs. This parser adds scoping information to the abstract syntax tree. A custom grammar for JavaScript written in Rascal was used to parse scripts in this replication. This meant that the JavaScript scoping model had to be implemented manually, see section9.1for more information.

Another challenge is the fact that there are aspects of the algorithms underlit in the original paper. An example is the process of rewriting a file to obtain dynamic call graphs. No details are given on this subject even though retaining the semantics of the original program can be challenging. The optimistic algorithm is another example, there is little information in the paper on how this works while its pessimistic counterpart is explained at length.

Unfortunately the source code of the algorithms in the original paper are not being made publicly available due to intellectual property issues. This could help understand parts of the specification when in doubt. Instead the authors created another implementation in Node.js but this sometimes acts differently than specified in their paper. For the same reason the sources of the analyzed

(9)

programs have not been made available either. If these sources would have been provided it would be possible to run the algorithms on them to see if the same output is produced as in the original experiments.

2.3 Added value

Researching the efficient construction of JavaScript call graphs is worthwhile, because this could create possibilities for more sophisticated tooling to gain insight in a codebase or refactor existing code [8]. Refactoring is the process of changing a software system in such a way that it does not alter the external behavior of the code yet improves its internal structure [9]. This practice has become an integral part of software methodologies such as Extreme Programming (XP) [3,7].

For each refactoring a set of preconditions is checked before the set of algorithmic steps is taken to transform the program. It is tedious and error-prone to do this manually [7]. For this reason IDEs often automate refactorings. Though the demand for such tools is growing, the dynamic nature of JavaScript makes the implementation of these tools challenging. The call graphs produced by the algorithms mentioned in this paper could be used for such features if they are sufficiently accurate. For example: a complete sound call graph could show exactly in what places a function is called. A refactoring rename could then safely change the name of the function and all its references.

2.4 Research question

The first part of the research consist of recreating the algorithms from the specification. After the implementation of the algorithms the analysis phase starts, in which the following research question is answered:

"Do the replicated static call graph creation algorithms analyze the source programs with the same level of accuracy as in the original research?".

The accuracy of the call graph is measured by its precision and recall. Precision is the percentage of ’true’ call graph elements, while recall is the percentage of correctly identified true elements.

(10)

Chapter 3

Background and context

3.1 Call graphs

According to the famous ’Dragon book’ [1] Compilers: Principles, Techniques, and Tools: a call graph for a program is a set of nodes (also known as vertices) and edges such that:

1. There is one node for each procedure in the program

2. There is one node for each call site, that is, a place in the program where a procedure is invoked 3. If call site c may call procedure p, then there is an edge from the node for c to the node for p. In languages like C and Fortran procedure calls are made directly (meaning not dynamically dispatched to a subtype like in object-oriented languages), and thus the call graph target of each invocation can generally be determined statically. In this case each call site has an edge to exactly one procedure in the call graph. Indirect calls are the norm for object-oriented programming languages however. When methods are overridden in subclasses, a use of method m may be dispatched to any of a number of different methods, depending on the subtype of the object the function is called on. It is not possible to know the exact method that is invoked without knowing the type of this receiver object [1].

Another problem is the use of procedure parameters, which are functions passed as an argument to other functions. The target of these parameters are generally not known until the program is run and may vary from one invocation to another. Then, a call site can link to many or all procedures in the call graph [1].

JavaScript is a language that supports passing functions as arguments. The language contains a prototype-based inheritance model. Objects can be created from prototype objects which can in turn be built from other prototype objects. This creates a prototype chain which ends in a null prototype. When a function m is called on an object and that object does not contain a property of that name the prototype chain is traversed until a property of that name is found or the null prototype has been reached [14]. This way, functions can be overridden just like in class-based object-oriented languages. This means that in JavaScript the same problem is present as with dynamic dispatch: in order to know the function that will be invoked it is necessary to know the type of the object it is called on.

(11)

3.2 Control flow analysis

Grove and Chambers [11] have written an article in which they propose a framework for call graph construction algorithms. In this paper it is also explained that while call graph analysis for procedural languages such as C or Fortran is straightforward this is more difficult in object-oriented languages and languages where functions can be passed as parameters (see section3.1).

To build a useful call graph it is necessary to determine the flow of values. There is however a circular dependency present [8,11]. Determining the flow of values requires an interprocedural data and control flow analysis of the program. But interprocedural analysis in turn requires that a call graph be built prior to the analysis being performed. This circular dependency is the key technical difference between interprocedural analysis of object-oriented and functional languages and interprocedural analysis of strictly first-order procedure languages [11]. They propose two different ways of breaking this circularity:

• Making a pessimistic (but sound) assumption. This approach breaks the circularity by making a conservative assumption and continuing from this. In the pessimistic algorithm of the original authors, the conservative assumption is the usage of a triple which contains an initial call graph created from all the one-shot calls to the functions they are enclosing, a collection of other (escaping) functions and a collection of other (unresolved) calls. This call graph consisting of one-shot closures is the basic for the rest of the analysis.

• Making an optimistic (but likely unsound assumption), and then iterate to a sound fixed-point solution. Just as in the pessimistic approach, an initial guess is made. In the case of the optimistic algorithm created by the original authors, this guess is the empty triple. The analysis is instead performed by seeking paths from functions to callees. The fundamental difference with the pessimistic approach is that because the initial guess may have been unsound, after the initial values are computed further rounds of iteration may be required to reach a fixpoint. This optimistic approach can create more complete call graphs than the optimistic one, but it is more complicated and computationally more expensive.

In the paper Grove and Chambers describe a hypothetical ideal call graph, that is the most sound and precise call graph possible. This can be seen as an upper bound for the precision of call graphs, and can be used for comparison with others. Unfortunately this ideal call graph is generally uncomputable. However, a loose upper bound can be established by using profile data to build an exact representation of the calls that occurred during one particular run of the program [11]. In the original paper [8] these are referred to as dynamic call graphs. This strategy is also described in another paper called "Comparing call graphs" [13] in which they explain that a good way of detecting missing edges is by comparing against dynamic call graphs.

3.3 Alternative algorithms

Call graph construction for JavaScript is difficult. Java IDEs take a type-based approach to call graph construction [5,8]: the possible targets of a method call are simply those admitted by the program’s class hierarchy. In JavaScript however, values are not statically typed. Though prototype objects are somewhat similar to Java classes, properties can dynamically be added or overwritten [8]. More complicating factors exist, such as the fact that JavaScript supports the passing of functions as arguments to other functions (higher-order-functions) and the fact that it supports arity mismatching, which means that a function can be called with an arbitrary number of arguments.

Type inference algorithms exist that can handle these complications [12] and deliver sound and precise call graphs but they don’t scale to real world programs [8,12]. One of the programs analyzed in a paper on type analysis for JavaScript [12] explains that of the programs they have benchmarked the largest causes out-of-memory errors and another runs for over six minutes, which is too long to be practical in an IDE. This information could be beneficial during call graph construction in order

(12)

to help remove incorrect edges. For example, if for an object it can be inferred that is of type Boolean and a call to a toString method is performed on it then we know that the function that is called is Boolean.prototype.toString and not Array.prototype.toString or Number.prototype.toString.

A flow-based solution other than 0-CFA is points-to analysis that statically approximates the flow of data to reason about function calls [8]. As shown in previous work however [15] this technique does not yet scale to real-world programs. One of the reasons for this performance problem is dynamic property access: performance scales badly when the numbers of properties increase. The dynamic features of JavaScript cause an increase in the worst-case running time of points-to analysis, which becomes O(N4), where N is the program size. This asymptotically fast growth rate makes it virtually impossible to analyse common JavaScript frameworks like jQuery. The authors of the original paper discovered in this previous work that often dynamic property access is correlated [15]. By this they mean that the value of property p is copied in one object to property p of another, like this:

1 var destination = {}; 2 for (property in source) {

3 destination[property] = source[property]; 4 }

To handle dynamic property access, the analysis must track the possible values of property-name expressions (like property on line 3) and use that information to reason about what properties a dynamic access can read or write. This value property could be bound to the name of any property of any object bound to source. The points-to set for the dynamic property expression source[property] consists of all properties of the source object. The same applies to the destination object. This causes a standard Andersen points-to analysis [2] to add all of the properties in source the points-to sets of all of the properties in destination [15]. Taking this approach would create many spurious edges in the flow graph.

In this case however the dynamic property read and write are said to be correlated, as the value that is read on the right side is written to the left side and they refer to the same name. Since the value of property cannot have changed between the read and the write, an enhanced analysis could reason that the value source[property] may only be copied to the property with the same name in destination. Instead of adding an entire points-to set consisting of all the properties in source to all the properties in destination, the sets are reduced to only contain equally named properties. This reduces the number of locations that need to be tracked, both increasing the performance and precision of the algorithm [15]. Some scalability issues still remain however, and even with the improved scalability the performance is not sufficient to use the analysis in an interactive environment such as an Integrated Development Environment (IDE) [8].

(13)

3.4 JavaScript

JavaScript is a dynamic programming language. It supports object-oriented programming as well as first-class functions and is most known as the scripting language for web pages but is now also commonly used in non-browser environments such as Node.js. JavaScript contains a prototype-based inheritance model in which inheritance is performed via a process of cloning existing objects that serve as prototypes [14].

In JavaScript any value can implicitly be converted to a boolean value. Values are commonly said to be either truthy or falsy. An example of this implicit conversion occurs if a value is passed to an if-statement. These implicit conversions are also important for the flow graph, as they influence the flow of data which will be made clear in chapter6. The EcmaScript specification [6], to which JavaScript conforms, defines the following boolean conversion table:

Argument type Result Undefined false

Null false

Boolean the result equals the input argument (no conversion).

Number the result is false if the argument is +0, -0, or NaN (not a number); otherwise the result is true.

String the result is false if the argument is the empty String (its length is zero); otherwise the result is true

Object [including functions]

true

In JavaScript an object can be extended dynamically by simply assigning values to properties [15], like this:

var object = {}; //Empty object

object.sayHello = function() {

return "Hello world!"; }

object.sayHello();

Another notable feature is that a function can be called with any number of parameters, a principle called arity mismatching [15,12]. Consider for example the following piece of code:

function f(x, y) {

return "X: "+ x + " Y: "+ y; }

f(1,2) //Returns "X: 1 Y: 2"

f(1,2,3) //Returns "X: 1 Y: 2", third argument is ignored.

f(1) //Returns "X: 1 Y: undefined"

f() //Returns "X: undefined Y: undefined"

All these function calls are valid and will return a value. Both of these features can effect the performance and soundness of static analysis in JavaScript.

(14)

Chapter 4

Original research

The intent of the authors of the original paper [8] was to "create a lightweight flow analysis specifically designed to efficiently compute approximate call graphs for real-world JavaScript programs". The algorithms take an abstract syntax tree as input and produce a call graph as output. In order to speed up the process a few concessions are described:

• The analysis is field-based, meaning that two properties of the same name will be represented with an identical vertex in the flow graph. As a consequence, two functions that are assigned to properties of the same name will become indistinguishable as call graph targets. For example, for the following piece of code the analysis will not be able to reason that the function it is referring to is on the object1 object and will return edges from the same callsite object1.f() to both functions f on object1 and object2.

var object1 = {

f : function() { return "Foo"; } };

var object2 = {

f : function() { return "Bar"; } };

object1.f();

• It only tracks function objects and does not reason about any non-functional values. For example, in the following snippet the analysis cannot reason about the value of property as string values are not tracked.

var object = { "a" : 0, "b" : 1 };

var property = "a";

object[property]; //Analysis does not know that the value of "property" is "a"

• It ignores dynamic property accesses, i.e. property reads and writes using JavaScript’s bracket syntax. Because the flow graph only tracks function objects, the algorithms cannot reason about a dynamic property access e[p]as can be seen in the example above. They state that this is unimportant since dynamic property access is often correlated [15]. Also, since the analysis uses a field-based approach equally named properties are merged anyway. The algorithms go further than track correlations between reads and writes and ignore dynamic property access altogether.

According to the authors of the original paper these design decisions reduce the size of the flow graph and thus dramatically improve scalability. They describe two algorithms to statically create call graphs: a pessimistic algorithm which only reasons about interprocedural flow in the special

(15)

case of one-shot closures like "(function( ¯x) { ... })( ¯e)" and an optimistic algorithm which takes all interprocedural flow into account. Both algorithms operate on a flow graph which represents the flow of data through an application, for example by assigning values to variables or by returning a value from a function. The static algorithms work by analyzing code constructs without running in a browser or other JavaScript engine. Because the flow of values may be dependent on runtime variables, the static call graph cannot always reason precisely about what functions could be invoked at the different call sites.

4.1 Research questions

The goal of the original research is to create call graphs in an efficient manner to make them suitable to be used in an IDE. Because a programmer relies on these tools to help reasoning about code while programming, they need to produce accurate results. Programmers expect these functions to be instantly available, but the dynamic nature of JavaScript hampers the scalability of sound static analysis. The authors were not aware of any static analysis capable of computing useful call graphs for large JavaScript applications.

In the original paper a research question is not stated. However, there are three evaluation questions asked to assess the quality of the call graph algorithms:

• How scalable are our techniques?

• How accurate are the computed call graphs?

• Are our techniques suitable for building IDE services?

4.2 Design

To answer the evaluation questions the authors of the original paper have analyzed ten JavaScript programs. Some of these programs are based on popular frameworks at the time while others are built without using any frameworks. The programs are described as medium to large browser-based JavaScript applications covering a number of different domains, including games, visualizations, editors, a presentation library, a calendar application and a PDF viewer.

To answer the question about the scalability of their techniques the time it takes to build call graphs for the subject programs of both algorithms was measured. The time it takes to parse every program was also measured separately. Timings were averaged over ten runs.

Measuring the accuracy of the computed call graphs is more difficult, as there is no existing sound analysis that can handle all the subject programs against which the results can be compared. In order to mitigate this problem they create dynamic call graphs by instrumenting the subject programs using a dynamic rewriter to record the observed call graph targets for every call that is executed at runtime. These instrumented subject programs were then manually executed. In order to get an idea how complete the dynamic call graph is, the function coverage was measured. This is the percentage of non-framework functions that were executed while recording the call graphs. The static call graph is then compared with the dynamic call graph to calculate the precision and recall. Precision is the percentage of ’true’ call graph targets among all targets, while recall is the percentage of correctly identified true targets. The call graphs created by the authors of the original paper are publicly available1.

Lastly, the suitability for IDE services was analyzed by comparing how well the algorithms do in comparison to three current JavaScript IDEs when analyzing the subject programs. This was measured by checking how many call graph targets the different tools could come up with in

(16)

comparison to the algorithms of the authors. Furthermore some other applications for the algorithm are mentioned: smell detection and bug finding.

4.3 Summary of the results

The authors of the original paper claim that both call graph construction algorithms scale very well, they are able to build call graphs for substantial applications in a few seconds. Comparing against dynamic call graphs, they found that the vast majority of dynamically observed call graph targets are predicted correctly by the analyses. On average the number of incorrectly identified calls is low. For almost all call sites, at most three possible targets are computed with the exception of the native function toString which has more than 100 call graph targets due to the field-based approach.

They claim that both analyses achieve almost the same precision on most programs. For both approaches the main sources of imprecision are claimed to be functions that are stored in properties of the same name, which are not distinguishable as call graph targets due to the field-based approach. The optimistic approach may also resolve callback invocations imprecisely. In this case the pessimistic analysis gives up which results in a higher precision. In most cases the precision of both algorithms is at least 80%. On two programs the precision is more modest. Beslimed, a game powered by the mootools framework scores 77.8% in the pessimistic analysis and 69.1% in the optimistic analysis. The other application is flotr, a plotting utility built on the prototype framework which scores 71.7% in the pessimistic analysis and 66.3% in the optimistic analysis.

Both analyses are claimed to achieve very high recall; in every case more than 80% of dynamically observed call graph targets are also found by the analysis. Missing call graph targets are claimed to be due to the unsoundness of the approach with respect to dynamic property writes, which is often used by framework code according to the authors. An exception is one programs called flotr in which the optimistic algorithm does significantly better, supposedly due to a liberal use of callback functions which are not handled by the pessimistic analysis.

(17)

Chapter 5

Approach

In the original paper, ten programs have been analyzed. Unfortunately the authors of the original paper could not supply the sources of these analyzed programs. They also didn’t always specify what versions were used or they used programs from unversioned sources (see section 11.2.3). They did however specify when they downloaded the files. In case the sources were available on a versioned location such as GitHub this made it possible to download the version that was the most recent at their time of downloading and assume they have used this one.

Of these ten programs one of the games called pong was broken on download and for a calendar application fullcalendar the optimistic analysis did not finish even after 35 hours. The other eight applications analyzed in the original paper were analyzed in this replication as well. By comparing the results of these eight applications with the ones from the original paper the internal validity of the original research is tested. To compensate for the two libraries that could not be properly analyzed two extra applications have been analyzed. Since these are not analyzed in the original paper they cannot be compared with the original results but running the algorithm on different sources can some give insight whether or not the conclusions drawn on the original set of programs also holds for others. Two of the source programs, beslimed and flotr, had to be analyzed without including the underlying frameworks as the rewriter broke these frameworks.

The JavaScript grammar is not completely finished, some manual rewriting of the sources remains necessary prior to analysis. Examples are the need to remove comments and splitting out some types of multiple statements placed on the same line. This prepared source program becomes the basis of the analysis, it is both analyzed by the static algorithms to create the pessimistic and optimistic call graphs. These adapted sources are also used as input for a dynamic rewriter, which creates an instrumented version of the program which records calls executed at runtime. This instrumented program is run and the call graph it produces is called the dynamic call graph.

These static call graphs are then each compared to the sound but incomplete dynamic call graph to calculate the precision and recall of the static call graphs. Precision is the percentage of ‘true’ call graph targets among all targets whereas recall is the percentage of correctly identified true targets.

(18)

5.1 Level of interaction with original researchers

Both me and another master student have been doing a replication of the same study at the CWI. We have asked the original researchers if they could send us the implementation of the algorithms they used for the research, but this was not possible due to intellectual property issues. We have also asked them for more information on how the dynamic rewriter works, as this is not clearly explained in the paper. They explained that it worked by recording call sites and targets, but did not go more in-depth than that. In practice it turned out that the rewriting of files can become complex. We also asked them if they could provide this rewriter to us but because of the intellectual property issues this was not possible either. Because we noticed that in the original research they were achieving high precision on their call graphs even though they were only covering a portion of the libraries such as jQuery we asked them how this was possible and they explained that they only measure those call graph edges in the comparison that were covered in the dynamic call graph. This is done by looping over the call sites present in the dynamic call graph, computing the precision and recall for this call site and then taking an average.

5.2 Changes to the original experiment

The first evaluation question asked by the authors of the original paper is how scalable the algorithms are. As I will implement the algorithms in a completely different language, the performance results will not be comparable to the ones in the original paper. They are running their algorithms on the JavaScript V8 engine, a project which is backed by Google, written in C++ and highly optimized. I am running the algorithms in the metaprogramming language Rascal, developed at the CWI. This language makes analyzing language constructs very easy, but it is understandably not as fast as the V8 engine. This makes it difficult to give an answer to the question whether or not the tool is sufficiently fast to be used in an IDE, and thus this will not be covered in this replication.

Another evaluation question I will not cover during my master research project is whether or not the algorithms are suitable for IDEs. In the original paper they try to answer this question by comparing the usefulness of the jump-to-declaration functionality enabled by the call graph with those of several IDEs used in the industry. Of the three evaluation questions mentioned in the original paper, this third evaluation question clearly shows the added value of the algorithms, as they do a better job than current IDEs, but the suitability for use in an IDE mostly follows from the accuracy and performance of the algorithms. I think it is a good thing that the authors of the original article showed the added value of the algorithm but I think the other two questions are more important for replication.

The algorithm described by the original authors operates on an abstract syntax tree. The result of a successful parse in Rascal is a parse tree (also known as a concrete syntax tree), which contains more information. It is possible to transform the parse tree to an abstract syntax tree but since Rascal allows pattern matching of concrete syntax, operating on a parse tree sufficed for this project. Since all the information present in the abstract syntax tree is also present here, this will not cause any differences in results.

Two of the programs analyzed in the original paper, pong and fullcalendar, were not fully analyzed during this replication. Pong was already broken after downloading and was difficult to repair, whereas fullcalendar’s optimistic analysis did not finish after more than 35 hours. I therefore decided to skip analysis of both programs, and replaced them by two different games found on github: a small Space Invaders clone written without using any framework and a large Super Mario clone built on jQuery. For beslimed and flotr the frameworks were omitted from the comparison as these were broken by the rewriter.

(19)

Chapter 6

Static call graph analysis

6.1 Introduction

The core of the original research paper consist of information about the creation of the static call graphs. In order to implement the algorithms several steps have to be taken. First there needs to be a parser which can translate the programs to parse trees. A scoping model needs to be implemented to gain insight in what functions could be called at which places. After this is done, the intraprocedural- and interprocedural flow can be added to the flow graph. The flow from native functions to properties is then added to the flow graph. The last step in extending the flow graph from the parse tree is adding the interprocedural edges by connecting the arguments of a call site with the parameters of functions and the return value of a function with the result of that function call. In the optimistic analysis this is done for all connectable functions all call sites, in the pessimistic analysis this is only done for one-shot closures while the rest flows through the Unknown node. Finally an optimistic transitive closure can be computed the complete call graph can be extracted.

Rascal is provided with a grammar from which a parser is generated. The scoping model has to be implemented manually. The flow is added by matching concrete patterns in the trees by using visit-statements which recursively visits elements in a tree or value and is a language-level equivalent of the visitor design pattern [4,10] (for more information see section9.2). This flow graph consists of the same structure as the final call graph, a set of relations between vertices.

In the article it is nowhere explicitly stated how to deal with multiple files. In Rascal, it is also possible to traverse a list of trees rather than a single tree in a visit-statement and thus the support for multiple files is implemented by simply parsing all these files and traversing them together.

The parsing of the source code and the creation of the symbol tables are not integral parts of the algorithm, it is preliminary work necessary for the algorithm to function. The algorithm starts when the intraprocedural flow is added to a flow graph. An overview of the steps taken in both of the two algorithms is shown in figure6.1:

(20)

Add intraprocedural flow Add interprocedural flow Add native flow Which analysis? Create optimistic transitive closure

Extract call graph

end Add pessimistic interprocedural edges Add optimistic interprocedural edges Pessimistic Optimistic start

Figure 6.1: Steps taken in the call graph construction algorithms

The algorithm presented in the original paper operates on a flow graph which is extracted from the source code. This is a representation of the possible data flow induced by program statements. In a flow graph the vertices represent functions, variables and properties while the edges represent value assignments. Each of the program elements are part of the abstract syntax tree (AST). These are tree structures representing simplified versions of source code. Every program element has as a position π in the AST. The set of all AST positions is written asΠ. When the authors use the notation tπ_{they mean a program element t (such as an expression, a function declaration or variable}

(21)

callee of call at π ith argument of call at π ith parameter of function at π return value of function at π result of call at π

flow not modelled by the flow graph Figure 6.2: Explanation of the different vertex types present in the graphs.

The function V(tπ_{) maps expressions to corresponding flow graph vertices. If the function is called}

on a variable, either a Var vertex or a Prop vertex is returned, depending on whether or not the variable is available on the current scope. If the function is called on a property, a Prop is always returned. If it is called on another type an Exp is returned.

After all the necessary edges have been added to a flow graph the transitive closure needs to be computed to determine all the function vertices Fun(π) from which a call site π0 is reachable. To prevent the introduction of imprecision however, flows through the Unknown vertex are ignored. This custom transitive closure is referred to as an optimistic transitive closureoptG rather than a regular transitive closure G on a call graph G.

6.2 Scoping

In the original paper the authors mention the existence of a "function λ for local variables such that λ(π, x) for a position π and a name x returns the position of the local variable or parameter declaration (if any) that x binds to at position π". In this definition it is shown that if the lookup function succeeds in finding the variable, it returns a variable vertex. Otherwise, a property vertex will be returned.

The lookup function described in the original article is used to check whether or not a variable- or parameter declaration is in a given scope. If the name that is being sought is available on the scope, the location of the declaration is returned. However, if the variable or function is declared at the root level of the file (and thus not a local variable or parameter declaration) a Prop(x) is returned instead. JavaScript contains function-scoping [6], which means that everything defined inside a function is only available inside that function. Variables and function declarations can be referred before they are defined, as these are hoisted to the top of the function [6].

(22)

S0

S1

S2

1 f(42); //Possible, since functions are hoisted 2 g(); //Error, not available on this scope 3 function f(n) { 4 function g(m) { 5 console.log(n * m); 6 } 7 g(2); 8 }

This code snippet shows an example of how scoping works in JavaScript: functions can be called before they are defined (call to f ), but everything defined inside a function is only available inside that function and thus the call to g on the second line fails. Everything available in a higher scope is also available in a nested scope, thus the parameter of the function f of the name "n" is also available in the function g. The availability of the three levels of scoping is shown by the bars on the left side of the image.

m line 4, col 77-78

g line 4, col 66-109

n line 3, col 59-60

f line 3, col 47-121

parent

S0:

S1:

S2:

parent

The scoping model used in the analysis consists of a symbol table data structure [1] which is used to create a mapping from variable names to the location where these are defined. Symbol tables can be nested to represent the scope of nested functions. A variable is then simply sought by searching each symbol table and traversing upwards until either the name is found or the root scope has been reached.

6.3 Intraprocedural flow

The initial flow graph is created by adding intraprocedural flow, this means data flow within a procedure. In JavaScript procedures are called functions. The flow is added by traversing the tree and creating vertices when elements conforming to the patterns specified in the next table from the original paper [8] are found:

rule node at π edges added when visiting π R1 l = r V(r) -> V(l), V(r) -> Exp(π) R2 l || r V(l) -> Exp(π) , V(r) -> Exp(π) R3 t ? l : r V(l) -> Exp(π) , V(r) -> Exp(π) R4 l && r V(r) -> Exp(π)

R5 { f : e} V(ei) -> Prop( fi)

R6 function expression Fun(π) -> Exp(π) , if it has a name: Fun(π) -> Var(π) R7 function declaration Fun(π) -> Var(π)

Table 6.3: Intraprocedural flow

(23)

matching language construct is found in the parse tree. The value π refers to the location of the parse tree node that is currently being visited. For example, for a function declaration an edge is added to the flow graph from a Fun vertex to a Var vertex which are both at the same location π. For rules where a call to V is made, for example in the case of R1 the function V(tπ_{) is called which}

returns a vertex of the appropriate type (see section6.1).

For all these rules, it is described what edges are added to the flow graph and what data flow these represent. To help understand this flow of data for each rule some vertices and edges will be shown that conceptually represent the flow of data which is added to the flow graph. Details such as vertex types have been omitted from the example figures to emphasize on the concept of data flow. Underneath each figure a snippet of code is placed. The bold part of this snippet represent the parse tree node that is being visited in the appropriate rule.

R1 - Assignments

One way data can flow within a function by assigning values to variables like ‘l = 0’. Another way of saying this is that the value 0 flows into l. Two edges are added to the flow graph. The first edge represents the flow from the right-hand side of the assignment to the left hand side of the expression to model the flow from a value being assigned to a variable. In this example, the left hand side would consist of a vertex representing the expression l while the right hand side would consist of a vertex representing the expression 0. The second edge added to the flow graph represents the flow from an expression 0 to an expression representing the entire assignment l=0. This represents the right-associativity of the assignment operator, allowing assignments to be chained like this: a=b=1.

0 l

l = 0

l = 0 R2 - Disjunctions

Expressions can also be composed of different expressions using an operator. An example is the disjunctive operator ||. This expression is truthy if either one of the values it is composed of is also truthy. For example: the values l and r can flow into the a disjunctive expression ’l || r’. In JavaScript a disjunction does not necessarily return the value ’true’ or ’false’, it returns the first operand if it is truthy and otherwise it returns the second operand. For example: in JavaScript the value 0 is falsy whereas the value 2 is truthy. Thus, the value of the expression ‘0 || 2’ is 2. A disjunctive expression such as ‘l || r’ would generated two edges in a flow graph. The first one goes from a vertex representing the expression l to vertex representing the entire disjunctive expression l || r. The second one goes from a vertex representing the expression r to the same disjunctive expression vertex. Put another way: an edge to the disjunctive expression is added from both possible value origins.

l

l || r

r

l || r

R3 - Ternary expressions

In JavaScript and other languages a ternary expression of the form t ? l : r evaluates the value of t and takes the value of l if t is truthy, otherwise it takes the value of r. Like in a disjunction this adds two edges from vertices representing l and r to the vertex representing the entire expression t ? l : r. The value of t does not flow anywhere as the expression will not take the value of t itself.

l

t ? l : r

r

(24)

R4 - Conjunctions

Expressions can also be composed using conjunctions. This expression is truthy if both of the expressions it is composed of are also truthy. In JavaScript the conjunctive expression ‘l && r’ can only take the value of l if it is falsy. Since the purpose of this flow graph is to track function flow, if l is not a function it is not interesting for this analysis. A function is always truthy in JavaScript so if the value of l is falsy it certainly not a function. For this reason, only an edge from a vertex representing the expression r to a vertex representing the entire conjunctive expression is added.

r

l && r

l && r

R5 - Object definitions

In JavaScript you can create an object by defining key-value pairs inside of curly brackets like this: ‘p = {name : "Jan", age : 30 }‘. For each key-value pair an edge is added from a vertex representing the value to a property vertex with the name of the key.

Jan

name

30 age

p = { name : "Jan", age : 30 }

R6 & R7 - Functions

In JavaScript there are two different types of functions: function dec-larations and function expressions. A function declaration looks like this: ‘function f(){}‘. It is not assigned to anything, it is declared and can from now on be referenced by name. This can be seen as a flow from the function to the variable name exposing it. To model this in the flow graph an edge is added from a vertex representing the function to a vertex representing a variable of name f .

A function expression looks similar but can be assigned to a variable or placed inside an object. It does not need to have a name of itself. For example in the snippet ‘var f = function(){}‘ the function is anonymous, it is just assigned to a variable named f. It is possible however to also give the function a name, like this: ‘var f = function g(){}‘. In the first case, an edge is added from a vertex representing the function to a vertex representing an expression consisting of that same function. Since it is also assigned to the variable f rule R1 also applies. In the second case both of these edges are added as well as an edge from a vertex representing the function to a variable, like in the case of a function declaration.

function f(){}

f

function f()

There are a few problems with the specification. First of all, in the original paper the lookup function λ is defined as a function that looks up local variables and returns the position of a local variable or parameter declaration. It is not explained how to deal with function declarations, but since these follow the same scoping rules in JavaScript it would be sensible to search for function declarations in the symbol tables as well. If this is not done, a call to a function declaration would always be processed as a Prop(x) which wouldn’t create an edge in the call graph as functions yield edges from Fun(π) to Var(π) . For example: the resulting flow graph of the following code snippet:

(25)

1 function f() { } 2 f();

would produce the following flow graph1:

Func(mismatched.js@1:0-17)

Var(f, mismatched.js@1:0-17)

Prop(f)

Callee(mismatched.js@2:17-20)

As can be seen in the picture, there is no path between the function and the callee. This means that in the transitive closure, an edge between these two vertices will not be created and thus an edge from the call site to the function will not show up in the call graph.

Another reason to retrieve function declarations using the lookup function is that if this is not done it would also create a difference between function declarations and expressions, as function expressions are placed on the scope in the form var func = function(){}. In the Node.js implementation provided by the authors a function declaration is also retrieved by the lookup function.

A similar problem with the specification becomes apparent when functions are called which are defined on the global scope. This means that by failing to look up the function in local scope, an edge Prop(x) to Callee(π) is created. In the table on interprocedural flow however, it is stated that a named function expression or function declaration creates an edge Fun(π) to Var(π). Like in the previous example, this will not create an edge in the call graph as there is no path from the function to the callee. In order to mitigate this problem, the specification is adapted to create edges from Fun(π) to either Var(π) or Prop(x) , depending on what scope the function is in. This same solution was also followed in the Node.js implementation provided to me.

6.4 Interprocedural flow

The second step in the creation of the flow graph is the addition of interprocedural flow. Another visit of the parse tree is performed and vertices and edges representing the interprocedural flow are added to the flow graph. In the following table from the original paper [8] the rules for interprocedural flow are shown:

rule node at π edges added when visiting π

R8 f(¯e)or new f(¯e) V(f) -> Callee(π) , V(ei) -> Arg(π, i) , Res(π) -> Exp(π)

R9 r.p(¯e) as (R8), plus V(r) -> Arg(π, 0) R10 return e V(e) -> Ret(φ(π))

Table 6.4: Interprocedural flow

Similar for the intraprocedural flow described in table6.3, the three rules mentioned in table6.4

add interprocedural flow graph edges to the graph. These rules can be explained as follows: 1_{Res -> Exp vertices left out for clarity}

(26)

R8

For calls in the form of ‘ f(¯e)’ or ‘new f(¯e)’ three edges are added. The first one mentioned is V( f ) to Callee(π), to model that the function f is called at π. Furthermore, for all arguments of the call, edges are added from V(ei) to Arg(π, i) to model the flow from the arguments of the

call to the parameters of the function. Lastly, an edge from Res(π) to Exp(π) is added, to model the flow of the result of a function call at π to an expression π. In the example flow graph on the right both the function f and the argument x are properties.

Prop(f) Callee(π) Prop(x) Arg(π, 1) Res(π) Exp(π) f(x) R9

For function calls of properties the same edges are added, as well as an additional edge from V(r) to Arg(π, 0) to expose the ‘this‘ scope of the object the function is in. If a function is placed inside an object, the function can refer to properties of this object by the this keyword. In

the example graph to the right r is a property. Arg(π, 0) Prop(r)

r.f(x)

R10

Lastly, for a return statement ‘return e‘ an edge from V(e) to Ret(ϕ(π)) is added. The ϕ(π)means the enclosing function of

the parse tree node at π. This is done to model the flow of the outcome of the expression e to the return value of the enclosing function. In the example graph to the right x is a property. The enclosing function is looked up by searching for the this value in the symbol table, which is set for each function.

Ret(φ(π)) Prop(x)

function f() { return x; }

One minor improvement over the specification I have implemented is the creation of edges if new f is called rather than new f() as this is syntactic sugar for an instantiation of f without parameters. In the paper under rule R8 the patterns " f(e)" and "new f(e)" are shown as calls. I have adapted this to "new e" so it matches all instantiation patterns like "new f ", "new p. f ", "new p. f()", etc.

(27)

6.5 Example flow graph

In this section an example of a flow graph is given. Consider the following code snippet:

1 function f(multiplier) { 2 return function(number) { 3 return number * multiplier; 4 }

5 }

6 f(2)(24);

Figure 6.5: Example code

The function f is called with one argument called multiplier and returns a new function to which another argument number can be passed that is multiplied by the first number.

Func(example.js@1:0-87)

Prop(f)

Callee(example.js@6:87-91)

Figure 6.6: Flow graph for first call of example code in figure6.5

In figure 6.6a path from Func(example.js@1:0-87) to Callee(example.js@6:87-91) via Prop(f) is shown which represents the flowgraph edges added for the first call of the example code snippet shown in figure6.5. No complex interprocedural reasoning is required for this call: the function f can directly be found on the scope and is not the result of another function call. The connection of the vertices Arg to Parm and Ret to Res is not necessary to create a path from Func to Callee and it does not cross the Unknown node. This means that both analyses will create an edge in the call graph from Func(example.js@1:0-87) to Callee(example.js@6:87-91).

(28)

Expr(example.js@2:33-84) Ret(example.js@1:0-87) Callee(example.js@6:87-95) Expr(example.js@6:87-91) Res(example.js@6:87-91) Unknown Func(example.js@2:33-84)

Figure 6.7: Pessimistic flow graph for second call of example code in figure6.5

Expr(example.js@2:33-84) Ret(example.js@1:0-87) Callee(example.js@6:87-95) Expr(example.js@6:87-91) Res(example.js@6:87-91) Func(example.js@2:33-84)

Figure 6.8: Optimistic flow graph for second call of example code in figure6.5

The result of f is another function, which is then immediately called. This call does require reasoning about interprocedural flow. The flow graph produced in the pessimistic analysis is shown in figure6.7and the flow graph produced in the optimistic analysis is shown in figure6.8.

Because the function is not a one-shot closure, the edge from Return to Result flows via the Unknownvertex in the pessimistic flow graph. Before the pessimistic call graph is extracted, the optimistic transitive closure is created. This filters out paths through Unknown and thus the second call will not be present in the pessimistic call graph.

The optimistic call graph does not differentiate between interprocedural flow for one-shot clo-sures or normal calls. Instead it directly connects the Return(example.js@1:0-87) vertex to the Result(example.js@6:87-91) vertex. Because there is a complete path from the function to the callee without going through Unknown here, the optimistic transitive closure will create an edge from Func(example.js@2:33-84) to Callee(example.js@6:87-95). Thus, the second call will be present in the optimistic call graph.

6.6 Native flow

In the original paper the authors mention that native functions are also added to the flow graph. To model this a call can have a Builtin vertex as its destination. They mention that for all standard library function such as Array.prototype.sort an edge from Builtin(Array_sort) to Prop(sort) is added. This list of standard library functions needs to be assembled somehow and can heavily affect the outcome of the call graph algorithm. For example, if a script contains 10 calls to a toString function and the list of native function is extended to contain 5 more toString functions for various types of native objects, there will be an increase of 50 call graph edges. Because this can influence the precision of the produced call graph, it is important to keep this list of native functions the same as the one used in the original research to keep it comparable. Unfortunately this list is not available, so the next best thing is the list of native functions used in the Node.js implementation. When comparing the static and dynamic call graphs with each other, it turned out that some native functions are missing from the list. These have been added when necessary.

(29)

6.7 Pessimistic call graph edges

In the pessimistic algorithm a conservative triple is created consisting of pairs of one-shot calls and the matching one-shot closures, the escaping functions and the unresolved call sites. First the one-shot calls and their matching one-shot closures are gathered, these pairs are referred to as C in table6.9. All the remaining functions that are not part of a one-shot closure are marked as escaping function and all the remaining call sites that are not part of a one-shot call are marked as unresolved call sites. Table6.9is a subset of the figure depicting Algorithm 1 in the original paper [8], where the subsequent steps such as calculating the optimistic transitive closure has been left out as this will be discussed later:

CONSERVATIVE TRIPLE CREATION

Outputone-shot calls to one-shot closures C, escaping functions E, unresolved call sites U 1 : C := {(π, π0)|tπ is a one-shot call to function fπ0}

2 : E := {π0|¬∃π.(π, π0) ∈C}

3 : U := {π|¬∃π0.(π, π0) ∈C}

Table 6.9: Creation of conservative triple

This conservative triple is then used as input to add the interprocedural edges connecting Arg to Parmand Ret to Res to the flow graph. The pessimistic algorithm only tracks interprocedural flow in the case of a one-shot-closure like "(function( ¯x) { ... })( ¯e)" where an anonymous function is directly applied to some other arguments and only called once. In all other cases interprocedural flow is modeled using the special Unknown vertex. All the arguments of the one-shot calls are connected to the parameters of the one-shot closures and the return value of the function is mapped to the result of the function call. For all other functions and calls the Unknown edges are added. Table

6.10is depicted as Algorithm 2 in the original paper [8]:

INTERPROCEDURAL EDGES - PESSIMISTIC

Inputflow graph G, one-shot calls to one-shot closures C, escaping functions E, unresolved call sites U

1 : for all(π, π0) ∈C do

2 : add edges Arg(π, i) →Parm(π0, i)to G

3 : add edge Ret(π0) →Res(π)to G

4 : for all π∈U do

5 : add edges Arg(π, i) →Unknown to G

6 : add edge Unknown →Res(π)to G

7 : for all π0∈E do

8 : add edges Unknown →Parm(π0, i)to G

9 : add edges Arg(π, i) →Unknown to G

Table 6.10: Addition of interprocedural edges to the pessimistic flow graph

6.8 Optimistic call graph edges

There is little information present in the paper about how to add the interprocedural edges for the optimistic call graph algorithm. In the original paper it is stated that the optimistic approach starts with an empty triple rather than a triple consisting of the initial call graph, escaping functions and unresolved call sites. They explain that the flow graph is built using the same rules as for the

Efficient Approximate JavaScript Call Graph Construction