Test-suite for automated invariant-based testing of Ajax web-applications

(1)

Master Thesis

Test-suite for automated invariant-based testing of

Ajax web-applications

Sven Rohde, 10378782

svenrohde.se@gmail.com

March 14, 2015, 69 pages

This thesis is submitted to the University of Amsterdam in partial fulfilment of the requirements for the degree

of Master of Science in Software Engineering

Supervisor: Vadim Zaytsev,V.V.Zaytsev@uva.nl

Host organisation: Guidion ICT Nederland B.V.,http://guidion.nl

Host organisation contact: Mike Out (Manager IT department),mout@guidion.net

Universiteit van Amsterdam

Faculteit der Natuurwetenschappen, Wiskunde en Informatica Master Software Engineering

(2)

Abstract

Manual testing of complex AJAX based web applications can require a significant amount of time and effort. At Guidion the time limitations for manual testing leads to the identification of errors after live deployment of a new release. The web applications of Guidion were used by around 700 users for their daily work, errors in the web applications can impact the work of these users. Automated testing of dynamic AJAX based web applications was researched by the work of Mesbah et.al in [MvDR12] using the open-source crawler Crawljax. This thesis replicates the case studies described in [MvDR12] and evaluates their findings on performance, invariant creation, required manual effort and fault revealing capabilities using Crawljax and the proposed plug-in suite called ATUSA. Furthermore is included a survey research which investigates how web developer can be supported to provide invariants for ATUSA. The results of the research forms the basis for the proposed invariant collector Chrome plug-in and web application. The effectiveness of the ATUSA approach and the invariants delivered from web developers is investigated in the last case study with the ATUSA stack integrated in a Jenkins deployment pipeline at Guidion. Based on the observations the re-implemented and proposed ATUSA plug-in suite is capable to be used for automatic invariant based testing of AJAX web applications. The proposed invariant support integrated in the web developer development process, stimulates and separates the invariant creation process apart from ATUSA. Web developers are able to create invariants without Java, Crawljax or ATUSA knowledge. The integration in a sample Jenkins deployment job enables to trigger the test execution after actual test deployment. The case study reveals that the ATUSA stack integration at Guidion augments the testing phase of the web applications.

(5)

Preface

This thesis was written as part of the requirements for the Master Software Engineering of the Uni-versity of Amsterdam. The study was conducted at Guidion B.V., in Amsterdam.

This thesis would never be possible without the help and support of my supervisors Vadim Zaytsev and Mike Out. Furthermore I would like to thank the whole Guidion developer team for their support and feedback during the research.

Furthermore I would like to thank my family, the love and support I received from my mother Marion Rohde and my twin brother Philip Rohde. Another special thanks goes out to my father Reinhard Rohde, a lost resource of support and geniality R.I.P 2007.

”One of God’s own prototypes. A high-powered mutant of some kind never even considered for mass production. Too weird to live, and too rare to die.” Hunter S. Thompson

(6)

Chapter 1

Introduction

Interactive “Web-2.0” [Mes09] applications are a well-known part of the world-wide-web. Still more and more known services as well as desktop applications move to the web. This reflects the trend to move applications to the web to provide more flexibility and require less effort from the end user. Reasons for this move to the web are [MvDR12]:

• No installation effort.

• Up to date software, reduction of maintenance and support costs.

• Unbounded access to user data with any device. (Assume Internet Connectivity) • New ways of communication and collaboration.

• Better chances for an international user audience.

These Web-2.0 applications are wide spread and are an inherent part in the personal and business domain. Popular examples for that are Microsoft Office 3651_{, the Google Service Stack (with GMail,}

Google+)2_{, Atlassian Stack (Jira, Bitbucket, Confluence)}3 _{and Social Media platforms}4_{. The host}

organization realized this trend and is using different web applications to operate. Besides using existing web applications, develops and maintains the host organization own web applications for supporting their business processes.

To provide more flexibility, information sharing and collaboration possibilities, the host organization started to change existing traditional web applications to interactive, responsive and complex web applications. This includes new features and constant refactoring to move existing functionality into the same direction. One of the technologies that supported and enabled this enormous change in the domain of the web is AJAX [MvD07]. AJAX is an abbreviation and stands for “Asynchronous JavaScript and XML” [Gar05]. It enables browsers to provide more functionality than only loading, rendering and navigating through static pages.

It enables the browser to provide a more dynamic and interactive user experience through respon-siveness and the use of user interface (UI) components [MvDR12]. These components are also known as partial site updates through asynchronous client-server communication and client-side JavaScript execution. Despite the fact that AJAX positively impacts the whole user experience when brows-ing the web, or usbrows-ing web based applications, it comes with new challenges for the developers. The main challenge is testability of these modern web applications, which stands in the main focus of this thesis. Testability of the AJAX UI components is a challenging task because of the asynchronous, event-driven and stateful nature of AJAX, and of client-side JavaScript based manipulation of the Document Object Model (DOM) [MvDR12] [vDM10].

In the host organization testing is a growing part of the development process, and besides functional testing of server-side components, client-side testing and UI testing gain more interest. Through the

1_{Microsoft Office 365}_{http://office.microsoft.com/} 2_{Google website}_{https://google.com}

3_Atlassian_{https://atlassian.com/}

(7)

fact that the host organization moves more and more new and existing features to a new and modern user interface with an implied move of server-side functionality to client-side functionality. These web applications support the core business processes and are under a constant and rapid change in the last year from traditional web applications to AJAX based web applications. Testing gets more important for the host organization but a solution is not yet implemented to test the new features, especially on UI side.

The host organization needs an automated testing approach after every release, which is not possible at the moment, only new features are manually tested which is a hard and time consuming task. This problem is addressed in the research “Invariant based Automatic Testing of Modern Web Applications” [MvDR12], published by A. Mesbah, A. van Deursen and Danny Roest. It proposes a Java based crawler named Crawljax to support automated AJAX based application testing by crawling the web application’s DOM, and inferring an (finite) event-state-machine by clicking all specified DOM events. The event-state-machine or state flow graph serves as a basis for testing approaches. Additionally to the use of Crawljax, the paper proposes a new technique and tools to use Crawljax for Regression testing of AJAX web applications including invariant based testing. This tool composition is called ATUSA, an abbreviation for “Automatically Testing UI States of Ajax” [MvDR12]. ATUSA provides new core features, for example generic comparators (for detecting relevant UI elements) for generic error detection (mostly DOM related errors), a plug-in API, generic and application specific invariant checking. Additionally it provides plug-ins for visualization and test case generation, which enables automated testing as well as a control mechanism.

Thesis focus One goal of this thesis is to replicate the case studies with ATUSA as described in the initial research. For validation purposes the case studies were replicated in confirmatory form.

The findings are used for comparison with the findings of the initial research, for providing more empirical evidence to support the use of Crawljax with the ATUSA extensions. Furthermore, the possibility to create application specific state and state-machine invariants to use with ATUSA is investigated. Creating these invariants is a pure manual approach and requires a understanding of the concept of invariants and Java programming skills, which is also identified as an thread of acceptance in the initial research [MvDR12] as well as in the research of van Deursen et al. [vDM10]. Addressing the problem of the invariant creation provides this thesis a investigation about an easier way to provide application specific invariants for non-academics for ATUSA. The main focus here lies especially on the manual effort to provide those invariants. Additionally a proof of concept is included to deliver these invariants for the automatic web application testing approach. The last goal of this thesis is the integration of ATUSA into an existing development pipeline and a investigation of its fault revealing capabilities and its impact on the initial research questions [MvDR12].

Summarized goals of this thesis

• Replicating the case studies in confirmatory form.

• Conducting an interview research about providing invariants and fault information delivery. • Creating a proof of concept about invariant creation support and error insights.

• Integrating the tools in deployment pipeline.

• Conducting a research about its effectiveness at the host organization.

1.1 Host organization

This thesis was conducted at Guidion ICT Nederland B.V.5 Guidion is a Dutch provider of a na-tionwide network of technical specialists. These technicians are the field force of Guidion and deliver technical support and services in every region of the Netherlands. The main sectors in which Guidion operates are:6

5_{Guidion ICT Nederland B.V. will be further referred as Guidion throughout this document} 6_{Information stated in brackets are business use cases examples.}

(8)

• Energy support sector (Smart Energy Devices)

• Telecom support sector (Installations, Technical all-round service) • Call Center support sector (Appointments, Surveys)

• IT Sector (Computer trouble-shooting) • Retail support sector (Printer support)

The company delivers service and works for well-known Dutch telecom and energy providers, inter alia for companies such as: UPC7_{, Eneco}8_{, XS4ALL}9 _{and Telfort}10_{. Guidion uses mainly web based}

applications to operate and their development and maintenance is for 80% in-house development and the rest is commercial software as well as outsourced development. The field force and the office busi-ness processes rely on these web applications. Uptime and correct functionality of the web application is essential for Guidion and their technicians in the field. For example, the technicians have to log every step they do at the end customers side, including logistics, task and time logging. If one of the tasks is not executable within the web application, the technician needs support from the office, which implies a longer stay at the end customers side, as well as a possible impact on the following appoint-ments of the technicians. The in-house developed web applications are under constant development. The IT department uses an agile development process with an existing deployment pipeline, which enables the developers to deliver new functionality in short development cycles. The IT department consists of inter alia five developers including UI front-end developers who are the main information sources for this thesis.

1.2 Thesis document structure

This thesis document is structured as follows. After this chapter follows the problem explanation, which explores the existing scientific problem and the existing real world problem of the host orga-nization Guidion. The problem section forms the basis to understand the motivation, which is the following section. Right after the motivation section follows the chapter research methods, which introduces the research method used for this thesis. After that follows background section, to provide the required information about Crawljax and ATUSA. The next chapter includes the research with detailed information about the conducted case study replications, the invariant research, the devel-oped proof of concept and the final case study about the effectiveness of the approach. The results and answers to the research questions are presented in the following final chapter of this thesis document, the analysis and conclusion.

7_UPC_{https://upc.nl} 8_Eneco_{https://eneco.nl} 9_XS4ALL_{https://xs4all.nl} 10_Telfort_{https://telfort.nl}

(9)

Chapter 2

Problem statement and motivation

This thesis addresses the challenge to execute automated invariant-based testing of modern web-applications.

At the moment client-side testing is executed by a small team of User Interface (UI) designers / developers. Due to a small time budget for testing, the developers execute the tests in a fast manner and without sufficient test reports. This leads to the problem that several bugs are encountered during the use of the application when certain functionality is needed but does not work correctly. This leads to delayed or blocked business processes at the host organizations head-quarter or at the field force side. In this case the development team has to react immediately. These critical bugs can cause delays in the business operation and create extra costs. Those bugs are mostly fixed in a quick manner without further validation of correctness. Guidion has no insight in the current state of the web applications, and has no insight if bugs are really resolved within the next release. Also reproducing bugs is not always possible, through a lack of adequate information delivery to reproduce these bugs.

Guidion has no existing basis as test framework, which focuses on user interface related testing. These problems are addressed throughout several research publications from A. Mesbah and A. van Deursen using the Java based open-source crawler Crawljax1_.

2.1 Evaluation of the proposed ATUSA approach

Since AJAX was coined in 2005 by J. J. Garret [Gar05], testing of AJAX web applications gave rise to new challenges. These challenges in testing AJAX applications are further described in section4.2. Mesbah et al. analyzed the testing issues with AJAX web applications and proposed a Java based open-source crawler called Crawljax [MBvD08] [MvDL12]. Crawljax uses a web browser to simulate user behaviour through clicking predefined HTML candidate elements. The basic functionality of Crawljax combined with the customizations and extensions provided with ATUSA address these challenges.

AJAX is commonly used to create interactive UI elements within a few or even a single web page. The advantage is that AJAX’s asynchronous nature enables the browser to update small DOM elements without the need of reloading the whole web page. Precisely this advantage is used to build UI widgets within a web application, which uses the client-side AJAX functionality to retrieve data from a server and update the document object model (DOM). These client-side DOM manipulations are hard to test in complex web applications, because the state of the web application depends on the user input. Traditional web application testing approaches reaches their limits by testing AJAX in combination with client-side JavaScript based DOM manipulation. The field of interest is here, that all AJAX functions are working correctly and manipulating the DOM in a valid manner.

Additionally to testing AJAX, web applications are a fast changing branch in the domain of soft-ware development. The existing testing tools to test AJAX based web applications require manual programming effort. Testing a web application after a new UI design also requires updating the

(10)

ing test suite. The proposed approach with ATUSA is already two years old but still unique and valid [GMBCM13]. The open source crawler Crawljax evolved since 2012, and was used in later research with resulting improvements [NMvD14]. Mesbah et al. states that more empirical data is required to validate the ATUSA approach as a valid approach for a automated way to test AJAX based web applications.

The main focus of the confirmatory case study replication lies on the evaluation of the following factors:

• Obtaining of invariants, and expressing them in ATUSA. • Measuring fault revealing capabilities and effectiveness. • Measuring performance and scalability.

• Measuring the level of automation and required manual effort.

An additional problem mentioned by Mesbah et al. is the acceptance of the invariant based testing approach. Obtaining and creating those invariants for web developers without a background in testing can lead to this problem. The invariant creation problem is the focus of the second part of this research and is described in detail in section5.9.

2.2 Manual testing at Guidion

Testing approaches at Guidion are only implemented in the field of server-side unit testing, only for new and complex features. This covers a few parts of server-side functionality, including server-side implementations for AJAX related calls. These functions are only used within AJAX requests to manipulate the DOM. More than a thousand users use these web applications of Guidion across the Netherlands. The users depend daily on these web applications to do their daily work. Without them, business processes are blocked and the field force technicians are unable to retrieve work relevant information, which can lead in worst-case scenarios to blocked work at the end customers side. Those problems lead besides additional costs to delays in the technicians schedule, frustrated employees, frustrated end users and a bad reputation for Guidion. The company relies on responsive and correctly working web applications.

The list below mentions only a few problems occurring through the lack of testing.

• Blocking user support, appointment creation. • Blocking work logging for field-force technicians. • Blocking end-user logistics.

• Blocking depot logistics.

• Blocking recruitment or certification processes.

These problems have direct impact on a correct and efficient flow of the core business processes, and every delay has direct impact on unnecessary cost creation. Waiting times and even extra working hours is a recurring outcome of those problems. Guidion is interested in a process to validate the web applications shallow working test, including a proof of concept with the possibility for regression testing of the web application. Furthermore is it interesting for Guidion to get insights in the current state of deployed web applications. Guidion employs besides a group of software developers also different interns. These interns are working on several feature branches as well as creating whole new applications in the scope of their intern-ships. Through the lack of technical support and control have those interns a high level of self-responsibility to deliver maintainable applications. So this implies that the complex web applications are partly unknown for the current UI development team. Most of the interns within the IT department of Guidion are front-end developers. They actively help delivering new features or modernize old ones. The challenge here is to support the UI web developers, which are partly non-academics or academics with a different background, in creating invariants for ATUSA.

(11)

At the moment Guidion does not have basic testing approach implemented in their development process. A selected group of power users test the new functionalities in staging environment. After approval of the new features they are released. One huge problem is the lack of time reserved for testing within Guidion. The selected group concentrates mostly on the new features and accepts those. An additional validation of business crucial application functionalities are not included or get skipped due to time limitations.

In the past, server-side code could be tested within a existing unit testing suite. The development team uses the unit testing framework PHPUnit [Ber05]. The technical change and the shift of server-side to client-server-side functionality is not the only problem identified within the development department. Several factors have an influence on the delivered code, functionality and its maintainability. Due to a small time budget for testing, the testers execute the tests in a fast manner and without satisfying test reports. This leads to the problem that several bugs are encountered during the work of the user and some can cause blocked business processes. In this case the development team has to react immediately. These critical bugs can cause delays in processing business processes and also extra costs in extra effort which is needed to create a workaround or process data manually. This causes problems within the development team such as:

• UI related quick fixes.

• Current development sprint freeze and stop. • Need for regression and smoke testing. • DOM based insights.

To avoid this Guidion currently is establishing its own testing strategy. This is supported through the facilitation of a deployment pipeline in form of a Continuous Integration (CI) system. The system of choice is a Jenkins CI2 _{Server [}_Sma11_{]. Jenkins is a widely used open-source system with a huge}

palette of extensions for nearly every testing framework as well as building systems. It is in Guidion’s interest that the approach with ATUSA can be integrated in the deployment pipeline.

Guidion web developers At Guidion a team of UI web developers create the AJAX client-side functionality as well as UI related functionalities. The team consists of two full time web developers and one intern. These UI web developers have an academic background, but with a different focus than software engineers. This means that the use of invariants as well as a basic understanding of tool supported testing is missing. Additionally to the lack of automated testing, these UI web developers have every half year one or even more new interns. Those interns have a high level of responsibility and the features they deliver are integrated into the main web applications after a quick validation of the required functionality. Due to time limitations, the UI web developers are forced to accept those features without an overview about all their UI elements, and DOM related manipulations. These web developers form the target group of this research regarding the invariants research in section5.9.

2.3 Thesis Motivation

This section describes the motivation behind this thesis and is divided into three parts, every part covers one aspect of the complete motivation.

Replication in confirmatory form to provide empirical evidence The first aspect is the replication of the initial research [MvDR12]. The motivation itself is to validate if the ATUSA approach as described is still applicable and delivers the comparable results.

Basis for invariant based web application testing Guidion currently has no existing testing basis for automated testing of the client-side behaviour of their web applications. Within the scope of the replication of the case studies, the ATUSA approach is theoretically suited to build a basis to

(12)

provide automated testing. The motivation here lies on integrating and supporting the test mentality at Guidion with providing a basis for further testing possibilities.

Basis for reporting on encountered errors The ATUSA approach include different plug-ins for example an overview about the crawled web application and a error report plug-in to inform about failed invariants. Especially at Guidion where the applications are maintained by different groups of developers and interns an overview and reporting mechanism deliver insight about the web applications state. The reporting mechanism also establishes a basic reporting manner for Guidion.

(13)

Chapter 3

Research method

This section describes three different research methods used within this thesis. Every research method introduces their target research questions. In order to replicate and validate the findings of [MvDR12] we have conducted three confirmatory case study replications. In addition to the replication a survey research was conducted to gather information of a group of UI web developers, about their current testing approaches, invariants support and expectations of the integration of ATUSA. Its result formed the basis of the proof of concept to support web developers to provide invariants for ATUSA. Finally the research includes a case study to evaluate the effectiveness of proof of concept integration.

3.1 Replication of case studies in confirmatory form

The replication is divided into three confirmatory case studies. The first case study focuses on the use of invariants and their expression in ATUSA. The second case study evaluates the fault revealing capabilities and scalability of ATUSA. The third case study is applied to a Guidion web application, to evaluate the ATUSA approach in an active development process. Every single case study includes a measurement about the manual effort required to configure and setup ATUSA. The replication covers the following research questions of this thesis.

• RQ1 - What kind of invariants can be obtained for AJAX applications and how can they be expressed in ATUSA?

• RQ2 - What is the fault revealing capability and effectiveness of the approach? • RQ3 - What is the performance of the proposed approach, and how well does it scale?

• RQ4 - What is the automation level and how much manual effort is involved in the testing process?

The conducted case studies were set up following the guidelines of Yin [Yin02]. because Mesbah et al. also followed those guidelines.

Confirmatory case studies 1.A - 1.D : RQ1 In order to validate the results on RQ1, the same setup, open-source web applications and techniques are used. According to [MvDR12] four different web applications were investigated, to create application specific invariants. The experiments per web application use the same invariants, and the same configuration for ATUSA as specified in [MvDR12] as well as described in earlier work [RMvD10]. Every case study has its own scenario, which is also included to provide a complete confirmatory case study replication. Besides identification and the possible expression of invariants in ATUSA, a measurement of the required manual effort per experiment is included, to provide complete and comparable findings.

Confirmatory case study 2 : RQ2 - RQ4 The second part of the confirmatory case study focused on the research questions RQ2 - RQ4. It follows the description of the initial research to replicate the case study with the open-source web application Tudu5.7.

(14)

Confirmatory case study 3 : RQ2 and RQ4 The third part focused on all four research questions RQ1 - RQ4. In this experiment one real world web application is used to find real bugs in a active development cycle. This experiment uses a web application of Guidion as test system subject. Hypothesis: RQ1 The validation and creation of the proposed invariants from the original research [MvDR12] identify the same errors by testing the web applications with ATUSA.

Hypothesis: RQ2 The implementation of the proposed test scenarios with their ATUSA configu-ration, identify the same or different errors as in the initial research. The term effectiveness is in the case of this research an objective and commonly used indicator of the quality of a test suite [MvDR12]. Hypothesis: RQ3 The improved version of Crawljax and ATUSA provides a better performance with respect to the case study replications as in the original research [MvDR12].

Hypothesis: RQ4 The detailed study of Crawljax and ATUSA requires a manual effort for the implementation of the proposed invariants and plug-ins which is comparable with the reported manual effort from the initial research [MvDR12].

Reporting on Replication The replication follows the proposal of Carver [Car10] on reporting on replications. This ensures that the confirmatory case studies report in the manner, level of detail and in a comparable form.

Validation of results The validation of the results from the confirmatory case studies, are com-pared with the results from the initial research [MvDR12]. The findings from the first part targeting RQ1, are compared with the findings from the initial research. This part focused on the invariant configuration of ATUSA. Additionally the findings from the initial research [MvDR12] on page 47 in section 8.2.6 are compared. For validating the findings of the experiment targeting RQ2 - RQ4 the findings are compared with the findings from the initial research [MvDR12] stated on page 48 in section 8.3.3, and for the second part the case study results from the real bugs case study are compared on their similarities in the findings.

3.2 Web developer survey research

Information gathering about the developed web application is the second part of this research. To collect evidence about the current testing habits within Guidion as well as development tools, a survey study in form of a questionnaire is conducted. The questionnaire is used to collect different information about the web developers development tools, their testing tools, knowledge about invariants, about mapping events as well as used JavaScript Frameworks for the current web application development and also their expectations of ATUSA’s integration. The main focus lies here to determine where web developers can be supported to create application specific invariants. The web developer research survey and the invariants research cover the research question:

• RQ5 - How can web developers be supported to create application specific invariants for ATUSA?

The results of the survey research are used to propose a application to support invariant creation for ATUSA.

Hypothesis: RQ5 The separation of the invariant expression for ATUSA from the Java based configuration via a mechanism integrated in the web developers test process, enables the web developer to create application specific invariants for ATUSA.

(15)

3.3 Case Study: Effectiveness of the integration of ATUSA

The final conducted experiment of this thesis is a case study about the effectiveness of the integration of ATUSA at Guidion. For this cases study, ATUSA will be integrated in a existing deployment cycle, and tests one web application of Guidion. In addition to show the crawl results and a possible historical evolution of the web application, a useful error report is delivered and information to support the invariant creation. For this case study, the integration fault revealing capabilities of ATUSA are evaluated.

The web developer research survey and the invariants research cover the research question:

• RQ6 - How effective is the integration of ATUSA at Guidion?

Hypothesis: RQ6 The usage of ATUSA in a Jenkins job on a Guidion web application delivers otherwise unavailable insights in the web application and can be used as additional testing phase, which tests the web application before user acceptance testing phase.

3.4 Summary

This section provides a complete list of all research questions, which are covered in this thesis.

• RQ1 - What kind of invariants can be obtained for AJAX applications and how can they be expressed in ATUSA?

• RQ2 - What is the fault revealing capability and effectiveness of the approach? • RQ3 - What is the performance of the proposed approach, and how well does it scale?

• RQ4 - What is the automation level and how much manual effort is involved in the testing process? • RQ5 - How can web developers be supported to create application specific invariants for ATUSA? • RQ6 - How effective is the integration of ATUSA at Guidion?

(16)

Chapter 4

Background and Context

This thesis project is based on the research by Mesbah et al. [MvDR12]. This research proposes a technique and a tool implementation for automatically testing JavaScript-based AJAX web applica-tions. The proposed testing approach focusses on testing dynamic User Interface (UI) states within a JavaScript-based AJAX web application. The goal of this section is to offer a basic background which is required to understand the replication and the contributions of this thesis.

4.1 Why is AJAX so interesting?

AJAX is interesting through the fact that it introduces so many new dynamic client-side possibilities in domain where dynamism was limited on the server-side implementation of web applications. It introduces the creation of the so called Web-2.0 [MvD07], which is basically defined as interactive web applications with user generated content. In addition AJAX also introduced Web User Interface (UI) widgets. This can be seen as widgets from the GUI application development. For example, a table component can be referenced and updated without updating or forcing a reload of the whole application. Also the browser takes care that the whole web application stays responsive until the table gets updated within the DOM1 _{and the result is rendered. AJAX enables new user interface}

features and improves the user experience, but it also introduces new challenges. For example, use of the browsers back button, to navigate to the last state does result in another state than expected. Getting indexed by a search engine crawler. In addition to those examples, the developer needs to take care of the DOM manipulation, which includes also a DOM clean-up after disabling visibility. A common error here is that the DOM grows unexpected as result of a missing DOM clean-ups.

4.2 Challenges in testing AJAX based web application

AJAX introduces also new challenges for testing. Especially automated testing is challenging through the following aspects. The research of A. Mesbah and A. van Deursen identifies the Reach, Trigger and Propagate challenges which complicate testing of JavaScript-based AJAX web application [MvD09]. A. Mesbah and A. v. Deursen identified in [vDM10] additional issues in testing of AJAX web appli-cations. Further research in this field conducted by A. Mesbah and A. van Deursen with help of S. Lenselink and D. Roest identified different challenges [RMvD10] [MvDL12] [Roe10] [MvDR12]. This section introduces those challenges, issues and problems identified, and lists a summary of them with a detailed description per challenge.

Reaching states In traditional web applications different states of the web application are reachable through a unique URL, including possible HTTP GET parameters. For reaching AJAX states a unique URL is not enough. Those dynamic AJAX states are hidden behind JavaScript events included in the DOM. This means to reach AJAX states those events need to be identified during the crawling

(17)

process. Testing these states, the crawler needs besides identifying those states a mechanism to execute the AJAX events, which can lead to another state. In addition to the reaching problem, another problem connected to the reaching state problem is the problem of bookmarking and sharing the web application page. Through the fact that the web application has only a few different URLs, creating and sharing bookmarks is a additional problem here. Another problem is back-tracking, which is further described in paragraph4.2.

Triggering events The identified events of a web application state need to be triggered/executed within a test environment to validate, if the event results in a new state. Not only triggering those events is important in this phase. AJAX states are not only bounded to an event, also user input needs to be considered here. User input can result in a different state, even if the same event was triggered. Before triggering an identified event, data input entry points need to be identified and data needs to be inserted before triggering.

Propagate result After executing the possible data inputs and events the result needs to be prop-agated. This results by AJAX in a client-side DOM manipulation. The manipulated DOM needs to investigated and compared with given states to determine if a new state was reached. Additionally to the new state identification the JavaScript errors need to be identified, to see if the new state does not only results of missing or incorrect implemented JavaScript event. The propagated result is the basis for invariant or condition based validation.

Asynchronous nature The browser uses a AJAX engine, which executes asynchronous Xml-HTTPRequest to request new information from the server. This gets executed in a asynchronous way, which means that the user can use the browser and can click on other elements on the same page, and the browser will work further without freezing an waiting for the AJAX response. Test-ing these asynchronous calls the testTest-ing approach needs to wait on the response of the server and the AJAX resulting DOM manipulation, to test the propagated state against the old state for state identification described in paragraph 4.2. In addition to waiting for the response, the wait time is important to determine if another candidate element event can be triggered without receiving other asynchronous AJAX responses. This makes event identification and result comparison of the DOM more complex.

Stateful behaviour Another challenge in deriving AJAX states is that the client-side gets more stateful. The switch from server-side to client-side functionality includes also a switch to investigate and store the state-full behaviour from the DOM states. This is especially interesting for deriving test cases from the crawled web application. The stateful behaviour needs to be reproduced within a stored test case to reproduce the same state results in the test case execution phase. The newly crawled states need to base on the same user behaviour, which is the navigation path as well as the used input values.

Backtracking The identified fact that the client-side gets more stateful, consistent back-tracking is a known problem in current AJAX web applications.[MvDR12] To test back-tracking collected states need to be bidirectional navigable and the states need to be stored and compared. This means that the target state needs to have an edge back to the source state. As briefly described before in section 4.1 the back-tracking problem is a known downside of AJAX web applications, and the test framework needs to support comparing the states if they support correct back-tracking for the crawled web application.

Correct DOM-tree management A additional problem through the use of AJAX in web appli-cations, is the correct DOM-tree management. This means that AJAX is used to make certain DOM elements visible, or add and remove elements. To avoid a growing DOM tree and a slower browsing experience, the DOM elements added through AJAX need to be removed correctly.

(18)

The oracle problem The oracle problem was identified in the research “Regression Testing Ajax Applications: Coping with Dynamism [RMvD10]”. The oracle problem states that fault positives occur in state comparison and its resulting creation. This means that states identified as two sep-arate states are the same. A example here is a date component on a web page, which shows the date and current time with a running clock. Comparing those states without stripping the textual information would result in additional states. To avoid additional and unnecessary state creation, oracle comparators are proposed by the research [RMvD10] to strip all unnecessary state information before actual state comparison. The oracle comparators proposed in [RMvD10] avoids also the state explosion problem. This problem is further described in “Avoiding state explosion problem of gen-erated AJAX web application state-machine using BDD [AS13]” which also proposes an additional approach to avoid state explosion in generating finite state-machines through crawling AJAX based web applications. The problem of state explosion does not receive further attention in this thesis, but could be interesting for further research.

State correctness: DOM validation The dynamic states are hard to test for validity. This means the check if the state DOM is a valid Document Object Model. The identified and collected states need to be validated on correctness. The Document Object Model relies on its corresponding HTML standard and can be validated against its specification. To check for state and DOM correctness the state DOM needs to be validated as valid Document and needs to be persisted for further validation. State correctness: Invariants To test state correctness during the crawling phase the initial research [MvDR12] proposes application specific invariants as a additional mechanism to test DOM validity. Those invariants can be used to check if a DOM of a state is also valid to its application specific DOM specifications. Invariants and application specific invariants are further described in detail in section4.4.

4.3 FSM based web application testing

Finite state-machine (FSM) based testing The state based testing approach was initially pub-lished by Marchetto et al. in “A case study-based comparison of web testing techniques applied to AJAX web applications [MRT08]”. This approach uses traces of the web application to infer a fi-nite state-machine (FSM). The FSM is redefined by the tester and further used as basis to generate test cases with help of sequences of “semantically interacting events [MvDR12]”. This approach also involves additional manual effort for redefining the FSM by the tester.

FSM - Invariant based web application testing The testing approach of the initial research is the first automated attempt to test AJAX based web applications. It uses a crawler to infer a FSM based on the approach explained above in paragraph 4.3. In the FSM are the different states the nodes and the events fired the edges, and it is the basis for further test generation.

Collecting AJAX states through crawling Crawling the web application using Crawljax and deriving a finite state-machine (FSM, state flow graph) from a web application was initially proposed in [MBvD08]. Based on the research [MBvD08] the authors released their crawler tool implementation “Crawljax”. Crawljax’s implementation and its crawling algorithm was improved through further research conducted by A. Mesbah and A. v. Deursen [MvD09], [RMvD10], [MvDL12], the initial research [MvDR12] and one research after publication of the initial research [NMvD14]. Crawljax uses the WebDriver API, which enables the use and control of the WebDriver browser interface. The WebDriver browser interface enables Crawljax to crawl the application with a full browser support. The browser support enables Crawljax to execute JavaScript related events.

Conclusion - Web application testing Web application testing has evolved to support mod-ern web applications. The host organization realizes the need of a automated testing approach to

(19)

reduce manual testing effort and to introduce a static test basis. The initial research and the Crawl-jax enhancement with ATUSA [MvDR12], provides a promising approach in testing modern web applications [AS12].

4.4 Invariants

Using invariants for validating programming functionality is well-known in the domain of software testing. In the domain of web application testing, invariant based testing can be understood as condition on validating the DOM of certain or all states. Traditional testing approaches also uses invariants for example checking on a functional level if pre or/and post conditions of a function are still valid. A simple example is in a unit testing framework such as PHPUnit the Assert statement to check with invariants. Assert checks in this case if the function still produces correct results or handles correct errors, and no side effects of the function can occur. Invariants for ATUSA are also named conditions, these conditions must be satisfied after a DOM manipulation otherwise the invariant is violated. The initial research [MvDR12] divides invariants for ATUSA into four types. These four types are generic state invariants, application specific state invariants, generic state-machine invariants and application specific state-machine invariants. Generic state invariants can be applied on every state in the state flow graph. These generic state invariants are validated with every new state and need to be valid for each inferred state of the crawled web application. Application specific invariants need to be specified by the developer or the tester. Those invariants are bounded to one specific web application, and need to be manually adjusted when the web application and its DOM structure changes. Generic state-machine invariants are invariants on the inferred state-machine, where every state in the inferred state flow graph includes a certain transition. The back-tracking invariant is a generic state-machine invariant. The last type of invariants are the application specific state-machine invariants. Those invariants need also be implemented by the tester as a plug-in which checks the inferred state flow graph includes certain transitions, e.g every state with a logout button has a transition to the log-out state. Every type is explained in detail in the sections below. In addition to the types of invariant proposed in [MvDR12] and implemented for ATUSA, this section further describes the current research status of collecting or automatically inferring invariants. State-machine and state invariants Mesbah et al. describes two main sorts of invariants which can be used for testing. These sorts are state invariants and state-machine invariants. Both sorts consist of generic and application specific invariants. The difference is that state invariants are used to test every new state and the state-machine invariants, are used to validate the inferred state-machine against certain constraints. Both are described in the sections below. At first the state invariants for DOM testing and then the state-machine invariants for validating against the inferred state flow graph (or FSM).

Generic state invariants The generic state invariants are invariants, which should be valid for every inferred state from the web application. The initial research states here two invariants, which are applied on every state in ATUSA. Dead Clickables and Back Button Functionality (back-tracking). In addition to those generic invariants used by ATUSA, generic invariants can also be used e.g. for error message detection. Every new state can be checked if it includes server side generated error messages. Furthermore for W3C HTML validation, detecting JavaScript errors and Image reference errors. More information about generic invariants and their implementation as well as use in ATUSA can be found in section5.9.

Application specific state invariants Application specific invariants are invariants specific to test the states of a special web application. Those invariants need to be specified manually by the developer or tester. The initial research mainly uses application specific invariants to add constraints on the DOM of the case study web applications [MvDR12]. Those invariants need to be implemented in Java and requires a general knowledge of Java and a Java friendly development environment. Mesbah et al. proposes different ways to express application specific invariants. More information

(20)

about the expression, implementation and the use of application specific invariants in ATUSA can be found in section4.5and section4.6.

Invariants, conditions and pre-conditions The invariants are based on condition, which need to be satisfied for every state or just a certain set of states. The set of states on which the invariants apply can be customized within Crawljax with the ability to specify pre-conditions. These pre-conditions need to be satisfied before a application specific state invariant is used as a DOM constraint. Generic state-machine invariants Generic state-machine invariants can be used to test against the inferred state flow graph (FSM) and its transitions. This offers the ability to test the state flow graph in its whole, and for certain transition constraints which need to be satisfied for the whole inferred state flow graph. Generic state-machine invariants are for example invariants to check the back-tracking functionality, which means that every state in the state flow graph should have a transition back to its source states. This generic state machine invariant can be checked against the whole state flow graph. Another example proposed by the initial research is to check the state flow graph for dead clickables, which means that for every state identified events are checked if they produce a error on execution.

Application specific state-machine invariants Application specific state-machine invariants can be created and applied by the developer or tester to check the state-machine of a web application on application specific constraints. Pressing a logout button in the web application must end in the login state, and this constraints need to be satisfied for every state where a logout button is visible. Those constraints are expressed in a “Source” → “Action” → “Target” manner [MvDR12], and validates transitions in the state flow graph.

Related work in deriving state invariants Inferring invariants automatically is still an open research topic. In the domain of invariant derivation, K. Pattabiraman and B. Zorn proposed a tool called DoDOM [PZ10] which is able to derive invariants automatically from a test system. DoDOM records user interactions and events and repeats these executions in different sequences and captures the DOM. DoDOM repeats these events six times to gain a good invariant coverage. DoDOM consist of three components a proxy, a JavaScript executor including logging and a invariant extractor. The proxy captures the different DOMs and events and the JavaScript logger executes them in the browser. The Invariant extractor infers the different DOM invariants. DoDOM comes with several limitations and trade off’s, one limitation to mention is that DoDOM is in the proposed tool only applicable for single page web applications, and one trade off is the execution of six times. DoDOM provides a promising approach to infer invariants, but is not directly applicable for this thesis, because DoDOM is mostly implemented in JavaScript and its source is as a open-source project public available. The implementation in ATUSA was out of scope for this thesis. Furthermore the approach requires in average six crawling executions to reach 100% coverage of state invariants. To execute ATUSA six times can require a long time when the test subject system consist of a large set of different states.

Another approach to automatically infer invariants was proposed by Groeneveld, Mesbah and van Deursen in [GMvD10]. They proposed a plug-in for Crawljax which is able to infer invariants within several crawling runs. It uses the proposed tool of [EPG+₀₇_{] called Daikon to infer DOM and}

JavaScript invariants. Groeneveld et al. released this plug-in composition for Crawljax version 1.9. Unfortunately the plug-in was not further maintained, and bringing it in a useful state for this thesis was out of scope.

The scope of the research is to provide a mechanism to express application specific invariants without any Java knowledge, so that web developers and designer can also provide invariants to use during the use of Crawljax within Guidion.

4.5 Crawljax

The automatic testing approach proposed in [MvDR12] is based on the Java open-source crawler Crawljax [Cra14]. Crawljax is a crawler application targeting JavaScript based AJAX web

(21)

appli-cations, and automatically infers a state flow graph of the crawled web application. Crawljax was initially proposed by A. Mesbah, E. Bozdag and A. v. Deursen in the research about crawling AJAX web applications and inferring a finite-state-machine [MBvD08]. The approach used to infer a finite state-machine was initially proposed by [MBvD08]. In addition to automatically inferring a state flow graph during the crawling process, Crawljax fully supports client-side JavaScript execution. Its approach was further applied and improved in additional work by A. Mesbah et al. in [BMvD09], [Roe10], [RMvD10], [MvDL12], [MvDR12] and [NMvD14]. Those publications improved Crawljax, and also a small but active user community, who uses, tests, reports and contributes improved Crawl-jax. The version used for this thesis is Crawljax version 3.5.1 [Cra14], which was at begin of this thesis the latest stable release.

State flow graph implementation notes The current version of Crawljax [Cra14] uses a open-source library called JGraphT for the state flow graph implementation. The state flow graph itself is implemented as a directed graph. JGraphT supports state and edge creation, relation creation as well as graph related functions and algorithms. A supported algorithm is the K-shortest-path algorithm, which is used to determine all sinks in the inferred state flow graph for the test case generation. For more information about JGraphT and its implementation details please have a look at the vendors website [JGr], or its implementation in Crawljax [Cra14] in the source pack-age “com.crawljax.core.state”.

Browser supported crawling An advantage of Crawljax is the use of a browser instance for crawling, simulating user behaviour and execute JavaScript AJAX functionality. Crawljax uses the WebDriver API, which offers the ability to control natively the browser. The WebDriver API is a fully object-oriented Java library and is also used as basis in and since the Selenium-2.0 2 web application test framework release, which substitutes the JavaScript based approach of Selenium-1.0 to execute user behaviour in the browser. In addition to the natively supported browser control, does the WebDriver API supports several different browsers. Crawljax uses it to crawl the target web application and to execute client-side AJAX functionality. The WebDriver API is a well-known Java library, and now part of the Selenium project.

Crawljax - Console interface (CLI) In addition to include Crawljax as a library in a Java application, it comes with a console interface [Cra13], which offers global configuration parameter to specify the crawl behaviour via the console. This enables the user, to use the tool right away without any extra effort. The console interface is predestined to be used within a continuous integration systems.

4.6 Automatically Testing UI States of Ajax (ATUSA)

This section describes the approach proposed in the initial research called “Automatically Testing UI States of AJAX” also called ATUSA [MvDR12]. ATUSA is based on Crawljax and introduces besides improvements and features at the Crawljax core implementation also additional plug-ins targeting the challenge to automatically test AJAX based web applications.

4.6.1 Core functionality

The features which improved the Crawljax core functionality are explained in this section.

Oracle comparators Oracle comparators in ATUSA are used to compare, determine difference or strip non-deterministic DOM elements before a states DOM gets further validated, compared and scanned. They target the oracle problem challenge in testing AJAX applications described in section

4.2. Using oracle comparators was at first introduced by [Roe10] and [RMvD10] and further used in the ATUSA stack in [MvDR12]. ATUSA provides a interface called “Comparator” which defines the following three methods:

(22)

• isEquivalent, validates the old state DOM with the the new state DOM. • getDifferences, lists of the DOM differences, when equivalence check fails. • normalize, strips non-deterministic or other DOM overhead.

The “Comparator” interface must be used when implementing own comparators, or the abstract “AbstractComparator” class can be extended. It provides default implementations of the comparator methods. Furthermore then offering the use of oracle comparators, ATUSA introduces a number of generic comparators, which are useful to ignore differences, strip DOM overhead before checking for DOM equivalence.

Generic comparators The generic comparators listed in the initial research include the following comparators, a White-space, Attribute, Style, Plain structure, List, Table, Regex (Regu-lar expressions) and XPath comparator. For example the provided plain structure comparator removes all element attributes, line breaks and the content before the DOM is used for compari-son. Since the publication of [MvDR12] and of ATUSA in 2012, Crawljax has improved but not the provided generic comparators.

Oracle Comparator pipeline The defined oracle comparators get pipelined in ATUSA, which means that every comparator passes its normalized result further to next comparator to eliminate as many differences as possible before state comparison. Further details about the comparator pipeline can be found in [MvDR12].

Oracle Comparator change The new version of Crawljax, version 4.03 _{will substitute the oracle}

comparators, with the so called DOM strippers. These DOM strippers have the same goal than the oracle comparators, but offer a simplified way to provide them. This will be achieved with the use of JSoup, a new Java HTML parser engine, which provides DOM element selection, traversal and manipulation as the JavaScript library JQuery4_{. Furthermore are the DOM strippers fully thread safe,}

and can be used with several browser instances. Their implementation just started during creation of this thesis and therefore not further included in the conducted case studies of this research.

Invariants The initial research [MvDR12] introduces with ATUSA the ability to use Crawljax for invariant based testing. With the use of invariants the DOM state or state-machine transitions can be validated against defined constraints. The invariants in ATUSA are distinguished into state invariants and state-machine invariants. Both variants offer a generic and application specific scope. Besides the scope lies the difference also in the manual effort required to implement those invariants in ATUSA. The section below describes the implementation of the state invariants, followed by a description about the state-machine invariants. For further information about invariants, their definitions and description about their goal are explained in section4.4.

State Invariants ATUSA supports state invariants expression in two types, generic state invari-ants and application specific invariinvari-ants. The difference between those two types is the scope of the invariants applicability, because some invariants can be in general used for all web applications and some are application specific. This describes the first type of state invariants and is defined as generic state invariants [MvDR12]. These invariants are useful to validate against every new detected state. The initial research describes the use of generic state invariants as oracles [MvDR12] which need to be satisfied on every new detected state. E.g. W3C HTML validation, which validates every state against the W3C HTML standard or Error message detection, to detect server-side produced error messages in the states DOM. The second type of state invariants in ATUSA are defined as application specific state invariants. These invariants include application specific DOM constraints targeting a specific web application in development. The application specific state invariants internally use the different generic conditions as constraint over the DOM. The initial research proposes five different

3_{Crawljax version 4.0}_{https://github.com/crawljax/crawljax/tree/crawljax-4.0} 4_JQuery_{http:/jquery.com}

(23)

generic conditions inclusive logical operators for more flexibility over defining conditions, and these are Regular expression condition, XPath condition, JavaScript condition, URL condition and Visible condition. These conditions are not bounded to the use of application specific invari-ants, these invariants can change more often than generic state invariants. Considering the example, a invariant for a application specific table structure including content validation compared with the changeability of the W3C HTML standard. The initial research does not state any example or imple-mentation detail of these generic state invariants. These invariants need to be implemented in Java inheriting the Crawljax invariant base class, or for several use cases also via the plug-in system. For example a PostCrawlPlugin which could be used as generic state invariant to validate the states and their corresponding DOM-trees from the inherited state flow graph.

Invariant Error reporting - Implementation details Invariant violation reporting is supported by ATUSA’s plug-in system and plug-in hook API. The plug-in system with API is described in section4.6.2. It offers the extensibility to customize error collection and crawl behaviour on invariant violations. A default invariant reporting mechanism is described in ATUSA’s error reporting plug-in in section4.6.2.

State-machine Invariants State-machine invariants are distinguished between generic and appli-cation specific state-machine invariants as defined in4.4. The main difference compared to the state invariants is that state-machine invariants are constraints on the complete inferred state-machine after the crawling phase.

Two examples of generic state-machine invariants are detecting dead clickables, a validation over all states if all candidate elements got successfully fired and backtracking validation, a validation over all states if all edges to other states are bidirectional. These two state-machine invariants target two challenges in testing AJAX web applications described in section4.2.

In general these invariants follow the pattern “Source → Action → Target [MvDR12]”. State-machine related invariants require a implementation as plug-in, because the condition based invariant object and its generic conditions does not support machine validations. Furthermore are state-machine suited for validations after complete creation of the state flow graph. The initial research does not provide further information about the implementation of the state-machine invariants used for conducting the case studies [MvDR12].

4.6.2 Plug-ins

The initial research proposed additionally a plug-in system with several plug-in hooks, to enable the in development and provide additional features for Crawljax within the crawling phase. The plug-in API provides several entry poplug-ints as plug-plug-in types. These plug-plug-in types are: OnFireEventFailed-Plugin, OnInvariantViolationOnFireEventFailed-Plugin, OnNewStateOnFireEventFailed-Plugin, OnRevisitStateOnFireEventFailed-Plugin, OnUrl-LoadPlugin, PostCrawlingPlugin, PreCrawlingPlugin, PreStateCrawlingPlugin. These plug-in types are used by the required plug-ins of ATUSA and are described in this section.

Overview plug-in Generates a overview report of the crawled web applications resulting state-flow graph, to validate crawling result for possible further adjustments to improve state coverage. The overview plug-in generates an HTML report with a graphical representation of the state-flow graph. In addition to the graphical representation it also displays detailed information for every state in the graph. Those information includes the state DOM, all detected candidate elements, a screen-shot including indication of detected and clicked candidate elements, the XPath of the detected candidate elements, simple edge statistics and a overview over the used Crawljax configuration for producing the overview output.

Testing Framework plug-in The test framework is one of the key plug-ins of the research. Sim-ply without this plug-in the whole automated testing approach of ATUSA would be missing. The framework provides a plug-in for test suite generation, a test execution controller for executing the

(24)

test suite and collecting test results and a error report plug-in. The error report plug-in collects the test result and generates a HTML report.

Test-suite generation For generating test-cases ATUSA uses the K-Shortest Path algorithm, this is a generalized version of the shortest-path problem. Which means where several path’s through the graph are thought with increasing order. ATUSA uses all sink nodes from the inferred state flow graph to search for the K-Shortest path from index node (State 0) to the sink nodes. In the initial research paper a short statement is given that loops in the state flow graph are included once. With help of Apache Velocity5, Java JUnit6 test cases are generated. Each test case includes the sequence of events from the index state to the sink state. The sequences include all events used to navigate from the index state to the target state. The test generation process is briefly discussed in [MvDR12], but it is a follow up research, on an earlier publication of A. Mesbah, A. van Deursen and D. Roest. The publication about regression testing [Roe10] and the pre created master thesis of D. Roest [Roe10] describe the test framework in more details.

Test-suite execution The initial research paper proposes with ATUSA also a solution to execute the generated test cases, without using an additional technique. The generated test cases make use of a test execution controller, which includes a embedded browser to navigate through the generated test cases. This means the browser navigates with the test case the new web application and tests its state with the test state. The test execution follows the specification in [MvDR12].

Test reporting Furthermore a test report controller generate a test report including failure and success information about the test execution. It also includes detailed information about, condition check results, invariant validation results and state differences. The test report is essential to see details about the actual error, the JUnit test report does only includes a global test method fail message as described in [Roe10].

4.7 Continuous Integration (CI)

Guidion is currently implementing its own testing environment. This environment is driven by a continuous integration system (CI ). CI enables that the software which is currently developed is continuously build after someone commits a change. Especially in large software projects developers test their own changes but not the whole application. Bringing the whole application in the state to use requires to much effort. CI helps here in automating testing, building and deploying the software to a testing or staging environment [HF10]. Guidion uses as CI system the open-source CI server Jenkins 7_{. The CI system builds the management system were the ATUSA stack should be}

implemented. At the moment of the thesis is one deployment pipeline implemented in Jenkins of Guidion’s web applications. The future vision here is that also the other web applications will follow and for each deployment pipeline a ATUSA testing phase exists. [HF10] states three preconditions before starting to integrate a software project into a CI system.

1. Use of a version control system. Guidion uses Bitbucket8 as Git version control system. 2. Use of an automated build. Most web applications of Guidion use a automated build. 3. Agreement of the team. The agreement of the team is asked within the survey research.

Jenkins Continuous integration server Jenkins is a full featured continuous integration server. It is a Java open-source CI server, with a wide variety of plug-ins, which can be used to create jobs to build, test and release a software project. It supports testing and building software continuously including monitoring and reporting. At Guidion the Jenkins server is installed on a own virtual server, and is fully customizable for integrating ATUSA.

5_{Apache Velocity}_{http://velocity.apache.org/} 6_JUnit_{http://junit.org/}

7_{Jenkins CI}_{http://jenkins-ci.org/}

(25)

Deployment pipeline A deployment pipeline is a manifested automated process to bring the software from the version control to a testing or staging environment. It include several steps to build, test and deploy the software project [HF10]. At Guidion the deployment pipeline consists of a few building steps and a few unit tests to deploy a system to a web server. The plug-in system of Jenkins provides additional support for reporting and uploading the build files to a test environment. After the web application is deployed to a test environment, it is fully prepared to get tested by the user. After the successful deployment to the test environment is the entry point of the ATUSA approach. This can help identifying errors before a group of tester validates the changes. The validation process of the group of tester can be saved when the ATUSA approach successfully test the web applications over a longer time period.

(26)

Chapter 5

Research

The replication of the case studies from the initial research [MvDR12] is divided into five sections. The first section provides a default configuration declaration of Crawljax / ATUSA. Followed by the first replication case study answering research question RQ1. The third section includes the case study replication answering research question RQ2 - RQ3. The following section includes the replication case study answering RQ1 - RQ4. Every replication section evaluates the correctness of the case studies described in the original values, and compares the findings of the replication with the findings of its original. The last section summarizes the replication findings and ends in a conclusion.

Goal of the case study replications The replications goal is to support Mesbah et al. with providing additional evidence on using the ATUSA approach for automatically testing AJAX based web applications. Additionally, it evaluates the described case study scenarios and the use and implementation of the proposed invariants, comparators and plug-ins. Its findings are compared with the findings of the initial research [MvDR12].

Additional Tools used for the replication For the evaluation and implementation of the repli-cation case studies additional tools were used. These tools are further explained in the section5.10.3. In addition to these tools the Netbeans1 _{IDE was used as main development environment for}

imple-menting all required features.

5.1 Cases study setup and default configuration

The replication case study setup followed the described setup from the initial research. In general are two main runs were executed per replication subject system. One in default setup, with additional click elements, and the second run including the created invariants, comparators and plug-ins. Every main run is further executed two times with different browsers.

5.1.1 Crawljax - ATUSA default configuration

The initial research uses Crawljax with its default configuration. The default configuration is not further described. The Crawljax evolution, especially [NMvD14] lead to a change in the default configuration which produces different replication results, and invariants e.g. the application specific state-machine invariants always fail. This is because the default activated clickOnce setting limits the state creation, through clicking identified elements only once. Setting the clickOnce to false solves this problem. Another identified key configuration is the default crawl strategy, which is defined as clickElementsInRandomOrder and set to false. Setting the clickElementsInRandomOrder to true produces different state coverage results, and is further described in paragraph5.1.1. Through the fact that the default configuration was not further described within the initial research, the

Test-suite for automated invariant-based testing of Ajax web-applications

Master Thesis