Tool features for understanding large reverse engineered sequence diagrams

(1)

Tool Features for Understanding Large Reverse Engineered Sequence

Diagrams

by Chris Bennett

B.A., Dalhousie University, 1987 BSc., Dalhousie University, 1995

A Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of

M

ASTER OF

S

CIENCE in the Department of Computer Science

c

Chris Bennett, 2008 University of Victoria

(2)

ii

Tool Features for Understanding Large Reverse Engineered Sequence Diagrams by

Chris Bennett

B.A., Dalhousie University, 1987 B.Sc., Dalhousie University, 1995

Supervisory Committee

Dr. Margaret-Anne Storey, (Department of Computer Science) Supervisor

Dr. Daniel German, (Department of Computer Science) Departmental Member

Dr. Yvonne Coady, (Department of Computer Science) Department Member

(3)

iii

Supervisory Committee

Dr. Margaret-Anne Storey, (Department of Computer Science) Supervisor

Dr. Daniel German, (Department of Computer Science) Department Member

Dr. Yvonne Coady, (Department of Computer Science) Department Member

ABSTRACT

Originally devised as a notation to capture scenarios during analysis and design, se-quence diagrams can also aid understanding of existing software through visualization of execution call traces. Reverse engineered sequence diagrams are typically huge and design-ing tools to help users cope with the size and complexity of such traces is a major problem. While preprocessing may be necessary to reduce the complexity of a sequence diagram, interactive tool support is critical to help the user explore and understand the resulting dia-gram. This thesis examines tool features necessary to effectively support sequence diagram exploration by reverse engineers. Features were derived from a literature survey and empir-ically evaluated using an exploratory user study. The tool features were further evaluated by situating them within theories of cognitive support.

(4)

iv

List of Tables

Table 3.1 Comparison of features implemented by selected sequence

visualiza-tion tools . . . 22

Table 4.1 Feature implementation in the OASIS Sequence Explorer . . . 26

Table 6.1 Assigned tasks . . . 32

Table 6.2 Tool operations mapped to tool features . . . 35

Table 6.3 Survey questions arranged by tool features . . . 36

Table 6.4 Feature usefulness and improvements . . . 46 Table 7.1 Mapping cognitive design elements to sequence diagram tool features 60

(9)

ix

List of Figures

Figure 1.1 A simple sequence diagram. . . 3 Figure 2.1 Thesis methodology . . . 9 Figure 3.1 ISVis scenario view showing overview pane. . . 12 Figure 3.2 Program Explorer invocation views plus definition and inheritance

views. . . 13 Figure 3.3 TPTP UML 2.0 sequence trace. . . 14 Figure 3.4 VET sequence diagram and class association diagrams plus filter pane. 15 Figure 4.1 The views of the OASIS Sequence Explorer . . . 24 Figure 6.1 Overall feature use for all users . . . 38 Figure 6.2 Percentage feature use by task . . . 40

(10)

x

Acknowledgement

I would like to thank Margaret-Anne Storey for wisdom and optimism, Daniel Ger-man for insights and ‘outside the box thinking’, and Del Myers for building an excellent sequence viewer. Thanks also to Martin Salois, David Ouellet, and Phillipe Charland for making the focus group and study a reality, Chris Callendar for supplying the target appli-cation for the study, Sean Falconer for his reviewing insights, and all the DRDC Valcartier employees, UVic, and industry participants who took part in the study.

(11)

Chapter 1 Introduction

Software maintenance is the most expensive part of the software lifecycle, consuming more than 50% of lifecycle costs [1]. At least half of the effort spent on software maintenance is devoted to reverse engineering, and in particular, program comprehension [2]. A strong motivation for the work described in this thesis is that program comprehension is a difficult and time consuming task. Extensive research has been carried out to develop tools to help the reverse engineer with this. The fruits of this research are beginning to appear in mainstream tools (e.g. integrated development environments such as Eclipse [3] and specialized reverse engineering tools such as Klocwork [4]). However, issues such as low industry adoption and tool inefficiency are still relevant [5].

Software maintenance tasks include fixing defects, adding new features, migrating soft-ware to new architectures, and refactoring existing systems to cope with entropy. A precur-sor to carrying out any of these tasks is understanding how a system implements a given piece of functionality. The code that implements the functionality to be fixed, modified, or ported must be understood, along with enough context to reduce the chance of undesirable side effects. Research into program understanding has found that this can be facilitated by mapping the programmer’s mental model of the executing application (the domain model) to the source code that implements the application (the program model) [6]. An intermedi-ate (or situation) mental model is required for this mapping [6]. This mental model cross references and ties together the program and domain models and may be built by either top-down hypothesis formation from the domain model or bottom-up chunking from the

(12)

1. Introduction 2

program model (or by a combination of these two approaches).

An intermediate ‘dynamic’ mental model can be created by manually tracing through the code to understand how domain behaviour is carried out by source code. Tool support is available within modern integrated development environments, including hyperlinking between method calls and source code formatting. A disadvantage of this approach is the uncertainty caused by multiple possible execution paths (especially when working with lan-guages that support late binding and polymorphism). Another barrier to understanding is the cognitive overhead of having to manually select and follow an execution path. Debug-ging tools provide an alternative approach to assist in this tracing. Tool support includes the ability to step through executing code, set breakpoints, watch variables, and examine the call stack. Some downsides to this approach are the inherently transient and linear nature of such a trace, the inability to return to an earlier state, potential problems with timing and thread synchronization introduced by the debugger, and the difficulty of deciding where to place breakpoints to begin examining the source code. A third method is to perform a ‘post mortem’ analysis of an execution trace. According to [5], this approach can yield a key intermediate mental model, a model that can be built with the help of reverse engineered sequence diagrams. We focus on this approach in this thesis.

Sequence diagrams depict the flow of logic within a system and can be a valuable aid to understanding system behaviour. The sequence diagram for an object-oriented system exposes system logic using a sequential ordering of messages (shown as labeled horizontal arrows) passing between objects (see Figure 1.1). Objects are listed across the top of the diagram from left to right with vertical ‘lifelines’ hanging down below each object The first message starts in the top left corner, the next message appears just below that one, and so on. Return values may be shown explicitly or implicitly.

The sequential system behaviour shown in a sequence diagram is known as a scenario. In Kruchten’s 4+1 architectural view model [7], scenarios are the glue that ties together the four primary system views, highlighting the most important requirements of a system. Sequence diagrams have their origin in various forms of message sequence charts such

(13)

1. Introduction 3

Figure 1.1. A simple sequence diagram.

as Arrow Diagrams, Extended Sequence Charts, Message Flow Diagrams, and Time Se-quence Diagrams [8]. While originally devised as a notation to capture scenarios during analysis and design (e.g. Rumbaugh’s OMT event traces [9]), sequence diagrams have been used for requirements elicitation [10], architecture validation [11] and test generation [12]. Sequence diagrams can also aid understanding the dynamics of existing software through visualization of execution call traces. Their power lies in their ability to represent selected behaviour at a suitable level of abstraction.

Reverse engineered sequence diagrams can be created through static or dynamic anal-ysis. When created by static analysis, the source code is analyzed and a call tree extracted and visualized. Using dynamic analysis, the system is executed and a call trace is cap-tured (e.g. by using instrumented source code). The advantages of dynamic analysis are increased precision, control over inputs and conditional behaviour, as well as reso-lution of polymorphism and runtime binding in object-oriented languages [13]. However, sequence diagrams based on dynamic call traces are typically very large, making them both technically difficult to display and cognitively challenging to explore and understand.

(14)

1. Introduction 4

The sequence diagram ‘size explosion’ [13] problem has been tackled in two main ways: through pre-processing (also known as analysis) to reduce the size of the initial sequence and through tool support for user interaction. Pre-processing techniques include: reduc-tion at the source through data collecreduc-tion techniques and sampling [13]; collapsing similar sequences using pattern matching (to identify loops, recursion, and non-contiguous repeti-tions); automatic detection of utility functions (e.g. by using fan-in/fan-out metrics) [14]; removing abstract operation calls [14]; hiding constructors and getters/setters [15]; limiting the depth of the call tree [14] [15]; eliminating self calls [16]; and selecting relevant areas and hiding details [17] [18].

While pre-processing may be necessary to reduce the complexity of a sequence dia-gram, interactive tool support is needed to help the user explore and understand the result-ing diagram. Storey et al. [19] propose that information exploration is the most important reverse engineering activity as far as program comprehension is concerned. Tool support for this is known as ‘cognitive support’ and refers to the assistance that a tool provides to users to help them with their thinking and problem solving [20]. Reverse engineering is mentally challenging, and effective cognitive support can improve the performance of re-verse engineers by offloading some or all of a user’s cognitive processing onto an external artifact (e.g. by reducing the need to remember large volumes of detailed information or to perform repetitive calculations) [20]. Such cognitive support can be provided by the fea-tures available in a sequence diagram viewer and applied to a range of reverse engineering tasks including design and architecture recovery, feature location, design pattern discovery, and re-documentation at various levels of abstraction.

As described above, considerable research has been done on how to automatically re-duce the size of sequence diagrams. However, less work has been done in the area of vi-sualization and user interaction. While several sequence diagram research tools have been developed over the last 15 years, the user interaction aspects of these tools has received limited formal evaluation (exceptions being [21], [22], and [23] as discussed in Chapter 8). In their paper on interactive exploration of UML sequence diagrams, Sharp and Rountev

(15)

1. Introduction 5

emphasize this by stating: “...what are the actual benefits of the visualization techniques? Experiments with programmers or university students should be used to quantify the bene-fits of individual tool features.” [24]

In parallel to the development of sequence diagram tools, researchers have developed theories of program comprehension and cognitive support (e.g. see [6] and [19]). These theories have begun to be applied to the development of reverse engineering tools. How-ever, this has yet to be done for sequence diagram tools. It appears that the features of these tools have mainly been derived from the experience and intuition of their builders (an exception being Zayour’s DynaSee [5], discussed in Chapter 8). It is important to be able to clearly explain why a tool feature provides cognitive support; this improves our ability to discuss and test features as well as to reuse design knowledge [20].

This work attempts to address these gaps in the research and begins to answer the following research questions:

1. What interaction and presentation features are found in state-of-the-art sequence di-agram tools?

2. How useful are sequence diagram tool features in helping with real-world reverse engineering tasks?

3. How can sequence diagram tools be improved?

4. What basis do sequence diagram tool features have in cognitive support theory? To answer these questions, we established a baseline feature set by reviewing the re-search literature on sequence diagram tools and investigating representative commercial tools. Features from each tool were used to compose a feature list for a hypothetical se-quence diagram tool. This is presented in Chapter 3.

Once we had a baseline feature list, we needed to validate the usefulness of these fea-tures. To do this, we first developed a sequence diagram tool that contained all the features identified in our survey. This tool, developed by a colleague in the CHISEL research group, is described in Chapter 4. We then ran a focus group to help us craft appropriate reverse

(16)

1. Introduction 6

engineering tasks for a user study. The focus group and its results are discussed in Chapter 5.

Next, using the focus group results, we designed and carried out a preliminary user study in which participants performed reverse engineering tasks with our tool. From this study we have begun to validate the usefulness of existing tool features and to identify some possible feature improvements. The study design, results, and discussion can be found in Chapter 6.

To validate the cognitive basis for tool features, we turned to research on program comprehension and cognitive support. Each tool feature was then analyzed against a set of cognitive design elements to situate the feature within theories of cognitive support. A brief background on theories of program comprehension and cognitive support in tools is followed by an analysis and results in Chapter 7.

Chapter 8 discusses related work, focusing on evaluation of sequence diagram tool features and Chapter 9 concludes and points to possible future work. We begin, in Chapter 2, by outlining the research methodology we employed.

(17)

7

Chapter 2 Research Methodology

The research methodology we used consisted of four steps: a literature survey focusing on sequence diagram tools, a focus group, a user study, and a theory-based analysis. Figure 2.1 shows these steps as ellipses and the artefacts used and produced by each step as boxes. The results are shown as a rectangle at the bottom of the figure. We chose this approach because it provided a solid baseline feature set, empirical validation of tool features, and a parallel theory-based validation of these features.

To answer research question 1, “What interaction and presentation features are found in state-of-the-art sequence diagram tools?”, we conducted a detailed review of the literature and commercial tools that support sequence diagram generation and exploration. From this review, we synthesized a summary of interaction and presentation tool features. Although this summary is useful, we learned that most tool features lack a solid user-based evalua-tion. Hence it is unclear how useful these features are during program understanding tasks. This feature set may also be incomplete or inaccurate, reflecting the experiences and biases of the individual researchers, rather than true end-user requirements.

To begin to address these issues, a focus group was carried out by a combined team of researchers from the CHISEL lab at the University of Victoria and researchers from Defence Research and Development Canada (DRDC) in Valcartier Quebec. The subjects in the focus group were software maintenance professionals from DRDC Valcartier, chosen for their experience in software reverse engineering. The purpose of this focus group was to help us understand their current reverse engineering methods and expose frustrations they

(18)

2. Research Methodology 8

encountered during their work. This information was valuable in the design of the realistic tasks used in the observational experiment. The focus group helped us to answer research question 2: “How useful are sequence diagram tool features in helping with real-world reverse engineering tasks?’.

Following the focus group, we carried out a user study that used a mixed-methods methodology. The approach was both explanatory and exploratory. The user study was explanatory in that it validated features and helped us to answer research question 2. The study was also exploratory in that it extended the feature set to meet actual user require-ments, answering research question 3: “How can sequence diagram tools be improved?”.

The user study consisted of a lab experiment, interviews, and a follow-up question-naire. An experiment to observe and measure how users exercised sequence diagram tool features was designed and carried out, again by a combined team of University of Victoria and DRDC researchers. The experiment asked participants to complete a number of real-istic software reverse engineering tasks using a sequence tool that contained the features identified by the tool survey. We were able to observe participants at work and quantify how they exercised each feature. We were also able to identify when participants had to use methods other than the sequence diagram tool to complete the assigned tasks. The effectiveness of each participant on these tasks was evaluated. The results from this ex-periment were then used to validate the initial feature set and to uncover possible feature enhancements. Findings from the experiment were augmented by interviews and a follow-up survey to further explore what features participants found useful and to identify features they felt were missing.

Cognitive support principles relevant to reverse engineering tools were selected, based on the Cognitive Design Elements proposed by Storey et al. extended by the perceptual substitution heuristic proposed by [20] on the basis of [25] and [26]. Cognitive Design El-ements were chosen because they represent a pragmatic mechanism at an appropriate level of abstraction for mapping cognitive support theories to tool development. We analyzed each tool feature in the context of these elements in an attempt to answer research question

(19)

2. Research Methodology 9

Figure 2.1. Thesis methodology

4: “What basis do sequence diagram tool features have in cognitive support theory?”. In the next chapter we will establish a baseline feature set for sequence diagram reverse engineering tools.

(20)

10

Chapter 3 Survey of Sequence Diagram Tool

Features

This chapter surveys sequence diagram tools to discover the features they provide and to answer research question 1: “What interaction and presentation features are found in state-of-the-art sequence diagram tools?”. A literature search was carried out to identify sequence diagram tools that in some way tackled the issues of visualizing large reverse en-gineered traces. The majority of tools surveyed implement some form of sequence diagram visualization, although some tools (e.g. SEAT [27]) were included because they provide a functionally equivalent visualization. Sequence diagram user interface features can be di-vided into two categories: 1) presentation or display facilities (Presentation Features), and 2) features that allow the user to interact with and explore the diagram (Interaction Fea-tures). Note that there may be some overlap between these sets. Presentation features are often both the result of interaction, and also a necessary precursor to it (e.g. highlighting and hiding could be considered interaction as well as presentation features).

3.1 Selected Sequence Diagram Tools

Before discussing these features, we consider a representative subset of tools from which these feature sets were obtained. This subset was selected based on the prominence of these tools in the research literature (or software development industry) and the availability of

(21)

3.1 Selected Sequence Diagram Tools 11

information describing their presentation and interaction features. The surveyed sequence diagram tools include research prototypes and one commercial tool. These tools are briefly described below and Table 3.1 compares the features implemented by each of tool.

Interaction Scenario Visualizer (ISVis) [28] is an architecture visualization tool helpful in gaining behavioural understanding, recovering design rationale, or validating the design and implementation of legacy systems (see Figure 3.1). As well as a detailed trace display, the tool incorporates an information mural display to provide a high level overview of large sequence traces. ISVis supports searching for and identification of interaction patterns -recurring patterns within a given trace.

Figure 3.1. ISVis scenario view showing overview pane.

Jinsight [29] was designed to explore and visualize aspects of a Java program’s runtime behaviour. The tool consists of an instrumented Java virtual machine for extracting trace and metric data and several types of dynamic analysis visualizations including sequence diagrams (invocation browser or execution pattern view), histogram view, execution view, and a reference pattern view that summarizes object connections.

(22)

3.1 Selected Sequence Diagram Tools 12

Program Explorer [30] is a combined static and dynamic C++ visualization tool that in-cludes structural views as well as a sequence diagram-like invocation view (see Figure 3.2). Views are linked and are also integrated with source code.

Figure 3.2. Program Explorer invocation views plus definition and inheritance views.

The Shimba [22] reverse engineering environment supports the understanding of Java programs and integrates RIGI [31] for static and SCED for dynamic analysis. Within this environment the Scenario Editor (SCED) tool [22] provides visualization of both state and sequence diagrams and the automatic creation of state charts from sequences. This allows two orthogonal views of dynamic behaviour - one that cuts across objects and one that

(23)

3.2 Presentation Features 13

examines an object’s internal behaviour.

SCENario Environment (SCENE) [32] provides Object Modeling Technique (OMT) style sequence diagrams linked to source code and static views via hyperlinking. Object state during execution can also be accessed. To cope with complexity, SCENE supports partitioning of diagrams into multiple parts, focusing on a singe object (projection), and collapsing of calls.

SEAT does not display sequence diagrams, but instead uses equivalent tree-structured trace views. It also employs techniques for handling very large traces, including flexi-ble and predefined filtering algorithms and group/hide functionality. SEAT uses Compact Trace Format for trace storage to help cope with technical scalability issues.

Together Control Center [33] is a commercial integrated software development environ-ment that supports both forward and reverse engineering. It provides the ability to reverse engineer sequence diagrams using static analysis.

TPTP [34] is part of the Eclipse project [3] and was designed to assist in profiling and runtime performance improvement of distributed Java systems (see Figure 3.3). As part of its thread analysis capabilities, TPTP provides UML 2.0 sequence diagram visualization from captured trace data.

Visualisations of Execution Traces (VET) [35] is a prototype sequence trace visualiza-tion environment that implements the visualizavisualiza-tion approach advocated by Card et al. [36] of showing all information up front in an overview and supporting exploration through zooming and detail on demand (see Figure 3.4). VET is implemented as a plugin and supports extensible filtering plugins. It also provides linking to static views.

3.2 Presentation Features

Presentation features affect the layout of the diagram, as well as facilities for showing multiple views, hiding information, and making the most effective use of animation and visual attributes. These features were extracted from a review of common user interface

(24)

Figure 3.3. TPTP UML 2.0 sequence trace.

functionality provided by the surveyed tools. Typically each feature is found in several tools, but may be implemented in a different way. Several papers refer to the same concepts using different terms (e.g. querying and searching or filtering and hiding). To deal with this, we selected what we felt to be the most easily understood term for our feature list.

3.2.1 Layout

An important presentation feature is laying out a sequence diagram according to some notational standard. Many tools use their own layout format or some variation on a standard format (e.g. UML 2.1), sometimes adding proprietary extensions to address a specific problem (e.g. how to capture conditional branches). Scene produces sequence diagrams

(25)

Figure 3.4. VET sequence diagram and class association diagrams plus filter pane.

according to Rumbaugh’s OMT notation [9]. SCED uses its own UML-like notation that provides constructs for nested sub-scenarios and repetition. TPTP also uses UML (see Figure 3.3).

3.2.2 Multiple Linked Views

It is often necessary to provide multiple views [7] as well as an overview. Views can be of the same type (e.g. to allow comparison of different parts of a trace) or different types (e.g. linked class diagram and sequence diagram views). Ovation [37] renders sub-trees using a number of alternative ‘charts’, including a static class list or a class communication graph. SCED supports sequence diagrams and state charts that show transitions within a selected

(26)

object. Linking these views so that they remain synchronized and can be easily navigated is another useful feature. SEAT provides links between sequence and source code views. Similarly, Scene links between sequence views and static class diagrams or source code views. An overview is provided by many tools. ISVis [28] provides a two window scenario view consisting of an information mural overview and a temporal message-flow diagram (see Figure 3.1). Scene displays a summary call matrix view alongside a sequence view.

3.2.3 Highlighting

Highlighting a section of a sequence diagram is often the expected visible outcome of a user selection or search. Tools that support manual selection of components may use high-lighting to indicate selection (e.g. ISVis). Highhigh-lighting can go beyond single components to show related objects or messages.

3.2.4 Hiding

Hiding information is commonly used to control complexity in sequence diagram tools. Hiding provides abstraction by removing detailed sub-message calls from below a parent call. Components can be hidden following preprocessing, a search (filtering), or a manual selection. ISVis supports hiding of classifiers within a subsystem, SEAT supports man-ual hiding, and VET hides elements following filtering (see Figure 3.4). Similarly, when grouping occurs in SEAT (described in more detail below), the grouped elements are hid-den ‘under’ a summary element. When components are hidhid-den as a result of filtering, it is important to indicate this so that the user can redisplay these components if required. One technique is to ‘grey out’ the components, rather than completely hiding them. There should also be an indication of why a set of components was hidden (e.g. as a result of loop detection or pruning of utility functions) [14]. Cornelissen et al. [15] propose hiding null return values or abbreviating return values and parameter lists.

(27)

3.3 Interaction Features 17

3.2.5 Visual Attributes

Colour and shape are useful ways to code additional information about a sequence. Ovation uses colour to differentiate objects and beveling to indicate that components are grouped under the beveled component. TPTP uses colour to indicate the length of time spent inside a method execution.

3.2.6 Labels

Classifiers, messages, and return values are usually labeled. Occlusion and legibility are challenges when displaying larger sequences. Techniques to cope with this include hiding labels, replacing them with rectangles when zoomed out (e.g. as implemented by the VET tool), or using mouse hovers (e.g. as in Ovation).

3.2.7 Animation

Many tools support animation. This comes in two varieties: one that supports stepping through a sequence diagram message by message, and another that uses animation to morph between diagram states to help the user maintain context. Scene supports single step ani-mation between trace calls. AVID [38] supports aniani-mation between component groupings.

3.3 Interaction Features

Interaction features allow the user to communicate with the tool to navigate, query, and manipulate the sequence diagram. Similar to the presentation features, these features were based on the common interaction functionality we saw in the surveyed tools and again, most features were seen in several tools. We aimed for a similar level of abstraction in our interaction features; i.e., if hypothetically, one was to implement a tool with these features, each feature would appear at the same level in the tool’s menu.

(28)

3.3.1 Selection

Manual selection of elements is a prerequisite for further interaction such as manipula-tion, filtering, and slicing. This is provided by most tools that support any degree of user interaction.

3.3.2 Component Navigation

Rapid, simple movement between components (traversing the call tree) is important to usability [14], as is the ability to move between instances of the same type of pattern (e.g. sub-scenarios) in tools that support grouping of similar patterns (e.g. SEAT).

3.3.3 Focusing

Focusing on a specific part of a diagram or behaviour has been identified as a problem when dealing with large traces [13]. Koskimies and M¨ossenb¨ock [32] note that it can be solved by techniques such as collapsing calls, partitioning sequences into manageable chunks, and selecting an object such that only related messages are shown. Single step animation can also be used to focus on individual messages.

3.3.4 Zooming and Scrolling

Zooming and scrolling [13] are standard techniques to cope with more information than can be legibly shown in a single window. Semantic zooming reveals more details as the user zooms in, while physical zooming simply enlarges the contents. VET (see Figure 3.4), Ovation, TPTP (see Figure 3.3), and Jinsight support zooming and scrolling. Jinsight sup-ports semantic zooming as well.

(29)

3.3.5 Querying and Slicing

Queries identify and optionally filter information within a sequence. Scenariographer [39] supports both relational SQL and set-based Software Modelling Query Language queries on underlying structured data. ISVis allows exact, inexact, and wild-card searches. VET provides graphical support for selection of objects based on class and name as well as selection of methods by type or time range. While these are more limited than language-based queries, they provide a much simpler solution. Slicing can be performed on either objects or methods and is a specific form of query that selects only entities related to the selected component (a slice through the sequence flow).

3.3.6 Grouping

Grouping can be the result of slicing or it can be done manually (e.g. AVID’s manual clus-tering and Ovation’s flattening and underlaying). This is usually indicated by some sort of icon or visual attribute of the summary component (behind which grouped components are hidden). Grouping of objects will result in collapsing the sequence horizontally, but may leave all messages visible (no vertical compaction). However, Cornelissen et al. [15] describe a technique to collapse lifelines that would eliminate calls between the merged objects. Grouping at the message level will hide messages called by the grouped mes-sages (vertical compaction). Grouped items can also be annotated with a label (and op-tionally comments) describing the grouped abstraction. Riva and Rodriguez refer to these approaches as vertical and horizontal abstractions [40]. In addition to preprocessing to de-tect repeating patterns, interaction support can allow manual selection and collapsing of repeated patterns such as loops. TPTP supports grouping of lifelines using predetermined levels of abstraction (host, process, thread, class, and object), grouping of method calls, and arbitrary user-defined groupings.

(30)

3.3.7 Annotating

Annotating can be used for many purposes: to describe why components were grouped [41], to capture user understanding during exploration of a sequence diagram [5], and to provide waypoints1 _{[42] and messages to oneself and others when the diagram is to be shared.}

Few tools provide annotation mechanisms, ISVis being an exception with its facility to describe user-identified sub-scenarios. More common in a web setting (where they are known respectively as social bookmarking and social tagging), bookmarking and tagging can be considered extended forms of annotation. They typically include support beyond basic labeling in such areas as metadata, organizing (e.g. into hierarchies), searching, and visualizing.

3.3.8 Saving Views

Saving views, either to share or to revisit, is also important when documenting a user’s un-derstanding of the diagram. A tool should be able to save the entire state of the visualization so it can be restored at a later time. Together with annotations, a saved view can tell a story about the diagram being visualized. Hamou-Lhadj et al. [14] discuss the need to save both the original trace and the transformations that were applied to reduce the complexity of the trace, although saving a record of user interactions is not discussed.

3.3.9 Summary

Table 3.1 summarizes these features and maps them to the subset of sequence visualization tools discussed above. Note that this feature matrix was derived from literature reviews and limited tool trials. The features listed can be implemented in a variety of ways and we considered a partial implementation to warrant a mapping from feature to tool (indicated

1_{Based on geographic coordinates used in physical world navigation, waypoints provide a mechanism for} marking locations in an artifact.

(31)

Table 3.1. Comparison of features implemented by selected sequence visualization tools

ISV is Jinsight Pr ogram Explor er SCED / Shimba

Scene Seat Together

Contr ol Center TPTP VET Presentation Features Layout • • • • • • • • •

Multiple Linked Views • • • • • • • • •

Highlighting • • • • • Hiding • • • • • • • Visual Attributes • • • • • • • • • Labels • • • • • • • • • Animation • • Interaction Features Selection • • • • • Component Navigation • • • • • Focusing • • •

Zooming and Scrolling • • • • • • • • Queries and Slicing • • • • • • • • •

Grouping • • • •

Annotating •

(32)

with a bullet in the table). In addition to research tools, several commercial and open source tools have capabilities to reverse engineer sequence diagrams. For Java, these include IBM’s Rational Rose Enterprise [43], Omondo’s EclipseUML [44], and Borland’s Together Control Center. Borland’s Together Control Center is included in the table because it is representative of the level of interaction support in commercial tools (i.e. basic support for generation of sequence diagrams via static analysis). The next chapter outlines how this feature set was implemented in the OASIS Sequence Explorer, a tool developed at the University of Victoria for research into sequence diagram visualization and exploration.

(33)

23

Chapter 4 Implementing a Sequence Explorer Tool

The OASIS Sequence Explorer (OSE) [45] was developed from the features detailed in Chapter 3 by Del Myers of the CHISEL research lab. It was implemented in Java as a set of four Eclipse views illustrated as A to D in Figure 4.1. The choice of a custom-built viewer for use in further user studies was driven by the fact that no existing tool incorporated all the features that were found during the literature review. Moreover, of the available tools, some were proprietary and others were not designed specifically to support general reverse engineering tasks (e.g. the TPTP tool [34] focuses on performance profiling and captures incomplete traces). Having complete control over the implementation of tool features also allowed integration with the Eclipse IDE; this was invaluable when observing sequence diagram use within the context of extensive program understanding tasks. Finally, the tool was optimized for performance on larger traces to help ensure that this was not a barrier to task completion. The design and implementation of this tool are described in [45] and are summarized below.

View (A) is the Sequence Diagram Editor, which is split into three sub-views or panes. The main pane (A-1) displays a navigable sequence diagram , the clone pane (A-2) displays the same information, but horizontal scrolling to support widely spaced method calls, and (A-3) contains a navigable package hierarchy of objects/classes in the sequence diagram.

Sequence diagrams are visualized using enhanced UML 2 notation. Diagrams consist of a series of labeled object/class lifelines with each containing one or more method activation boxes. Sequence diagrams may be too large to view at once. To cope with this, the user

(34)

4. Implementing a Sequence Explorer Tool 24

Figure 4.1. The views of the OASIS Sequence Explorer

may scroll, zoom, fit to window, or specify a legible text layout of a portion of the diagram (the default view). A linked outline view (Figure 4.1(B)) displays the complete chart with the visible area shown as a user-controlled blue rectangle.

A method activation may have child activations shown using a coloured border and a small plus sign. Clicking on an activation expands it to show child activations. Similarly an expanded activation (shown by a minus sign) may be collapsed. Lifelines are not shown unless there is a visible path from the chart’s root activation to an activation on that lifeline. This allows the user to group and hide activations and lifelines according to their place in the call hierarchy. It is also possible to re-root the chart by focusing on an activation and its children (hiding everything else). A breadcrumb trail showing the activations from the original root to the currently selected root is shown at the top of the editor and any level in this call hierarchy can be selected to reset the root.

(35)

The Sequence Diagram Editor supports grouping in two other ways. Lifelines can be grouped under their package in the package pane (which shows the static class hierarchy). Automatic grouping is also provided through loop recognition during sequence trace pre-processing. Activations with the same method signature and which originate from the same activation are considered to be equivalent. A repeated sequence of equivalent activations is identified as a loop iteration. Repeated activation sequences are enclosed in a box labeled with the number of iterations and the number of the currently displayed iteration. The user can change the displayed loop iteration, all loops can be shown, or loops can be collapsed into a vertical group. By default, only one iteration is displayed.

The OSE tool supports searching using substring matching and regular expressions. An Eclipse Search view (Figure 4.1(C)) is populated with the search results, which are linked to and highlighted in the Sequence Diagram Editor.

Elements of the sequence diagram may have properties that are not visible in the Se-quence Diagram Editor (e.g. activations have a time associated with them). These prop-erties can be displayed in the Eclipse Propprop-erties view (Figure 4.1(D)). Annotations can be entered using this view and these can also be searched and later edited.

The OSE tool also supports various other features including package filtering to remove selected elements, highlighting of activation sub-trees, and animation during diagram re-layout to help maintain user context. Double clicking on an element opens a source code editor for that element if the source code is available. Finally, the state of the Sequence Diagram Editor is saved so that the diagram returns to a familiar state when reopened. Ta-ble 4.1 maps each feature described in Chapter 3 to the way that the feature is implemented in the OSE tool.

In the next chapter we look at a focus group that was carried out to discover how soft-ware reverse engineers go about their jobs and to help us understand how to design a lab experiment to validate sequence diagram tool features.

(36)

Table 4.1. Feature implementation in the OASIS Sequence Explorer Feature Implementation

Layout Sequence diagram layout; fit-to-screen layout.

Multiple Linked Views Linking between editor, outline, search, package pane and main/clone panes, properties view, and source ed-itors.

Highlighting Highlighting according to selection and search matches.

Hiding Filtering based on packages; grouping/collapsing ele-ments implicitly hides other eleele-ments.

Visual Attributes Use of colour to indicate activations that can be ex-panded/collapsed; ’+’ and ’-’ decorations for ele-ments with children; figure shapes modeled after fa-miliar UML notation.

Labels Every element has a label drawn according to familiar UML notation.

Animation All interactions with the diagram are animated to maintain context.

Selection Every element is selectable.

Component Navigation Elements can be navigated to using the search view. Also a breadcrumb trail can be used to navigate up the call hierarchy.

Focusing Focus on an activation to make it the root of the dia-gram.

Zooming and Scrolling Standard zoom in/out operations; zoom to marquee; standard scrolling using scrollbars; linked outline view scrolls the Sequence Diagram Editor.

Queries and Slicing Ability to search on any element in the sequence. Grouping Grouping of activations, packages, repeated patterns. Annotating Annotating in properties view and within the

Se-quence Diagram Editor.

(37)

27

Chapter 5 Focus Group

5.1 Focus Group Design

The focus group session was carried out with three professional software developers from DRDC Valcartier. The goal of the focus group was to explore reverse engineering tasks that software developers carry out in their normal work, with an emphasis on how sequence diagrams are, or could be used to assist in these tasks. According to [46] a tool will be most useful when the problem representation and task align with each other. The focus group was intended to begin to answer research question 2: “How useful are sequence diagram tool features in helping with real-world reverse engineering tasks?”.

In the first part of the focus group, participants were asked to describe the tasks they performed as part of their job, including design, coding, maintenance, code review, bug fixes, and testing. They were also asked to describe the tools and techniques they used in their work (e.g. use of debugging and visualizations). More focused questions were posed to determine how participants carried out feature location, as well as other program under-standing tasks relevant to software maintenance. In the second part of the focus group, we demonstrated the OSE tool and asked participants to comment on the tool and to describe how they might use it in their work. Minutes from the focus group session can be found in Appendix B.

(38)

5.2 Focus Group Findings 28

5.2 Focus Group Findings

Each of the three professionals in the focus group (referred to as participants F1, F2, and F3) had different work roles and varying perspectives on the reverse engineering process. Par-ticipant F1’s main focus was on software development, debugging and testing. ParPar-ticipant F2’s work involved integration and investigation of open source components. Participant F3 primarily performed code and technical reviews. All three participants used Eclipse and Java in their work and worked on teams of from four to six people.

In their jobs, F1 and F2 were concerned with finding features and understanding how they were implemented in software. To do this, F1 mostly read the code, documenting his understanding with pen and paper. He did not use a search facility and rarely used a debugger, finding it too difficult to set breakpoints in interesting places. F2, on the other hand, used third-party software to create class diagrams. He then used a debugger to under-stand the functionality of the software. Finally, he drew small sequence diagrams manually using a design tool to document this understanding. F3 was more concerned with under-standing the impact of code changes. For this, he depended on source code repositories, synchronization tools, and a debugger. He first located the code changes and then used the debugger to analyze their impact. F2 observed that few of his colleagues used sequence diagrams in their daily work.

Following the initial discussion, the focus group participants provided feedback on the OSE tool. F2 said that he liked the split panes and felt it could replace some of the other tools he used. He did note it would be preferable to be able to specify the start of the se-quence trace, rather than tracing an entire program run. F3 noted that the tool could also be used to optimize a program if it displayed profiling information such as method execution duration and memory used. He also suggested that being able to visualize multiple threads and their interactions would be useful. All participants agreed that the ability to annotate diagrams would be useful, but not a critical feature because they typically documented for personal use, not for sharing with others.

(39)

5.3 Focus Group Discussion 29

5.3 Focus Group Discussion

The feedback of focus group participants was used to guide the design of our study. Al-though the number of participants in the focus group was small, it appears from the discus-sion that not all reverse engineers use sequence diagrams for program understanding. This led us to add a basic training session on sequence diagrams to our user study. Participants indicated that they used sequence diagrams to support their own understanding but did not share their notes or diagrams with other team members. Consequently, the study was set up to focus on individual tasks rather than collaborative tasks. The focus group also led us to include tasks that dealt with understanding how specific features were implemented (i.e., tasks T5 to T8 as discussed in the next session).

The next chapter describes the user study that builds on the information we gained from this focus group.

(40)

30

Chapter 6 User Study

6.1 Study Design

The user study was the primary mechanism we used to understand the effectiveness of se-quence diagram tool features and to answer research question 2: “How useful are sese-quence diagram tool features in helping with real-world reverse engineering tasks?” and research question 3: “How can sequence diagram tools be improved?”. While the focus group shed some light on reverse engineering practices and the use of sequence diagrams in reverse en-gineering, it was clear that a user study would be needed to evaluate the relative usefulness of tool features. The user study consisted of a lab experiment designed to capture feature use on a set of reverse engineering tasks, an interview to elicit supplementary qualitative information, and a follow-up survey to fill in gaps in our understanding of the experiment results. The OSE tool described in Chapter 4 was used in this user study.

6.1.1 Lab Experiment

For the lab experiment, six participants were recruited using convenience sampling. Five computer science graduate students at the University of Victoria and one participant from industry were recruited. All participants had some experience programming in Java and were familiar with the Eclipse software development environment,. However, the level of programming and reverse engineering experience varied considerably. Each participant was given the same tasks related to one application, a multi-player distributed Tetris game

(41)

6.1 Study Design 31

written in Java and consisting of 10,500 source code statements and 57 classes in 8 pack-ages.

This multi-player Tetris application was selected because it was used in a previous study [47] and was considered appropriately complex for the study tasks and time available. The application was also developed by a colleague in the CHISEL research lab, providing a privileged understanding of its architecture and functionality.

The experiment began with a comprehensive thirty minute tutorial to provide hands-on experience using the OSE tool. During the tutorial, the participant created a program trace and navigated, searched, and manipulated the resulting sequence diagram. The participant was then asked to create a new program trace by running and interacting with the Tetris application. This trace capture resulted in creation of three sequences - one for each thread. Thread 1 contained the main application setup and processing, thread 13 captured the Java AWT user interface events, and thread 15 contained the game block monitoring processing. Following trace capture, each participant was then given 90 minutes to complete nine assigned study tasks, the selection of which was based on the focus group and previously published studies [23] [48]. Consideration was given to the common tasks identified during the focus group. Tasks were arranged in order of increasing difficulty to guide the partici-pant from a high-level understanding of both the tool and the Tetris application to a more detailed understanding of both. The tasks assigned are shown in Table 6.1.

Task T1 required that the participants map overall application behaviour to each thread. Tasks T2 and T3 involved gaining an understanding of static structure. Task T4 focused on understanding the behaviour of a single class and tasks T5 through T7 were chosen as typical program maintenance tasks. Tasks T8 and T9 were optional, due to their difficulty and time constraints of the study. Participants were asked to use the OSE tool to solve the assigned tasks but were also encouraged to browse the source code if they wished. Participants were also allowed to ask questions if they had difficulty using the tool, and were allowed to use any other features provided by Eclipse’s Java development tools. The

(42)

6.1 Study Design 32

Table 6.1. Assigned tasks Task Instructions

T1 There are three threads used by the program at runtime (numbered 1, 13, and 15). For each thread, describe your understanding of its respon-sibility.

T2 For thread 13, identify the packages that it depends on at runtime and document the number of classes that are called at runtime.

T3 Choose one class (from a list of given classes) and document the classes it depends on in the context of thread 1.

T4 For the class chosen in T3 describe its runtime responsibilities in the context of thread 1.

T5 Describe the user-initiated shutdown process of the program.

T6 List the methods that new mouse handler(s) would have to call in order to enhance the current keyboard controls with mouse buttons and the scroll wheel.

T7 The program currently chooses blocks randomly. Which code would need to be modified or replaced in order to change this process so that it is not random?

T8 The program is written using Java Swing and AWT. Describe the cou-pling of the program to Swing/AWT, and briefly describe what would need to be done to replace Swing with another GUI framework.

(43)

6.2 Data Analysis Approach 33

participants’ answers were verified by one of the study designers, based on information provided by the developer of the Tetris game. The success of each answer was rated from 0 to 2 where 0 was unsuccessful, 1 partially successful, and 2 completely successful.

The sessions were videotaped and the screens were captured for later coding and anal-ysis. The same Windows XP workstation with a dual core AMD Athlon 2.2 GHz CPU, 2 GB RAM, and a 1200 by 1600 resolution 21 inch monitor was used for each participant. Eclipse Europa (version 3.3) and the OSE tool were installed on this workstation.

6.1.2 Interview

Immediately following the lab experiment, participants were independently interviewed to help understand their experience while carrying out the tasks. A total of sixteen questions were asked to understand:

• The overall experience using the tool • Frustrations experienced and barriers to use

• Effectiveness of the tool in understanding both static structure and behavioural as-pects of the subject system

• Different strategies used on each task • Opinions on information overload • Opinions on tool performance

The full text of these questions can be found in Appendix A.

6.2 Data Analysis Approach

The OSE tool implemented the feature set described in Chapter 3 using a set of 35 tool operations, each operation supporting one or more features. Using video captured during the experiment, these operations were coded according to participant and task. The order and frequency of use of each operation were also recorded. Table 6.2 maps operations to

(44)

6.3 Survey 34

features; operations are listed down the first column and features across the top of the table. Note that some operations are cross-referenced to more than one feature (e.g. opening, closing, and expanding activations are considered to be both part of the Grouping feature and the Hiding/Expanding feature).

Some features could not be mapped to observable tool operations (e.g. animation). These features were explored using the survey described in the next section.

6.3 Survey

To help understand the usefulness of those features that could not be directly evaluated during the experiment, a questionnaire was sent to all participants following the experiment. The questionnaire asked participants to rate the usefulness of seven features on a scale of 1 to 5 where 1 was Useless, 2 - Not Very Useful, 3 - Somewhat Useful, 4 - Useful, and 5 - Very Useful. The questionnaire also asked each participant to provide a qualitative assessment of each feature. Features surveyed and related questions are described in Table 6.3.

6.4 User Study Findings

The following subsections describe the results from the lab experiment, post-study inter-view, and post-study survey. These findings are further explored in the discussion and summarized in Table 6.4.

(45)

6.4 User Study Findings 35

Table 6.2. Tool operations mapped to tool features

F ocusing Gr ouping Hiding/Expanding Lay out Multiple Link ed V iews Component Na vigation

Queries/Slicing Selection Zooming/Scr

olling W orking Outside the T ool Breadcrumb Navigation • • Focus on Activation •

Focus on Parent Activation •

Activation Close • •

Activation Expand Children • •

Activation Open • •

Class Close Activations • •

Class Expand Activations • •

Collapse All • •

Expand All • •

Loop Change Iteration •

Loop Hide • •

Loop Open All Iterations •

Package Close • •

Package Expand Activations • •

Package Open • •

Fit to Screen •

Clone Pane Close •

Clone Pane Open •

Clone Pane Resize •

Navigate to Code •

Navigate to Thread •

Outline View •

Package Pane Close •

Package Pane Open •

Package Pane Resize •

Search Navigation • Search • Activation Select • Class Select • Package Select • Scroll • Zoom In/Out • Zoom to Marquee •

(46)

Table 6.3. Survey questions arranged by tool features Feature Question

Layout The Sequence diagram was laid out according to invocation time for activations, size of labels, etc. How useful was the layout to you?

Multiple Linked Views

There were several views available to you: the sequence diagram; the overview; the search view; the properties view; the package hierarchy; the clone pane; and the source code. Did you find the linking useful?

Highlighting During your interaction with the Sequence Diagram, highlighting was used in several ways: elements were made bold when they were selected or moused over; matches to searches were high-lighted in yellow. Was this feature useful to you?

Hiding It was possible to filter some calls from the sequence viewer. How useful did you find this?

Visual At-tributes

The visual attributes of the various views were used to attempt to indicate how you could interact with them. For example, ac-tivation boxes that had sub-acac-tivations (children) were outlined in green. When you moused over elements that could be ex-panded/collapsed, a +/- symbol was displayed. Packages had dif-ferent colours and shapes that classes. Were these visual attributes useful to you?

Labels How useful was it to you that you could clearly see the textual labels for the elements in the viewer?

Animation While you interacted with the sequence viewer, the diagram ani-mated to react to your interactions. Was this useful to you?

(47)

6.4.1 Lab Experiment

Each participant captured three program traces by running and interacting with the Tetris application. These traces averaged more than 11,000 calls with a minimum call count of just over 3000 and a maximum of more than 23,000 calls. Trace depth averaged between 10 and 11 levels with a minimum of 5 levels and a maximum of 12 levels.

During the lab experiment, we captured participant interactions to determine how many times a tool feature was used to accomplish each of the assigned tasks (see Table 6.1 for a list of tasks). While the primary purpose of this experiment was to understand the process followed and features used by participants, we also evaluated participant success on the assigned tasks. Two participants completed all mandatory and optional tasks. Two partici-pants completed all the mandatory tasks and part of the first optional task. The remaining two participants did not finish the mandatory tasks and did not get started on the optional ones.

All participants were able to complete task T1 successfully. Disregarding those who were unable to complete a task due to time constraints, participants performed better for functionality-related tasks T4 to T7 than they did for structure-related tasks T2 and the first part of T3. The participant with the best success for task T8 worked outside the OSE tool more than inside it for this task.

Of the fifteen features described in Chapter 3, eight were measured by counting explicit user operations and ‘Working Outside the Tool’ was also measured. Several features were implicitly involved in many or all operations and are discussed in the post-study section below. One feature, annotation was not required by any of the tasks. Figure 6.1 shows the total count of features used by all tasks for all users. Both this figure and Figure 6.2 were determined from operation counts and are independent of participant success on a given task.

Figure 6.1 shows that the most frequently used feature was zooming/scrolling. Oper-ation counts showed that scrolling represented the vast majority of these operOper-ations, with only ten zooming operations executed and nearly 700 scrolling operations. The second

(48)

Figure 6.1. Overall feature use for all users

most used feature was multiple linked views. ‘Working Outside the Tool’ is related to this feature because most of the navigations to linked views were for the purposes of brows-ing source code. Groupbrows-ing and hidbrows-ing/expandbrows-ing operations, typically used to reduce the amount of information presented, showed significant use. The selection feature was the next most used and was typically used when considering a specific activation, class, or package and its related entities. Queries, component navigation, and focusing received significantly less use.

Figure 6.2 shows the relative use of features per task. Each feature bar is composed of several patterned regions - one for each of the tasks that made use of that feature. The size of the region indicates the percentage use made of a feature during that task. This figure reveals that participants working on the thread responsibility task (T1) made the most use of several features including grouping and hiding/expanding features. Focusing

(49)

was used fairly evenly across tasks, and the grouping and hiding/expanding features have nearly identical use (in the OSE tool implementation, one is almost always caused by the other).

Figure 6.2. Percentage feature use by task

6.4.2 Interview

After the experiment, we interviewed the participants individually to gain a better under-standing of their experiences with the tool. It should be noted that these responses may be biased because some of the study participants were known to the researchers. Five of six participants said that the OSE tool helped them understand the functionality of the soft-ware. One of these five noted that it was also necessary to view the source code. The sixth participant said that the tool “came close” in helping understand the functionality of the software. This participant also liked the tool’s presentation of the program as a “visual

(50)

stack trace”. Three participants found the tool effective for understanding the structure of the software but one of these three suggested that there were other visualization tools better suited for understanding structure. When asked to list the most useful tool features, five participants mentioned the package pane and the ability to expand and collapse packages into a single lifeline. One of these, however, said that this feature could be improved if it did not collapse sub-packages at the same time. Among the least useful features, three participants said that they did not use zooming, preferring the default zoom level.

6.4.3 Survey

For some tool features, it was difficult to determine how often they were used or how useful they were from observations alone. This is because some features are either pervasive, occurring as a result of most other actions (e.g. animation), or are implicit, occurring without conscious user intervention (e.g. saving of views). To address this, we conducted a follow-up survey to ask participants to evaluate and discuss the effectiveness of these features. Only one feature, filtering, was rated poorly (2 out of 5 mean score). All other features scored 4 out of 5. This might be because people did not feel comfortable criticizing the tool in front of its designer (a typical bias in this kind of study).

All participants found the search feature and selection highlighting feature useful, but requested more of a visual difference between expanded and collapsed sub-sequences. There were several comments in the survey related to filtering. While the default package filtering was found to be effective, some participants would have liked more advanced and intuitive filtering at different levels. A common request related to filtering was to support focusing on a class, filtering unrelated details.

Two participants complained about tool performance, the other four participants find-ing it good or adequate. While labels were found to be very helpful by all participants, more than one person suggested that activation labels should be placed closer to the target activation. One participant also found that method activations were, at times, too small to easily click on. Finally, most participants agreed that animation between view changes

(51)

6.5 User Study Discussion 41

helped maintain context. As with the interview, these responses may be biased because some study participants were known to the researchers.

6.5 User Study Discussion

In this section, we draw some conclusions with respect to research questions 2 and 3. After answering these questions, we discuss limitations of the study and conclude with some implications for tool designers. We begin with some general observations.

In general, tasks with lower success rates (e.g. T2 and T3 - refer to Table 6.1 for a list of study tasks) were those that required structural understanding (for which the sequence diagram is clearly not an ideal tool). While structure can be inferred from dynamically created sequence diagrams, these are unlikely to provide a complete picture and will also be more difficult to use than views that explicitly show structure. This was also confirmed by the interview where only three out of six participants thought the tool helped with structural understanding. Two participants did not complete the mandatory tasks. In one case this was most likely due to capturing an abnormally large execution trace and in the other case because the participant chose a complex class for tasks T3 and T4.

A visual comparison of the feature use between participants does not show a correlation between task success and the use of specific features. Perhaps this is because of the small number of participants and the diversity of their approaches to reverse engineering tasks. Also, ironically, heavy use of a feature (e.g. repeated scrolling) may not indicate that a tool feature is helpful in solving a problem, but may actually be a sign that the user is unable to find what he or she is looking for.

Task T1 (thread understanding) made the most use of several features but this may be in part due to it being a larger and more difficult task than many of the others (as well as being the first task). Task T5 (understanding the shut-down process) was also a difficult task and required comparatively more operations. Interestingly, task T3 (class dependencies) also required a lot of effort, possibly because it was a structural analysis task that was not well

Tool features for understanding large reverse engineered sequence diagrams