Reminding and refinding: examining how software developers use annotations

(1)

i

Reminding and Refinding:

Examining how Software Developers use Annotations

by

Jody Ryall

B.Sc., University of Victoria, 2005

A Thesis Submitted in Partial Fulfillment

of the Requirements for the Degree of

MASTER OF SCIENCE

in the Department of Computer Science

Jody Ryall, 2008

University of Victoria

(2)

Reminding and Refinding:

Examining how Software Developers use Annotations

by

Jody Ryall

B.Sc., University of Victoria, 2005

Supervisory Committee

Dr. Margaret-Anne D. Storey, (Department of Computer Science)

Supervisor

Dr. Hausi A. Müller, (Department of Computer Science)

Departmental Member

Dr. M. Yvonne Coady, (Department of Computer Science)

Dr. Raymond G. Siemens, (Department of English)

(3)

iii

Supervisory Committee

Dr. Margaret-Anne D. Storey, (Department of Computer Science)

Supervisor

Dr. Hausi A. Müller, (Department of Computer Science)

Dr. M. Yvonne Coady, (Department of Computer Science)

Dr. Raymond G. Siemens, (Department of English)

External Examiner

Abstract

Softw are d evelop m ent requ ir es u nd erstand ing and navigating com p lex softw are sp aces. Develop ers frequ ently u tilize annotation s in sou rce cod e to help them externalize inform ation they need to rem em ber, su ch as tasks and im p lem entation d etails. Althou gh som e tool su p p ort exists in m od ern integrated d evelop m ent environm ents for au thoring and navigating these annotations, w e have observed that they often fail to rem ind d evelop ers abou t tasks that need to be p erform ed and are som etim es d ifficu lt to find . We p resent the resu lts from fou r em p irical stu d ies d esigned to better u nd erstand how d evelop ers create and m anage their inform ation u sing annotations. We also exp lore the u se of hierarchical tagging cap abilities to enhance these annotations. Based on the find ings from these stu d ies, w e p rovid e suggestions on how annotation tools m ay be im p roved .

(4)

List of Tables

Table 1: Vocabulary issues related to tagging ... 16

Table 2: Relationship between study methods and research questions ... 31

Table 3: Do you use Eclipse bookmarks in your source code? ... 36

Table 4: Which of the following Eclipse task tags do you use? (Select all that apply) ... 37

Table 5: Do you add any additional details to your comments? If so, what details do you add? (Select all that apply.) ... 38

Table 6: When collaborating on a team has your team agreed to use the same keywords? ... 39

Table 7: Task annotation usage in Java files of selected projects ... 40

Table 8: Percentage of task annotations that are auto-generated ... 41

Table 9: Tag taxonomy (version 1) ... 48

Table 10: Classification of user created tags ... 49

Table 11: Tag reuse at the end of the study ... 49

Table 12: Tag taxonomy (version 2) ... 56

(8)

List of Figures

Figure 1: Eclipse Bookmarks View ... 20

Figure 2: Eclipse Tasks View ... 22

Figure 3: TagSEA 0.6.4 for Eclipse. ... 24

Figure 4: Waypoint (dark) and task annotation (light) use in VizTK, PIM and TagSEA ... 57

Figure 5: Frequency of tag reuse ... 58

Figure 6: VizTK tags visualized as a tag cloud ... 60

Figure 7: PIM tags visualized as a tag cloud ... 60

(9)

ix

Acknowledgements

N ot long ago this thesis seem ed far aw ay and u nlikely to be started , let alone finished . It is only w ith the su p p ort of m any friend s and colleagu es that I m ad e it this far.

To m y su p ervisory com m ittee: Peggy Storey for her creativity, gu id ance and friend ship ; Yvonne Coad y for her u nlim ited enthu siasm and encou ragem ent; and , H au si Mü ller for his u nfailing su p p ort.

I w ou ld like to thank the m em bers of the TagSEA team for taking p art in this research. Sp ecial thanks go to Del Myers for his w ork d evelop ing and m aintaining the TagSEA tool, and for his assistance analyzing the log d ata it p rod u ces. To Janice Singer and Peggy Storey for their tim e and exp ertise in cod ing the qu alitative d ata and m aking the p rocess seem p ainless. Thanks also to Li-Te Cheng for recru iting p articip ants in the ind u stry case stu d y and for help ing to gather their feed back. To Ian Bu ll for interview ing the Eclip se d evelop ers w hile on-site in Toronto. And , finally, thanks to Michael Mu ller for his inp u t on social tagging behav iou r. Ou r com bined efforts help ed to m ake this research stronger than it w ou ld have been ind ivid u ally.

I can also not forget, or tru ly thank, all the p articip ants w ho gave their tim e to p articip ate in this research. You r feed back and insights are m ost ap p reciated .

To m em bers of the Com p u ter H u m an Interaction & Softw are Engineering Lab (CH ISEL): thanks for m aking each d ay fu n and enjoyable. There w as alw ays lau ghter to share and w hen it cam e tim e to w ork, insightfu l com m ents and w ise ad vice. Sp ecial thanks to Chris Bennett and Trish d 'Entrem ont for their feed back on this thesis.

Finally, I w ou ld esp ecially like to thank m y friend s and fam ily, for their im m easu rable su p p ort and for p rovid ing balance to the som etim es all-consu m ing natu re of grad u ate school.

(10)

1

Introduction

“I long to accomplish a great and noble task, but it is my chief duty to accomplish small tasks as if they were great and noble.” - Helen Keller, ‘Optimism’ (1903)

OFTWARE d evelop ers m u st u nd erstand and navigate com p lex softw are sp aces to p erform their d aily tasks. Details for a single task are often scattered across m u ltip le p ackages, classes, and m ethod s. When a softw are d evelop er is w orking on a large p roject w ith m any tasks to p erform , rem em bering all the relevant d etails u sing hu m an m em ory alone is nearly im p ossible [48].

The p roblem of d ealing w ith inform ation overload arises, in p art, from m u ltitasking. With m u ltip le ongoing tasks, d evelop ers have to choose w hich task they w ill p erform next, and store the rest for later. Develop ers m u st be able to collect, m anage, categorize and store these tasks and their associated inform ation, and then recall them as need ed . In p articu lar, tw o p arts of this p rocess are challenging: categorizing and recalling inform ation [47][52]. To cop e w ith these d ifficu lties, p eop le have d eveloped variou s strategies to externalize their m em ory, su ch as creating task lists or em ail rem ind ers.

Annotations have been u sed as an aid to externalize m em ory for centu ries. For exam p le, annotations can be u sed to record tasks that still need to be p erform ed , m ark locations on a m ap to visit or revisit, and record thou ghts or im p lications abou t w hat has ju st been read . Softw are d evelop ers have also been know n to create annotations to record inform ation for fu tu re recovery.

Softw are d evelop ers have created p rocesses and tools for u sing annotations in their everyd ay w ork. The sim p lest p ractices available tod ay inclu d e creating bookm arks and leaving m em orable keyw ord s in the cod e, su ch as “todo”, “fix me” or the author‟s initials. Modern IDEs provide tools for

S

(11)

2

creating and m anaging annotations; how ever, these tools d o not alw ays ad equ ately su p p ort rem em bering and refind ing inform ation . For exam p le, bookm arks are hard to locate and keep synchronized . Sim ilarly, task annotations – specific keywords recognized and highlighted by the compiler – are often rigid ly grou p ed into a hand fu l of p red efined categories.

While w e have intu itions on how d evelop ers u se annota tions to cop e w ith m u ltitasking, little research has been d one in the acad em ic com m u nity . We p rop ose to exam ine how d evelop ers create and m anage their inform ation u sing softw are annotations based on the follow ing research qu estions:

Q1: What is the content and stru ctu re of the annotations being created ? Q2: H ow are the d evelop ers u sing these annotations?

Q3: Where d o they store this inform ation? Q4: Who are these annotations intend ed for? Q5: Why are these annotations beneficial?

By answ ering these qu estions w e hop e to lay the grou nd w ork for fu rther exam ination of annotation u se in softw are.

Ou r earliest find ings led u s to hy p othesize that there are tw o key w eaknesses to cu rrent tool su p p ort: (1) lack of navigational su p p ort for linking related cod e (i.e. cross-cu tting concerns), and (2) lack of m etad ata and stru ctu re for the m anagem ent of these annotations. Seeing these shortcom ings, ou r research grou p has d evelop ed a tool called TagSEA (Tags for Softw are Engineering Activities) that allow s u sers to em p loy u ser -d efined keyw ord s [27] and to stru ctu re them hierarchically. We aim to exp lore the p otential benefits of these techniqu es for im p roving cu rrent tool su p p ort.

Ou r research is gu id ed by a m ixed -m ethod s d esign [13]: collecting and com bining qu antitative and qu alitative d ata from varying sou rces. In this thesis w e p resent fou r em p irical stu d ies w ith p rofessional softw are d ev elop ers exam ining the situ ation in tw o d irections. First, w e consid er how softw are d evelop ers u se cu rrent annotation techniqu es. Second , w e exam ine how softw are d evelop ers u se TagSEA, focu sing on u ser-d efined vocabu lary and hierarch y u se.

(12)

Based on ou r find ings, w e d evelop a set of im p lications for softw are p rocess and tool d esigners. In the p rocess w e u ncover ad d itional qu estions and avenu es for fu tu re exp loration. We hop e that by exp loring this top ic w e m ay im p rove annotation tools, so that they m ay better su p p ort d evelop ers as they externalize their m em ory.

1.1 Thesis Outline

In Chap ter 2, w e exam ine issu es that su rround m anaging tasks and inform ation in a m u ltitasking environm ent. Externalizing m em ory u sing annotations is a w ay of cop ing w ith the sid e effects of m u ltitasking. In Chap ter 3, w e look at how annotations in three contexts can be u sed for externalizing m em ory. By d oing so, w e can d eterm ine w hat consid erations to take into accou nt w hen d esigning tools to su p p ort softw are d evelop ers. The cu rrent tool su p p ort available is d escribed in Chap ter 4, along w ith w hat researchers have learned abou t the annotation habits of softw are d evelop ers.

Ou r objective in this research is to better u nd erstand w hat role annotations p lay in the w ork p ractices of softw are d evelop ers. To accom p lish this goal, w e have d esigned fou r stu d ies follow ing a m ixed -m ethod s m ethod ology, w hich are ou tlined in Chap ter 5. There are tw o p hases to this research. First, in Chap ter 6, w e exam ine how softw are d evelop ers u se cu rrent annotation m echanism s in their w ork by cond u cting a qu estionnaire, stu d ying op en sou rce cod e, and cond u cting d eveloper interview s. Second , in Chap ter 7, w e exp lore how d evelop er s can u se tagging to m anage their annotations by stu d ying d evelop ers u sing the TagSEA tool. In Chap ter 8, w e synthesize and d iscu ss the find ings from the stu d ies in light of ou r research qu estions. Im p lications of these find ings for annotation tool d esigners are p resented in Chap ter 9, along w ith the lim itations and contribu tions of this w ork.

As a collaborative ap p roach w as taken in this research, m y ind ivid u al contribu tions are ou tlined in Ap p end ix A.

(13)

4

Chapter 2:

Multitasking & Memory

“What memory has in common with art is the knack for selection, the taste for detail…. More than anything, memory resembles a library in alphabetical disorder, and with no collected works by anyone.” - Joseph Brodsky, ‘In a Room and a Half’, Less Than One: Selected Essays (1986)

H EN w e have only a single task in m ind , w e can easily im m erse ou rselves in the d etails at hand and m ake p rogress on that task. H ow ever, w hat hap p ens w hen there is m ore than one task? Or m ore im p ortantly, m ore than one task that d em and s ou r attention? We m u st m ake d ecisions abou t how to m anage ou r tim e – which task to do first and how m uch time to spend on it. But, it is not quite that sim p le. We m u st also exp end consid erable m ental effort to sw itch tasks, rem em ber w hich tasks need to still be com p leted , and regain any context w e lost w hen w e retu rn to an incom p lete task. All of this p u ts a strain on hu m an attention and m em ory, w hich w hile cap able in m any instances, benefit s from external help .

As m u ltitasking is an essential p art of everyd ay w ork, p eop le have d evelop ed variou s cop ing strategies to m inim ize the m ental effort involved in m anaging m u ltip le tasks. These strategies inclu d e sim p le things, like m aking notes and lists, u sing a calend ar to keep track of d ead lines, or u sing p hysical item s to serve as rem ind ers. These strategies act to externalize ou r m em ory, allow ing u s greater ability to m anage and rem em ber tasks and their associated d etails.

In this thesis, w e exam ine how annotations can be u sed by softw are d evelop ers for m anaging d evelop m ent tasks and other related inform ation. Before looking at annotations, w e m u st exam ine the issu es su rrou nd ing m anaging tasks in a m u ltitasking environm ent. By d oing so, w e can d eterm ine

W

(14)

w hat consid erations to take into accou nt w hen d esigning tools to su p p ort softw are d evelop ers.

2.1 Multitasking Activities

Activities su rrou nd ing m u ltitasking can be d ivid ed into variou s facets. Based on the research literatu re, I have grou p ed these activities into fou r categories, each of w hich w ill be d iscu ssed below :

1. task sw itching

2. m anaging inform ation and tasks 3. task categorization

4. rem em bering and locating tasks

2.1.1 Switching between Tasks

With m u ltip le ongoing tasks, som e sw itching is inevitable. When sw itching occu rs, w e m u st store the inform ation abou t the postp oned task, as w ell as rem em ber to retu rn to it [15]. Unfortu nately, hu m an m em ory has lim its and can fail. Mem ory failu res can take tw o form s. We are p robably m ost fam iliar w ith

retrospective memory failures – inability to recall p reviou s inform ation; how ever,

there is another category as w ell, prospective memory failures. Prosp ective m em ory involves form ing an intention to p erform a task and then acting on that intention in the fu tu re [20][51]. Inability to rem em ber to p erform that task on tim e is consid ered a p rosp ective m em ory failu re. For instance, a p erson cou ld d ecid e to p hone their friend on their birthd ay. Forgetting w hich d ate the birthd ay falls on w ou ld be a retrosp ective m em ory failu re. Forgetting to call on that d ay w ou ld be consid ered a p rosp ective m em ory failu re. Interestingly, p rosp ective failu res occu r in everyd ay life as often, or m ore often than, retrosp ective m em ory failu res [46].

Based on the varied reasons that cau se interru p tions [3][11], and that p eop le tend to interru p t them selves nearly as often as they are interru p ted

(15)

6

[16][31], avoid ing d isru p tions and m em ory failu res entirely is not an op tion. Fortu nately, p rosp ective m em ory failu res can be p revented w hen ad equ ate

reminders are available. Sim ilarly, retrosp ective m em ory failu res can be averted if

w e have su p p ort for m anaging and refinding inform ation. Peop le have d evelop ed strategies to m anage m u ltip le tasks and avoid m em ory failu res, su ch as creating em ail rem ind ers, task lists, or calend ar entries [6]. These strategies p rovid e varying levels of su p p ort for rem ind ing and refind ing.

2.1.2 Managing Information and Tasks

Softw are d evelop ers and other know led ge w orkers m u st not only m anage their tasks, bu t also the inform ation associated w ith these tasks. In a broad sense, Barreau and N ard i [5] fou nd that know led ge w orkers have three typ es of inform ation: ep hem eral, w orking and archival.

Ep hem eral item s have a short shelf-life. Bernstein et al. [7] referred to this ep hem eral inform ation as information scraps – “short, self-contained p ersonal notes that fall ou tsid e of trad itional filing schem es”. Exam p les inclu d e: “to d o” lists, m em os and note p ad s. Users in Barreau and N ard i‟s stu d y p referred to keep this inform ation visible rather than filing it. This allow s it to be u sed for qu ick reference. As a sid e effect, how ever, these notes can p ile u p , creating clu tter.

Working item s involve inform ation related to cu rrent tasks. The shelf life is generally m easu red in w eeks or m onths. Dep end ing on p ersonal p reference, these item s are either filed or p iled [52][81]. Regard less of how it is stored , it is u su ally close at hand . Rep eated u se of this inform ation over its shelf life, u su ally m easu red in w eeks or m onths, m akes it easy to rem em ber w here it is located .

Archival item s generally consist of com p leted w ork. More often than not, these item s are filed and are u su ally stored for m onths or years.

(16)

An interesting exam p le to d em onstrate how p eop le m anage inform ation and tasks in an electronic environm ent involved exam ining how p eop le m anaged their em ail. As w ith p ap er d ocu m ents [52], there are d ifferent strategies for m anaging the em ail inbox. Whittaker and Sid ner [82] id entified three categories of em ail u sers: no filers, frequ ent filers and sp ring cleaners. N o filers u sed the inbox for all of their em ail, w hile frequ ent filers m a d e d aily p asses of the inbox to keep the nu m ber of item s d ow n. Sp ring cleaners p eriod ically cleaned -u p their inboxes, u su ally once every one to three m onths.

For all typ es of em ail u sers, the au thors fou nd that sp ecific typ es of m essages w ere not d ischarged im m ed iately. These m essages tend ed to be kep t to act as rem ind ers (e.g. to-d os and to-read s) or to p rovid e im p ortant inform ation abou t ongoing tasks (e.g. ongoing corresp ond ence). H ow ever, they noted that as the inbox becam e large, its valu e as a to-d o list fad ed and there w as a shift tow ard s u sing it for m ore active tasks. They also fou nd that p eop le d id not w ant to file em ails that w ere active – aim ing to keep the inform ation available and read ily accessible in the inbox for later refind ing. The reasons for not filing inclu d ed : not w anting to file m essages that w ou ld not be u sefu l later (esp ecially since u sefu lness of d ata changes over tim e), not w anting to forget w here it w as stored , and that it w as d ifficu lt to categorize the inform ation for filing.

2.1.3 Categorizing Activities

Categorizing is trad itionally thou ght of as an activity for archiving inform ation; how ever, it also can be u sed for ep hem eral or w orking inform ation to provid e m ore stru ctu re, p articu larly for large am ou nts of inform ation (e.g. com p lex task lists or bu g tracking system s in softw are).

Decid ing how to categorize item s is cognitively d ifficu lt [47][52]. Part of this d ifficu lty arises from the actu al act of classification. Peop le d o not classify inform ation as a librarian w ou ld , bu t rather in the context that the inform ation ap p ears [47]. Item s d o not alw ays neatly fit into one category. The other p art of

(17)

8

the challenge com es from creating a classification that p eop le can u se to retrieve the inform ation in the fu tu re.

Peop le d eal w ith th e challenge in d ifferent w ays. Ravasio et al. [65] rep orted on an ethnograp hic stu d y that looked at how u sers m anaged and located files on com p u ters. In p articu lar, they fou nd that p articip ants u sed d ifferent strategies for storing d ocu m ents. For self-created files there w as less of a challenge, w ith 12 of the 16 p articip ants saving the files d irectly to a location in the filesystem . H ow ever, for non -self-created d ocu m ents, behaviou rs d iffered . Five p articip ants saved the d ocu m ents to the d esktop , six saved it to a location that the u sers thou ght w as ap p roxim ately the right higher level fold er, tw o saved it to a p re-d efined tem p orary fold er, and three saved it d irectly to the fold er they thou ght w as ap p rop riate.

Malone [52] m entioned three w ays to assist w ith classifying item s: (1) allow ing m u ltip le classifications, so the u ser d oes not have to d ecid e on a single rigid classification system , (2) allow ing u sers to d efer classifications (i.e. creat ing p iles of inform ation), and (3) allow ing the com p u ter to au tom atically classify item s. While the first tw o op tions show p rom ise, on this last p oint, Whitta ker and Sid ner [82] p rovid ed a w arning. They observed that u sers d id not w ant au tom atic filing of em ail becau se they w anted to be aw are of w hat w as ar riving, and not ignorant of the existence of p otentially im p ortant inform ation.

2.1.4 Remembering Tasks

A classification system is d esigned to provid e stru ctu re for refind ing the inform ation that w e are storing. Whether w e store inform ation by d ate, keyw ord or som e other d im ension, w e m u st still rem em ber rou ghly w here w e p u t it. Retrieving this inform ation requ ires tw o p sychological p rocesses: recall d irected search, follow ed by recognition based scanning, also know n as brow sing [47]. While inform ation retrieval u su ally involves both p rocesses, one tend s to d om inate. When the inform ation sou ght is know n or the p roblem of find ing it is

(18)

w ell-stru ctu red , the overall strategy w e focu s on is d irected search (i.e. selecting, sp ecification, and end ing) [10][79]. Otherw ise the overall strategy em p loyed w ill be brow sing (i.e. scanning, learning, recognition, etc.) [79].

In the case of rem em bering tasks and inform ation that w e have stored , the issu e tend s to be one of refind ing inform ation. Refind ing is actu ally a d ifferent cognitive p rocess than find ing inform ation [10]. With refind ing, the u ser has alread y seen the inform ation at least once. As su ch, the u ser m ay be able to recall associations, cu es or asp ects of the inform ation that w ill aid retrieval. Regard less of w hat is rem em bered , refind ing still involves searching and brow sing [77].

Elsw eiler [22] su ggested that tools shou ld p rovid e cu es along variou s d im ensions to aid in refind ing. These d im ensions cou ld be visu al, sp atial, tem p oral, contextu al, etc. This requ ires provid ing facilities for storing and m anip u lating m etad ata associated w ith the tasks and inform ation being stored . Since p eop le only rem em ber p artial d etails abou t w hat they have stored [47], flexibility in this regard cou ld be beneficial.

2.2 Summary

In this chap ter w e have looked at som e of the issu es su rrou nd ing m u ltitasking and how there are variou s techniqu es for cop ing w ith the sid e effects of sw itching tasks. The research literatu re su ggests that to m inim ize the strain on hu m an m em ory, w e need to ensu re that classification is as sim p le as p ossible and that the techniqu es available allow for w ays to aid rem ind ing and refind ing of inform ation. With an id ea of the challenges that know led ge w orkers face, w e tu rn ou r attention now to how annotations m ight be u sed to extern alize m em ory and su p p ort rem ind ing and refind ing.

(19)

10

Chapter 3:

From Scribbles to Collaborative Annotations

“This making of notes, however, is by no means the making of mere memoranda… in fact, if you wish to forget anything on the spot, make a note that this thing is to be remembered.” - Edgar Allan Poe, ‘Marginalia’, Southern Literary Messenger (1849)

N N OTATION S are a m eans of su p p orting hu m an m em ory, w hether to rem em ber tasks or to record d etails abou t an item . Once an item is w ritten d ow n it can be largely forgotten, leaving sp ace for other cognitive p rocesses. The annotation, if w ell form ed , can help rem ind the au thor of the item and refind any p ertinent inform ation.

Whether as simple as a bookmark in a novel to keep one‟s place, as p ersonal as colou r cod ed notes taken in a textbook, or as brief as a scribbled note on a notep ad , annotations take m any form s. H ow ever, regard less of the form s they m ay take, annotations are com p osed of tw o item s: an anchor and content [9]. The anchor can be exp licit, as in a circled p iece of text, or im p licit, su ch as a scribble in the m argin of a book [9][55]. Content is optional for annotations; if p resent, content either has an exp licit or telegra p hic (i.e. p ersonal) m eaning [55].

To u nd erstand m ore on how p eop le can u se annotations in electronic environm ents, this chap ter su rveys: p ap er and d igital annotations, w eb based annotations, and social tagging.

3.1 Paper versus Digital Annotations

Marshall [55] looked at how annotations are changing as w e m ove from a p ap er -based society to a d igital society. In one stu d y the au thor looked at annotations in u sed textbooks p u rchased from a u niversity bo okstore. The textbooks w ere taken from across d iscip lines. Based on the annotations fou nd in these books, the au thor d evelop ed a list of reasons for m aking them . Annotations m ay serve as:

A

(20)

1. p roced u ral signals (acting as a reminder of som ething to read or re-read ), 2. p lacem arks (record ing im p ortant locations for refinding later),

3. a visible trace of a read er‟s attention , 4. an in situ w ay of w orking on p roblem s,

5. a record of interp retive activity (interpreting the m aterial), and 6. incid ental m arkings.

The first three categories w ere characterized by the u se of highlighting, u nd erlining or sym bols – u su ally sim p ly an anchor w ithou t content . The fou rth and fifth categories tend ed to involve w ord s, p hrases or equ ations. The final category involved other m arkings, like d ood les and m essages u nrelated to th e text. Dep end ing on the context, these annotations can act as rem ind ers (e.g. show ing w hat has been read ) or as a cu e to d raw attention for easy refind ing later (e.g. an asterisk in the m argin).

Marshall also noted the p ersonal natu re of m any of these annotations. Read ers w ou ld m ark u p the texts w ith variou s sym bols, d ifferent p en colou rs, and shorthand notes. This freed om to choose an annotation schem e that is m em orable and inform ative to the au thor u su ally rend ers the annotations m eaningless to others. Even to the creators of these annotations, their notes m ay not m ake sense over tim e, as they m ay forget their original intent. In a related stu d y, p articip ants of an acad em ic read ing grou p w ere asked to u se a d igital tool for m aking notes [57]. Particip ants w ere interview ed abou t the events of tw o w eekly gatherings. Even after only a w eek, the sp ecific intent of the annotations w as som etim es lost.

In a related p ap er, Marshall and Bru sh [56] cond u cted a stu d y w ith 11 grad u ate stu d ents taking a sem inar in H u m an -Com p u ter Interaction. The stu d ents w ere asked to read three p ap ers p er w eek and to p articip ate in an online d iscu ssion u sing a w eb annotation tool. Stu d ents had the op tion of annotating d irectly on the w eb or to annotate on p ap er and then p ost com m ents u sing the tool. Annotations u sing the tool cou ld be m ad e p rivate. All o f the stu d ents chose to read and annotate on p ap er and then transcribe their

(21)

12

com m ents on to the w eb. At the end of the stu d y p eriod the researchers collected the p rivate p ap er-based annotations and com p ared them to the p u blic w eb -based ones. The researchers fou nd significant d ifferences b etw een w hat p eop le w rote on p ap er and w hat they chose to share. Stud ents frequ ently exp and ed u p on the com m ents they had m ad e on p ap er . They also w rote com m ents w hen p reviou sly there had only been an anchor. While one w ou ld exp ect that p eop le m ight m od ify their annotation behaviou r w hen they know som eone else is read in g it, this show s that the norm al annotations that p eop le m ake for them selves tend to be less d escrip tive than those intend ed for sharing , and m ay be u nintelligible to others. The researchers su ggested that at best 7.8% of the p ersonal annotations translated into collaborative u se.

3.2 Bookmarks on the Internet

One of the m ost com m on w ays of annotating in electronic environm ents is by bookm arking. In the case of the Internet, the w ebp age is the anchor and the content is a string chosen by the u ser to id entify th e p age. By d efau lt, the title of the w ebp age is u su ally u sed . The p u rp ose of these typ es of bookm arks is to allow u sers to easily refind w ebsites that they think m ight be im p ortant for a cu rrent or fu tu re task.

Abram s et al. [1] stu d ied the bookm arking of w ebp ages. They cond u cted a su rvey w ith 322 resp ond ents, inqu iring abou t the nu m ber of bookm arks that p articip ants had accu m u lated . The au thors also collected bookm ark files from 56 ind ivid u als in 1996. While bookm ark u se has p r esu m ably changed since then, they p resent som e interesting find ings. The au thors claim that bookm arks can (1) red u ce cognitive load of m anaging URLs, (2) facilitate the retu rn to grou p s of related p ages, and (3) enable u sers to create a p ersonal inform atio n sp ace.

Perhap s m ore interesting, Abram s et al. observed fou r m ajor m etap hors for how p articip ants p erceived their bookm ark u se: id entification, collection, m ovem ent, and ep isod es. Peop le that relate to id entification w ou ld view their

(22)

bookm arking as labeling or m arking inform ation . The collection m etap hor su ggests that p eop le u se the bookm arks to retrieve inform ation from a vast inform ation sp ace w hile rem aining stationary. The m ovem ent m etap hor involves navigating or traveling to the inform ation, w ith the bookm arks acting as land m arks. Finally, the ep isod e m etap hor refers to a chronological list of sessions – essentially a history of the browsing activity. All of these metaphors are variations on refind ing, bu t u tilize d ifferent d im ensions for organizing and retrieving the inform ation: classification, task-based , sp atial, and tem p oral. They are all sim p ly d ifferent w ays of concep tu alizing the inform ation.

Abram s et al. also exam ined how p articip ants organized their bookm arks. They fou nd that 30% of resp ond ents w ere active filers, 44% engaged in sp orad ic filing, and 26% never organized their bookm arks, thereby keep ing them in chronological ord er. The sp orad ic filers w ere fou nd to engage in „sp ring cleaning‟ behaviour occasionally. Those that accumulated lots of bookmarks often created a hierarchy to help w ith organization. This hierarchy w as u su ally slow to d evelop and becam e rigid over tim e. Finally, in a tem p oral sense, they fou nd that the m ed ian tim e since a bookm ark w as last visited w as 100 d ays. This su ggests, as the au thors noted , that bookm arks are p rim arily an archival form of inform ation. Fu rtherm ore, this su ggests that they are ineffective as a tool for actively rem ind ing u sers or as a tool for refind ing p ertinent inform ation.

Jones et al. [41] observed u sers engaged in a w eb-intensive task to u nd erstand how p eop le kep t track of w ebsite locations. The task w as self-chosen as som ething the u sers w anted to d o w hen they had 30 m inu tes of free tim e. While all 11 p articip ants claim ed to u se bookm arks, only one actu ally d id d u ring the stu d y. Som e of the p articip ants claim ed their bookm arks need ed cleaning u p . As a resu lt, the au thors conclu d ed that bookm arks have a low rem ind ing valu e. They su ggest that they cou ld be im p roved by inclu d ing m etad ata that w ou ld allow u sers to recall how the w ebp age w as fou nd , w hy it is relevant, and w hat actions rem ain to be taken, if any.

(23)

14

3.3 Social Tagging

Social bookmarking originally w as based on the sharing of w eb bookm arks.

H ow ever, these system s m et w ith lim ited su ccess [58]. Recently there has been a resu rgence in sharing annotations in the form of tags. A tag is sim p ly a u ser-d efineser-d keyw orser-d that is m eaningfu l to the inser-d iviser-d u al [29]. System s have em erged to allow p eop le to tag a variety of item s, inclu d ing: p hotos, vid eos, books, p rod u ct d escrip tions and p eop le‟s p rofiles. When collected together these tags form a large rep ository of inform ation. From m y review of the literatu re, a few them es em erge that are relevant to this thesis: (1) m otivations for tagging, (2) benefits of free-form organization, and (3) the choice of vocabu lary.

3.3.1 Motivations for Tagging

As w ith bookm arks on the Internet, the m otivation for tagging is largely p ersonal. Peop le create a w eb bookm ark if they sense it has p otential for fu tu re u se [1]. While w eb bookm arks are restricted to w ebsite URLs, tagging can be ap p lied to alm ost anything. It is also relatively easy to ad d tags in m ost system s, red u cing the effort need ed to p articip ate [28]. Motivations can range from being solely for the ind ivid ual to being altru istic for the grou p [36].

Am es and N aam an [2] stu d ied tagging behaviou r for p hotograp hs. They fou nd that m ost p eople p articip ated for one or tw o p rim ary reasons, and that few p articip ants tagged solely to organize their ow n p hotos. The au thors d evelop ed a taxonom y based on tw o d im ensions: sociality (self or social), and fu nction (organization or com m u nication). For exam p le, a u ser m a y choose to tag p hotos for them self for the p u rp oses of refind ing the p hotos again later (i.e. self and organization). Or a u ser m ay choose to tag p hotos as a w ay of p rovid ing social signals to others (i.e. social and com m u nication).

(24)

3.3.2 Organization

Regard less of the intent, the collective gathering of these tags creates an inform al classification system . While p rofessionally created classification system s, s u ch as those fou nd in libraries, are characterized by high qu ality m etad ata and a p red efined stru ctu re and syntax, they d o not scale to large inform ation sp aces easily [29]. With collaborative tagging system s, p eop le are free to u se their ow n w ay of thinking and see w hat d evelop s. This classification system is u ser -d riven follow ing a bottom -u p ap p roach [67].

3.3.3 Vocabulary

Sen et al. [68] noted that allow ing u sers to invent p ersonally m eaningfu l tags can m ake tasks, su ch as organization and refind ing, easier for the ind ivid u al. H ow ever, these ind ivid u al inventions m ay m ake it m ore challenging for other u sers to locate item s. This situ ation w as d escribed by Fu rnas et al. [27] as the

vocabulary problem. Peop le tend to have d ifferent w ord s for the sam e things. For

instance, sod a or p op ; sofa, cou ch or chesterfield ; and bu g, issu e or w ork item . While the creation and tagging of item s has becom e easier in this typ e of u ser -d efine-d system , the m anagem ent an-d location of term s is less reliable. The m u ltitu d e of vocabu lary w eakens the classification system if p eop le m u st figu re ou t the term to search by [30]. Table 1 show s issu es com m only associated w ith the vocabu lary p roblem .

(25)

16

Table 1: Vocabulary issues related to tagging

Problem D escription Example

Plu rals The d ifference betw een singu lar and p lu ral form s of a w ord .

Cat and cats

Sp elling N u m erou s w ord s have sp elling variations. Theatre and theater H om onym s A w ord has m u ltip le u nrelated m eanings. Bow (front of a ship ,

tied ribbon, w eap on, etc.)

Polysem es A w ord that has m ore than one related m eaning.

Wood (p iece of a tree, and a forest)

Synonym s Mu ltip le w ord s that have the sam e m eaning.

Stu d ent and p u p il

H yp onym s A w ord that m ay be rep laced by another, bu t not vice-versa w ithou t changing the m eaning.

Tu lip and flow er

Meronym s A w ord that rep resents p art of another concep t.

Knee and leg

Social tagging researchers have noticed that, d esp ite these d ifferences in vocabu lary, u sage tend s to converge u p on com m on term inology [30][68]. This can be exp ed ited by the u se of recom m end er system s, w hich encou rage u sers to choose existing term s [68]. Another op tion is to u se thesau ru s system s to m atch sim ilar term s [80]. Desp ite the w ork in this area, this p roblem is often d escribed in the literatu re, bu t no solid solu tions have been fou nd .

(26)

3.4 Summary

All of the annotations w e have seen are easy to create and can be u sefu l for externalizing m em ory to su p p ort rem ind ing and refind ing . In the follow ing chap ter w e consid er tool su p p ort for creating annotations in softw are that su p p ort rem ind ing and refind ing, su ch as bookm arking and tagging in sou rce cod e.

(27)

18

Chapter 4:

Annotations in Software Development

“The medium is the message.” - Marshall McLuhan, Understanding Media (1964)

N TERN AL d ocu m entation [66] – com m ents stored d irectly in the sou rce cod e – is often seen as a vital p art of the softw are d evelop m ent p rocess [24]. This typ e of annotation serves as a w ay of com m u nicating d etails abou t the cod e to anyone exam ining it in the fu tu re, inclu d ing the author. Com m enting cod e is a fairly com m on p ractice althou gh it is less p rom inent in som e softw are p ractices, su ch as agile softw are d evelop m ent.

We su m m arize the state of the art of tool su p p ort for creating and m anaging u ser d efined annotations for rem ind ing and refind ing. Follow ing this, w e then consid er w hat the research com m u nity has learned abou t how these annotations fit w ithin the w ork p ractices of softw are d evelop ers.

4.1 Tools

Mod ern p rogram m ing langu ages and Integrated Develop m ent Environm ents (IDEs), su ch as Eclip se [19], typ ically offer variou s m echanism s to su p p ort rem ind ing and / or refind ing. In this section w e d escribe this set of tools and synthesize their u nd erlying d esign d ecisions to highlight gap s in rem ind ing and refind ing tool su p p ort.

4.1.1 Unstructured Source Code Comments

The sim p lest w ay for d evelop ers to annotate locations of interest in volves inserting com m ents into the sou rce cod e. These locations can be relocated easily by brow sing or searching. These com m ents can be u sed to assert inform ation

I

(28)

abou t the cod e, or to ind icate locations for fu tu re w ork. A fam iliar exam p le is the p revalence of inform al exp ressions and keyw ord s – su ch as “hack”, “fix m e”, or the developer‟s initials – to highlight suspect code. Such comm ents may be scattered throu ghou t the p rogram if the featu res being d ocu m ented cross -cu t the established softw are stru ctu re. A d raw back of refind ing d istribu ted in -line com m ents is that the d evelop er need s to rem em ber either the keyw ord s or the location in the cod e.

4.1.2 Programming Language Support for Navigation

To help p rovid e su p p ort for navigation via in -line p arsable com m ents, variou s p rogram m ing langu ages have sp ecial syntax. For exam p le, Javad oc documentation has the “@see” and “@link” tags, which are accompanied by notation referring to parts of cod e (e.g. p ackages, classes, and m ethod s) or URLs [74]. Mod ern Java IDEs au tom atically tu rn these tags into clickable hyp erlinks.

Java annotations can p rovid e sim ilar afford ances, bu t can also affect how p rogram s are com p iled and ru n [73]. A d evelop er can d efine a cu stom annotation (see Listing 1) and em bed it alm ost anyw here in the cod e, not only in a com m ent. These annotations can be u sed for variou s p u rp oses, su ch as to generate reports or to id entify the files to be checked -in to a sou rce cod e m anagem ent system .

@FeatureRequest(

id = 247835169,

message = "Enable teleporter", author = "Ethan",

date = "4/8/2008" )

public static void travelToDestination(Date destination) { ... }

(29)

20

4.1.3 Bookmarks

Bookm arks are a feature that allow s a d evelop er to m ark a location of interest in the cod e w ithou t storing the annotation d irectly in the cod e itself. Bookm arks in softw are are analogous to w eb bookm arks. For Eclip se Bookm arks (Figu re 1), the annotations are stored in the d evelop er‟s w orksp ace. To refind these bookm arks, there is a Bookm ark View . In ad d ition, Eclip se p rovid es visu al cu es of the bookm arked location in the m argin of the ed itor.

Figure 1: Eclipse Bookmarks View

The research d one by Mu rp hy et al. [61] su ggested that the Bookm arks View is rarely u sed in Eclip se. The au thors cond u cted a stu d y that logged the interaction of softw are d evelop ers as they u sed Eclip se in their everyd ay w ork. Using the Mylar Monitor, “a standalone framework that collects and reports on trace information about a user‟s activity in Eclipse”, they reported on the activities of d evelop ers w ho had significant u se of the Eclip se Java Develop m ent Toolkit (JDT). Throu gh an analysis of Mu rp hy et al.‟s d ata logs, w e saw that bookm ark selection events w ere as low as .02% of total view selection events, and that only fou r of the 42 p rogram m ers u sed bookm arks at all. We have also received anecd otal feed back from u sers of other IDEs, su ch as Visu al Stu d io, that bookm arking featu res are rarely u sed .

(30)

Ou r research [72] su ggests this low u sage is d u e to fou r reasons. First, althou gh UI su p p ort m akes it easy to bookm ark an elem ent in the cod e, bookm arks lack su fficient sem antic inform ation to facilitate later recovery. Second , bookm arks su ffer from a lack of visibility becau se they requ ire a sp ecialized view er to see them and therefore are easily forgotten, d ifficu lt to m anage, and th u s qu ickly becom e ou td ated . Third , bookm arks are typ ically stored in the user‟s workspace and cannot easily be shared across teams of d evelop ers. Finally, as the cod e evolves, bookm arks can becom e u nsynchronized and lose their relevance, as they are not an chored to the cod e. Sim ilar to Abram

et al.‟s resu lt for w eb bookm arks [1], w e find that bookm arks p rovid e w eak

su p p ort for rem ind ing and refind ing inform ation.

4.1.4 Task Annotations

Many IDEs allow d evelop ers to create annotations for record ing tasks by inserting sp ecific keyw ord s into cod e com m ents. Certain keyw ord s are p red efined by the p arser to ap p ear as task annotations. For exam ple, in Eclip se the predefined keywords are “TODO”, “FIXME” and “XXX”. These keywords m ay ap p ear in low ercase, althou gh they are trad itionally all in cap itals. FIXME is assigned a high p riority by d efau lt. Ad d itional keyw ord s m ay be d efined by the u ser in a p reference d ialog; how ever, this featu re ap p ears to be rarely u sed and m any d evelop ers are not aw are that the keyw ord s m ay be cu stom ized .

Sim ilar to bookm arks, navigation is p rovid ed via a sp ecialized view in the IDE that allow s for filtered view ing. Clicking on an annotation brings a p rogram m er to the location of that annotation. Since task annotation s are inserted d irectly w ithin cod e com m ents, they are im p licitly shareable and can be m anu ally u p d ated as the cod e evolves. Tool su p p ort for task annotations w as likely ad d ed in resp onse to the p revalence of d evelop ers ad d ing easily searchable keyw ord s in the cod e to facilitate refind ing (see Section 4.1.1) [84].

(31)

22

Figure 2: Eclipse Tasks View

To gain insight on how frequ ently this view is u sed in Eclip se (Figu re 2), w e d id a fu rther insp ection of Mu rp hy et al.‟s d ata [61], and saw that task selection events w ere as low as .2% of the total u ser view selections, w ith only 13 of the 42 p rogram m ers in that stu d y u sing them at all d u ring the stu d y p eriod . This low u se cou ld be d u e to lim itations of the view . The Task s View p rovid es lim ited su p p ort for sorting or filtering, w ith the latter only available throu gh a p reference d ialog. The lim ited task vocabu lary m eans that the annotations are grou p ed into a few broad categories. This lack of su fficient m etad ata m akes it challenging to find them and to d eterm ine w hich ones are im p ortant.

Since task annotations also ap p ear d irectly in the cod e, the Tasks View is only one w ay to access and m anip u late them . Ying et al. [84][85] cond u cted a stu d y of task annotations from one p roject, creating a classification of u ses. They fou nd that su ch com m ents w ere u sed by p rogram m ers to “talk” to each other as w ell as to su p p ort navigation. Thu s, it ap p ears that sou rce annotations d o su p p ort, at least at a ru d im entary level, both rem ind ing and refind ing.

4.1.5 TagSEA

TagSEA (Tags for Softw are Engineering Activities) [69][70][71] is a fram ew ork for tagging locations of interest w ithin the Eclip se IDE. Designed by m em bers of ou r research grou p as a set of p lu g-ins, TagSEA integrates id eas from social tagging w ith the aim of m aking it easier to find inform ation. There w ere tw o

(32)

m ain id eas: (1) p rovid ing m ore stru ctu re for organizing annotations by allow ing d evelop ers to d efine their ow n vocabu lary , and (2) allow ing d evelop ers to link cross-cu tting concerns in the softw are.

Tagging is not a new concep t to softw are engineering. Tags have been u sed for d ecad es for annotating check-in and branching events in softw are version control system s, as w ell as for d ocu m enting bu gs in issu e tracking systems. Brothers‟ ICICLE was an early exploration of a limited, controlled -vocabu lary of tag-like stru ctu res d u ring cod e insp ection [8]. More recently, Cod eSnip p ets [24] is an Eclip se p lu g-in that aim s to u se Web 2.0 tagging to help softw are d evelop ers retrieve cod e snip p ets from a repository. H ow ever, the research com m u nity has not yet ad equ ately exp lored tagging as a m echanism for categorizing and retrieving lightw eight annotations in sou rce cod e.

TagSEA has gone throu gh a series of iterations and changes d u ring its lifesp an. In this section w e d escribe the cu rrent version of TagSEA (0.6.4). As this thesis focu ses on the em p irical d ata collection and analysis su rround ing the u se of annotations in softw are and not an evalu ation of TagSEA, t echnical d etails related to the im p lem entation of the tool w ill not be d iscu ssed in this thesis.

TagSEA is based on a waypoint m etap hor. Wayp ointing com es from the navigation of rou tes in geograp hical sp aces [18][64]. Wayp oints are created by m arking a location of interest. They have associated m eta d ata, su ch as creation tim e and au thor. Wayp oints can be shared across u sers and ap p lications and m ay be gathered into rou tes.

(33)

24

Figure 3: TagSEA 0.6.4 for Eclipse. (A) a w aypoint containing tags; (B) a tabular list of w aypoints; (C) a tree of hierarchical tags; (D ) an input box for filtering

the tags that are visible in C.

In TagSEA, a waypoint is sim p ly a tagged location in the IDE. Ind ivid u al locations can be tagged w ith single or m u ltip le keyw ord s (Figu re 3A). In ad d ition to tags, other m etad ata can be au tom atically associated w ith each w ayp oint; in this version the m etad ata su p p orted inclu d es a m essage, the creation d ate and the au thor ‟s nam e. The w ayp oint, acts as an anchor, w hile the tags and m etad ata associated w ith the w ayp oint serve as the content.

Develop ers create w ayp oints by associating tags w ith p arts of the sou rce cod e u sing a Javad oc-style keyw ord . Sp ecifically, a w ayp oint is created by the d evelop er typ ing “@tag” in a com m ent block, follow ed by the tag keyw ord s. Descriptive free text messages can also be ad ded after a “:”.

Ind ivid u al tags are d elim ited by sp aces. A d evelop er can also create hierarchical tags u sing d ot-sep arated notation as follows: “@tag

(34)

bug.performance”. This indicates that there is a waypoint with the tag “bug” w hose su btyp e is “p erform ance”. The Javad oc-style syntax allow s for easier ad op tion of TagSEA by Java d evelop ers, w ho are alread y fam iliar w ith sim ilar conventions, such as “@author” and “@version”. The resulting waypoints are au tom atically associated w ith the closest Java elem ent (e.g. a m ethod ). Once w ayp oints are created , they can be u sed to navigate and u nd erstand the cod e , or to share information with others by upload ing “waypointed” cod e into a source cod e m anagem ent system .

TagSEA also has su p p ort for refactoring tags, allow ing them to be easily renam ed , reorganized or d eleted . Red u cing the nu m ber of u niqu e tags created can be ad d ressed by u sing a consistent set of tags over tim e [36]. TagSEA p rovid es an autom atic tag com p letion (content assist) featu re to su ggest existing tags based on a p artially typ ed tag.

Waypoints indicated by the “@tag” notation are highlighted in the editor‟s text, left-m argin, and scrollbar so that they stand ou t as clear land m arks as d evelop ers brow se throu gh the cod e (see Figu re 3A). Using the Wayp oints View er (see Figu re 3B), d evelop ers can search for and navigate to a p articu lar w ayp oint. Managing a grow ing sea of tags is a concern in large softw are system s. To ad d ress this concern, TagSEA provid es initial su p p ort for d ynam ic filtering of w ayp oints (see Figu re 3D). Every keystroke in the text box im m ed iately filters the list of tags and d isp lays those that partially m atch the entered qu ery, allow ing a u ser to cond ense an d exp lore tag sp aces. Users can also sort w ayp oints via m etad ata.

As an exam p le, a d evelop er cou ld view all locations tagged w ith a sp ecific bu g id or task, or all locations tagged by a particu lar d evelop er. Selecting one or m ore tags in the Tag Tree (see Figu re 3C) reveals the associated w ayp ointed Java elem ents in the m id d le p ane. Then, clicking on the entrie s in the Wayp oints p ane op ens the associated file ed itor, p ositions the ed itor at the ap p ropriate location, and highlights the w ayp ointed Java elem ent. Thu s, d evelop ers can qu ickly navigate to p laces of interest.

(35)

26

At first glance, TagSEA m ay look sim ilar to the task annotation tool su p p ort cu rrently available in m ost IDEs. H ow ever, it is d ifferent in several key w ays that w e feel are im p ortant. First, tags are created on the fly by the d evelop er in a bottom -u p fashion; they are not p red efined nor d o they need to be cu stom ized in a sp ecial d ialog before they are em bed d ed in the cod e. This is consistent w ith earlier research, in w hich Fu rnas et al. show ed that u sers p refer to u se their ow n vocabu laries for com m on objects and concep ts [27]. Second , the d evelop er can create hierarchical tags, thu s p rod u cing a lightw eight navigational taxonom y. This m ay also ad d ress p otential occlu sion p roblem s cau sed by task annotations. The tagging hierarchy can also be refactored , w hich allow s the developer to easily change tagged code, e.g., changing a tag from “bug14.fix-this” to “bug14.fixed”. Finally, the waypoints in w hich the tags appear have ad d itional m etad ata that fu rther facilitates refind ing.

4.2 The Use of Comments in Source Code

While som e strategies have been em p loyed to m ake sou rce cod e itself clear and u nd erstand able, su ch as literate p rogram m ing [44], softw are d evelop ers often rely on com m ents to com m u nicate d etails. Sim ilar to the annotations w e have alread y seen, com m ents in sou rce cod e have an anchor and a m essage. Van De Vanter [78] refers to these com p onents as content and p lacem ent. In general, the link betw een com m ents and the sou rce cod e that it relates to is not exp licit. The p roxim ity betw een these tw o elem ents u su ally form s an im p licit link. While cod ing conventions bring ad d ed consistency, in som e cases it can be d ifficu lt to d eterm ine w hat a com m ent refers to sp ecifically or w hen it stop s being relevant. Kaelbling [42] lam ented the lack of an exp licit link. H e called for scoped comments that w ou ld clearly show w hat each com m ent refers to.

A sid e effect of this im p licit link is that cod e can often change w ithou t the related com m ents being u p d ated , cau sing synchronization p roblem s. Particu larly w hen d evelop ers are w orking tow ard s a d ead line, the cod e w ill

(36)

often take priority over keep ing the com m ents u p -to-d ate [33]. As a resu lt, Grogono [33] su ggested that w ise p rogram m ers w ill ignore com m ents w hen m aintaining aged cod e. Even d u ring regu lar d evelop m ent, Ko et al. [45] fou nd w hile stu d ying stru ctu red ed itors that com m ent ed its accou nted for only 3% of the changes m ad e by d evelop ers. And , only arou nd 40% of these com m ent ed its involved creating annotations, w ith 60% involving d evelop ers tem p orarily com m enting ou t cod e.

In the softw are m ining com m u nity, Jiang and H assan [39] p rovid ed a p relim inary resu lt of a stu d y on the evolu tion of com m ents in PostgreSQL. They d iscovered that the num ber of com m ents in sou rce cod e rem ains constant after the initial stages of d evelop m ent have p assed . Flu ri et al. [23] looked at the ratio betw een com m ents and lines of cod e, based on three op en sou rce projects. While they m ake som e interesting assertions related to the ratio of com m ents to sou rce cod e and abou t d evelop ers m aking changes to the com m ents in the sam e revision as the cod e changes, they d o not exp lain the 77% of the com m ent changes that they claim w ere not d u e to cod e changes. Also, th ey chose to stu d y only three projects, of w hich all of them ap p ear to follow agile d esign p rocesses.

Marin [53] stu d ied w hat m otivates p rogram m ers to com m ent, by exam ining nine op en sou rce p rojects and extracting their cod e fro m the sou rce cod e m anagem ent system s. Three of his find ings are of interest:

1. the rate of com m enting d iffers not only betw een p rogram m ers, bu t also for the sam e p rogram m er w orking on d ifferent p rojects;

2. larger cod e m od ifications tend to be com m ented m ore th orou ghly; and , 3. p rogram m ers tend to com m ent m ore in cod e that is alread y thorou ghly

com m ented .

This su ggests that even am ongst d evelop ers w ho d o com m ent, how or w hat is com m ented d ep end s on the p roject, team , and ind ivid u al d evelop er.

Referring to the list of p u rp oses annotations can serve d evelop ed by Marshall [55], softw are com m ents can act as p roced u ral signals, p lacem arks, or to record interp retive activity. In general, com m u nication is the p rim ary reason

(37)

28

for creating com m ents, p articu larly for record ing assu m p tions that are not exp licitly exp ressed by the sou rce cod e itself. H ow ever, this com m u nication is not p erfect. Given that com m ents are w ritten in natu ral langu ages it is com m on for there to be som e m isu nd erstand ings cau sed by these com m u nications. H ow ever, as the softw are evolves, if the com m ents are not kep t in sync w ith the cod e, m iscom m u nications and m isu nd erstand ings can be exacerbated . Throu gh a p relim inary exam ination of the Mozilla p roject, Tan et al. [75] fou nd that these p roblem s resu lted in the introd u ction of new bu gs. Fu rther, in the FreeBSD p roject, the au thors fou nd at least 62 bu g rep orts abou t com m ents that are incorrect or confu sing.

As p reviou sly m entioned , Ying et al. [84][85] cond u cted a stu d y of the TODO task annotations in an IBM p roject looking at the d ifferent characteristics and u ses of these annotations. They d evelop ed a taxonom y to classify the t asks. Their taxonom y has seven categories: com m u nication, p ointer to a change requ est, bookm ark, cu rrent task, fu tu re task, location m arker, and concern tag. Som e of these categories w ere refined fu rther into su bcategories. Their w ork su ggests that d evelop ers com m u nicate to each other throu gh these annotations. Ou r w ork bu ild s on the w ork by Ying et al., as w e attem p t to classify the tags created by p articip ants in ou r stu d ies (see Chap ter 7). Our find ings exp and u p on this w ork and raise som e lim itations for this form of inqu iry.

To m y know led ge this is the only attem p t m ad e so far to analyze how d evelop ers are u sing these typ es of annotations in softw are. Ying et al.‟s taxonom y provid es u s w ith som e d irection for how these annotations m ight be u sed . H ow ever, they only looked at one project and d id not follow u p w ith the d evelop ers to see how accu rate their find ings w ere. So, the qu estion rem ains, how d o these annotations fit w ithin the softw are d evelop m ent p rocess?

In the rem aind er of this thesis I d iscu ss the em p irical stu d ies that w e cond u cted to investigate how d evelop ers u tilize som e of these tools for m anaging tasks and inform ation in their d aily w ork practices. In p articu lar, these stu d ies focu s on task annotations and the tags created u sing TagSEA.

(38)

29

Research Design

“The outcome of any serious research can only be to make two questions grow where one question grew before.” - Thorstein Veblen, University of California Chronicle (1908)

UR objective in this research is to better u nd ersta nd w hat role annotations p lay in the w ork p ractices of softw are d evelop ers. W e have d esigned a series of stu d ies u sing a range of d ata collection and analysis techniqu es [13][50]. In this chap ter, I reiterate ou r research qu estions and ou tline ou r ap p roach to answ ering these qu estions.

5.1 Research Questions

To u nd erstand how d evelop ers create and m a nage annotations, w e have set out to answ er the follow ing qu estions:

Q1: What is the content and stru ctu r e of the annotations being created ? Q2: H ow are the d evelop ers u sing these annotations?

Q3: Where d o they store this inform ation? Q4: Who are these annotations intend ed for? Q5: Why are these annotations beneficial?

These research qu estions have evolved th rou ghou t the variou s stages of this w ork. As a resu lt these qu estions are related , bu t d ifferent, to those that w e w ere attem p ting to answ er at the tim e of each of these stu d ies (cf. [69][72]).

(39)

30

5.2 Overall Approach

Ou r research has flu ctu ated betw een exp loratory and exp lanatory p hases [13], as w e have looked at the issu es u nd erlying how softw are d evelop ers annotate softw are. In general, w e have been gu id ed by a m ixed -m ethod s ap p roach [14][40][49]. Mixed -m ethod s research com bines the collection and analysis of qu antitative and qu alitative d ata [76].

Ou r p u rp ose for m ixing m ethod s has been for com p lem entarity [32].

Complementarity aim s to elaborate and enhance the resu lts from one m ethod

u sing resu lts from other m ethod s. In ou r research, the qu alitative d ata allow s u s to d elve into the intricacies of the collaborative softw are p rocess, the ind ivid u alities of the softw are d evelop ers, and the p ractical u se of the tools they u se. The qu antitative d ata allow s u s to take the observations w e have and valid ate to som e d egree how likely ou r fin d ings are rep resentative of the softw are d evelop m ent com m u nity. While there are w eaknesses, as w ith any research d esign, the com bination of these techniqu es help s to ad d ress the lim itations w e w ou ld find by u sing any single m ethod .

5.3 Studies

We cond u cted fou r em p irical stu d ies: tw o based arou nd d eveloper u se of task annotations, and tw o focu sed on u sing tagging for m anaging annotations. The resu lts and find ings from each of these stu d ies help ed to constru ct answ ers to m y research qu estions (see Table 2).

(40)

Table 2: Relationship betw een study methods and research questions

Q1 Q2 Q3 Q4 Q5

Task Annotation Research (Chapter 6)

Survey of Softw are D evelopers

Qu estionnaire X X

Eclipse Task Annotation Study

Cod e analysis X X X X X

Interview s X X X X X

TagSEA Tag Annotation Research (Chapter 7) Industry Case Study

Pre and p ost qu estionnaires X X X

Focu s grou p X X X

Com m ent change analysis X X X X

Longitudinal Case Study

Annotation analysis X X X X X

Interview s X X X X X

The stu d ies are ou tlined here and d iscu ssed in m ore d etail in Chap ters 6 and 7. I synthesize and d iscu ss the find ings as they relate to m y research qu estions in Chap ter 8. Lim itations of this research are d escribed in Chap ter 9. Given the overlap p ing natu re of this research, the stu d ies are p resented in

Reminding and refinding: examining how software developers use annotations

Reminding and Refinding:

Examining how Software Developers use Annotations

by

Jody Ryall

B.Sc., University of Victoria, 2005

A Thesis Submitted in Partial Fulfillment

of the Requirements for the Degree of

MASTER OF SCIENCE

in the Department of Computer Science

Jody Ryall, 2008

University of Victoria

Reminding and Refinding:

Examining how Software Developers use Annotations

by

Jody Ryall

B.Sc., University of Victoria, 2005

Supervisory Committee

Supervisory Committee

Abstract

Table of Contents

List of Tables

List of Figures

Acknowledgements

Introduction

S

1.1 Thesis Outline

Chapter 2:

Multitasking & Memory

W

2.1 Multitasking Activities

2.1.1 Switching between Tasks

2.1.2 Managing Information and Tasks

2.1.3 Categorizing Activities

2.1.4 Remembering Tasks

2.2 Summary

Chapter 3:

From Scribbles to Collaborative Annotations

3.1 Paper versus Digital Annotations

A

3.2 Bookmarks on the Internet

3.3 Social Tagging

3.3.1 Motivations for Tagging

3.3.2 Organization

3.3.3 Vocabulary

3.4 Summary

Chapter 4:

Annotations in Software Development

4.1 Tools

4.1.1 Unstructured Source Code Comments

I

4.1.2 Programming Language Support for Navigation

4.1.3 Bookmarks

4.1.4 Task Annotations

4.1.5 TagSEA

4.2 The Use of Comments in Source Code

Research Design

5.1 Research Questions

5.2 Overall Approach

5.3 Studies