Music-based quality control in software engineering

(1)

Bachelor Informatica

Universiteit van Amsterdam

Music-based quality control in

software engineering

Kas van ’t Veer

June 11, 2013

Supervisor(s): Raphael Poss

Inf

orma

tica

—

Universiteit

v

an

Ams

terd

am

(2)

Abstract

This research examines the use of music in giving feedback to programmers. Because there are tools available to automatically evaluate programming code, it is possible to automatically give feedback to programmers who are part of a large software engineering project. Because many programmers listen to music while writing code, it is possible to give feedback about the status of their project by changing their music. This research paper describes how different methods of changing music are evaluated, how a tool has been written to perform operations on a programmers music and how it can be determined if such a tool could indeed be useful in professional programming.

(3)

Introduction

When developing software with groups of people, there are lots of ways to give feedback to the programmers about the current situation and progress being made. The most classic way to do this is of course by looking at the code that a coworker has submitted, evaluating the code manually and inform the coworker by email if something is wrong. However, it is going to cost lots of time (and therefore money) if people are manually evaluating other peoples work all of the time. That is why some tools have been de-veloped to automate the evaluation process of code and show test results on a webpage or through sending automated chat and email messages to the programmers. However, when someone is working on a software project, we can assume he will probably not actively check his emails, chat messages or those webpages periodically as it would be distracting from the programming. So it might be very useful to look for alternatives to these methods of feedback that are more subtle, yet noticable. A recent alternative method of feedback would be build light indicators. These are big lights placed in an office which will light according to which stage the project is currently in (determined by the automated evalution software) so the employees can view the progress in the blink of an eye at any time. The disadvantage of this is that these lights would need to be placed in every working area of the programmers and that they have to be in everyones line of sight. Next to that, the programmers still need to actively look at these lights, which again would not be desirable because they may not want to have to think about looking at the lights all the time. However, lots of programmers seem to like to listen to some music on the background while programming. This background music could be another great way of giving feedback if we can alter this music in a subtle, but noticable way. Not only could this be a good non-invasive way of giving feedback, it also will not require any material to be placed, because any programmer that likes to listen to music on the background will have a pair of earbuds or headphones or a speaker system al-ready. When there is quiet background music being played in the working area through speakers, we can modify that music in the same way as long as the music is coming from a computer and without needing any additional physical equipment. This thesis will research if and why music-based quality control could be useful and in which way the music could best be changed to give feedback to programmers (chapter 3), what would be the best way to implement this through a simple software tool and to what extent it is possible to change the music (chapter 4), if such a tool can be functional and how it should respond under erroneous conditions (chapter 5) and how to validate if such a tool could indeed be useful for giving feedback to programmers in a real-life situation (chapter 6).

(6)

To provide this kind of musical feedback there will be quite a few challenges lying ahead. First of all we have to find a reliable way to pass data from the automated code eval-uation software to the tool we are going to develop. Secondly, we must find a reliable way to somehow connect the tool to the software that is playing music and influence the music. Finally, because different people use different music players on different operating systems, we also must find a way to make the tool modular enough to be used with a variety of different players.

The repository containing all of the code written during this research, including instal-lation instructions, can be found here: https://github.com/SBEXOBZO/MusicQC

(7)

CHAPTER 2

Related work

2.1 Build automation software

There are lots of tools available already for automating the evaluation of code and there are lots of differences between these tools. There are tools available that automatically attempt compilation of a complete software project according to a given set of options, which are called build automation tools. These tools have to be configured manually ac-cording to the project. Their configurations usually consist of a list of system commands and parameters required to actually initiate the compilation process, the structure of all the files in the project, how those files are linked together, locations of external re-sources or packages and more. These tools are often used to achieve so called continous integration. This term suggests that programmers integrate newly written code into the existing code continuously. Build automation tools help with this because they can assist in finding out if the code has been integrated successfully. There are lots of differences between these tools, some build automation tools can only be used with projects of a specific programming language and some tools do not require many system resources while other tools can be computationally heavy. Other differences are that some tools require manual activation and some of them can be linked to a remote repository that all of the programmers involved in the project submit their code to. When the tool is linked to a repository, each time new code is submitted the tool tries to build the code to see if it is able to compile. Many of these tools are also open-source projects that are still being updated by their communities and available for free. A short overview of a few well-known build automation tools is given below.

Make

Make is a Unix command that almost every computer science student will know. It basically runs the contents of a given makefile through the command line or teminal. This makefile contains a list of commands that can have variable parameters. Advantages to this are its built-in support on most Linux systems, its simplicity and the fact that it can compile any kind of project because it merely executes some commands. Makefiles are clear, readable and can therefore be used to automate the building a medium-sized framework of code in an easy and adaptable manner. Since make is merely a command, it requires manual activation, so it will not be automatically activated each time code is submitted to a repository.

(8)

Apache Maven and Apache Ant

Maven1and Ant2are two open-source build automation tools by Apache that are written in Java. Both tools use an XML file to describe a building process. While they are mostly used for Java projects, they can be used for many more languages as well. The XML files contain information about the structure of the files, the order in which files have to be compiled and locations of external resources. The required XML files however are sometimes regarded as hard and cumbersome to write and update by some programmers. Both tools do not have native support for automatic start of a build.

Buildbot

Buildbot3 is a piece of software that can be used to automate building and testing and is oriented at continous integration. It is written in Python and was first used as a more lightweight alternative to Mozilla Tinderbox. Nowadays it has grown so much in popularity that it is used by many developers including Google and Mozilla themselves. Buildbot performs a preconfigured build and (potentially) test cycle each time the source code of a project is changed in the repository which it is linked with. There are many supported subversion management systems such as Git and SVN. Another nice feature is that the Buildbot software is didived into one master and one or multiple slaves. The master is responsible for getting the slaves to download the new code and test it each time new code is submitted and it is responsible for collecting the results of all the slaves. The slaves themselves actually do the building and testing, while the master controls these slaves. One of the advantages of this is that you can configure lots of slaves with different operating systems or different versions of programming languages. That can be useful because you can test if the project runs stable on all of these different platforms or not.

(9)

2.2 Methods for status indication

Automated music modification according to the results of automated code evaluation tools has not been done before at the moment of writing or in knowledge of the writer. However, some tools have already been developed that make use of the output of au-tomated code evaluation in an alternative way. Two examples of this are build light indicators and lava lamp indicators. These tools have been described by programmers as both useful and fun.

Build Light Indicators

Build light indicators are lights placed in a working area that indicate the status of the latest build. These lights are placed at a location of a working area where all the programmers can see them. There can be multiple different lights, which can be colored or labeled to indicate what they mean. A common setup consists of a few labeled lights which indicate different levels of success. For example, there can be two lights, one which lights up when the latest build builded successfully and another one which lights up when the pre-defined unit tests have succeeded. So theoretically, there can be an infinite amount of different variations of setups of lights, depending on the project which is being worked on and the preferences of the programmers. The lights are not only useful because they give information to the programmer about the build, but they can also have a positive effect on the performance of the programmers, because the lights offer a clean overview of the status of the last build4, which allows teams to work more structured. When lights go off or turn red (depending on the setup), everyone immedi-ately knows there is something wrong with the latest build. Teams have reported they can track down the issue and get it resolved within a much shorter time by using these lights5. The only downside of these lights is that they have to be constructed6 and placed so that everyone working on the project can see them.

Figure 2.1: From left to right: A setup of build light indicators, lava lamp indicating succes, lava lamp indicating failure

(10)

Lava Lamp Build Indicators

A very interesting variation of build light indicators indicators can be made by using a red and a green lava lamp instead of regular lights7. These indicators work the same as build light indicators, but lava lamps have some interesting traits to them. Inside lava lamps there is translucent liquid mixed with other opaque liquid can move up and down in bulbs and emit a colored light when other light is project through it. The opaque liquid of the lava lamps starts moving when it gets hot. When the lava lamp is turned on, it will start to warm up and light will start to be projected through the liquid. Because the lava lamp only heats up and cools down slowly, it takes a significant amount of time before the colored opaque liquid starts moving after turning it on. It also takes time before the liquid in the lava lamps settles down again after turning it off. In this setup, when a build fails, the red lamp is turned on and when a build succeeds the green lamp is turned on. This means that if someone submits code that causes the build to fail, the red lamp starts projecting light through the liquid, but the red colored opaque liquid is still in the bottom of the red lamp. Since it takes time before this liquid is actually moving up and down througout the lamp, it will first only be a small red bulb in the bottom of the lamp. If the build gets fixed in time, the red liquid will never start to rise up and down, but if the failed build is not fixed in time the red liquid will start moving up and down. This can create a little game for programming teams, because they can try to have the red liquid moving as little as possible and having the green liquid move as often as possible. If the build fails and the red liquid in the other lamp starts glowing, the team has to try to fix the build as quickly as possible in order to prevent the red liquid from rising, which makes for a faster working team. Just like with build light indicators the only real disadvantage to this is that the lights have to be placed in a place where everyone who is working on the project can see them.

(11)

CHAPTER 3

Use cases

3.1 Advantages of using music

Build light indicators have proven to be a good method of informing the programmers about the status of the current build and can be motivational as well. However, many software engineering teams also actively work on projects at home, communicating with their team through the internet without being in physical proximity of each other. This also happens a lot within open-source communities which can be spread all over the world working on a single project together. In such scenarios, any type of build light indicator would not be wishful as people work separately and possibly even at variable locations. It would not be practical to have each programmer place build light indicators at each location at which they work. It would be interesting to look for an alternative which only makes use of resources already available. Since many programmers seem to like to listen to some music on the background while working, this might make for a very suitable alternative. The only piece of equipment required are speakers or a pair of earbuds or headphones, depending on whichever the individual prefers. We can assume that any programmer that likes to have some music playing while coding has access to this already, which completely cancels out the need of any additional physical equipment. Wherever the programmer may be while programming, he can have access to these materials. This even applies to situations in which the programmer is travelling, assuming that if a person can carry a laptop with them, they can also carry a pair of earbuds or headphones (or use built-in laptop speakers).

When a person is listening to music, the listener does not expect the music to make unexpected changes. When the music does make an unexpected change, the listener will notice this because he did not expect this event. We can use this to our advantage by changing the music unexpectedly to gain attention from the listener. Different methods of doing this are explained below.

3.2 Parameters in music

There are a lot of different ways to give feedback through music. For example, the volume can be modified, we can skip to another track, the music can be paused completely and so on. When a programmer is listening to music, there will always be an expectation of continuity. By changing the music we can break this expectation to gain attention from the user. In other words, when a programmer has to be notified there should be an unexpected change in the music so we can get his attention. We can create drastic

(12)

changes in music or we can make a more subtle change. Some users may prefer a distractive change, in order to have their attention grabbed, but some may not like to be startled. Drastic measures like stopping the music or increasing volume instantly could be too invasive and be regarded as annoying, but subtle measures may go by unnoticed. It is also worth noting that effects of temporary nature will be completely missed if the programmer mutes his music, takes off his headphones or earbuds or takes a break from work. We shall elaborate a bit on which parameters could be useful below.

Volume

Volume is a well-known and evident parameter and it could be used in several different ways. We can instantly increase or decrease volume or increase or decrease it in the span of a few seconds. A sudden change in volume can be used to draw the attention of the user quickly, but this can also be perceived as startling or annoying. If such an event is unwishful, the volume can be changed more subtly by incrasing it over time. An advantage of volume is that it is a permanent change, so when the programmer takes a break, he can still notice that something has changed when he comes back. A problem with volume is that some users could max out software volume and adjust their volume by hardware (such as a potentiometer to change the volume), in which case volume cannot be increased without risking audio clipping, which is an unwanted distortion effect that occurs when the sound pressure level exceeds the maximum sound pressure level for the resolution of the used digital audio file. A second problem is that even if the user leaves a margin for volume to be increased through software, the volume will be louder than the listener initially set it, depending on the initial volume and volume gain, this might cause hearing damage! Lastly, most listeners will probably have a personal preferable range of volume at which they like to listen. In order to make this effect noticable and reduce the risk of having the volume modification pass by unnoticed, the volume has to be changed by a significant amount. Changing the volume by a significant amount may leave the volume outside of the listeners range of preference.

Advantages Simple and recognizable parameter. Permanent effect; can indicate states. Disadvantages Volume modification might be either too drastic or too subtle. Not possible through software in some events. Risk of hearing damage.

Pausing

Another simple parameter is pausing the music. Pausing the music altogether is an invasive way of informing a user and also requires additional measures to get the music running again. When a programmer is writing code and his music suddenly stops, it may be annoying if the music suddenly stops. This however can also be regarded as a positive effect when the attention of the programmer is needed. Even though pausing is a permanent effect, the programmer might pause his music already when taking a break, which still makes the effect unnoticable when returning from a break. One instance in which pausing could be used to indicate a state is by completely pausing the music when a build breaks and only resuming playback as soon as the build gets fixed. If this is used for one indivudual, this requires the individual not to pause his music manually, which is a restriction that may not be wishful.

Advantages Simple and recognizable. Could be used to indicate a state

(13)

Switching tracks

The concept of switching tracks can be used in many different ways. One could just head to the previous or next track in the list, restart the current track or change to a completely different playlist, internet radio station or stream. These kind of options make switching tracks a very interesting parameter, because when using playlists or streams of different genres, the different types of music can be used for portraying different atmospheres and therefore different states of the project, which could be last build failed or last build successful. For instance, one could have a playlist with fast rock music and one with calm acoustic songs. Unlike using a sound effect (more on this later in this section), this method keeps the artists intent of the songs intact, it even utilizes it by categorising songs of similar type. Switching playlists can be done in a non-invasive gentle manner by fading out a song and fading in the next one of another playlist or by queueing the first song of a new playlist after the current one. This could also be used in the same way as the lava lamps (see chapter 2) to make everything into a game. This could be achieved by making a playlist with music that you do not like as much, which will be played when the last build has failed. This could motivate the user to fix broken code so the music that he likes will be played instead of music that he does not like. To stimulate faster addressing of errors even more, you could make a playlist with a likable song in the beginning and progressively less likable songs as the playlist goes on. In such case, if a build gets broken, the programmer would not be annoyed at first and may prevent future annoyance by fixing the build quickly so the unlikable songs will not come up. Probably the best feature of this parameter is that you can indicate states which your project is in by continously playing a specific playlist.

Advantages Can indicate states. Can use songs meaning or feeling to its advantage Disadvantages Not useful if listener wants to be in control of files being played

DSP effects

Digital Signal Processing effects are effects applied to a digital sound stream to alter its result, for instance by adding a slight echoing effect to the sound, changing the presence of certain ranges of frequencies or changing the pitch of the sound. Most effects have configurable settings which alter the intensity of the effect. Modifying these settings can make an effect more subtle or noticable. Foobar20008 is a media player which supports a few DSPs out of the box and even more with additional plugins9. There are many types of sound effects, but their advantages and disadvantages are mostly the same. Ef-fects among each other are pretty distinguishable, which gives a lot of different possible scenarios to identify. To not have the effect go by unnoticed, the effect has to be set to a pretty high intensity. Effects that are set to a high intensity give a distinct change to the sound, which is often not desirable over a long period of time. To still make use of the effect, it can be used for only several seconds. The problem with this however is that it will go by unnoticed if the programmer is taking a break and it also cannot indicate a state.

Advantages Can be made as subtle or as noticable as the user wants. Effects are easily distinguishable. User can be in control of files being played.

Disadvantages Subtle effects might go by unnoticed. Very noticable effects are un-wanted for long periods of time. Can not indicate state if used temporarily.

(14)

Examples of DSP effects

Frequency boosts Frequency boosts, EQ or equalizers work by increasing or decreasing the appearance of certain frequencies in music. This makes the music sound different, but it still sounds like the same song, so the listener will not be annoyed and the intent of the artist will stay unscathed (given that amount of boost or drain is kept within a reasonable margin). For instance, by making treble more appearant than bass, the music will start sounding a bit more tinny. If bass is made more appearant than treble, the music will start sounding more boomy or muddy. By using different options like these, we can create several different modifications to the sound to indicate different events. A highpass or lowpass filter is another DSP effect which works a little bit the same, except that these completely remove all sound above or below a certain frequency instead of only making them quiter.

Playback speed Another option of altering musical playback is by speeding tracks up or down. In this case, to avoid off-pitch music when speeding up, the pitch has to be lowered by the inverse amount with which the music is sped up. Still the acquired result may not always be desirable, since changing the speed will probably cause the music not to be represented as the artist intended it. For instance, speeding up a slow mellow track might result into an awkward sounding song and lose its meaning. Next to that, the listener might know a lot of the songs already and hearing them in another pace might probably only leave the listener annoyed. Like other effects it could be used for a short time as a temporary effect.

Playback pitch Another parameter to change would be the pitch of the music, however, like playback speed, this might only leave the listener annoyed. Again, using this effect will probably leave the music altered in a way not intended by the artist and kill the original feel or emotion associated with the song. Like other effects, it would still be useable as a noticable temporary effect.

Reverb Reverb adds an echo-like effect to the music, effectively simulating a room in which the walls reflect sound. Depending on the magnitude of the effect this becomes either noticable or not. This effect is actually used in the post-production process of any vocal recordings because of the desired sound changes. The user might not notice this effect and the event to indicate could go by unnoticed. With different settings, the effect may be very noticable. In that case it could be used as a noticable temporary effect. Stereo to mono Stereo to mono basically merges the two stereo channels into one channel, which changes the directional origin of the sound. This effect is slightly noticable while using speakers, but it is a lot more noticable when using earbuds or headphones. While using headphones the music sounds more as if it is inside your head instead of coming from the sound drivers. This is also a distinguishable change which could be used for a short time as a temporary effect.

Distortion Distortion would be an unwanted effect but again could also be used for a short time just to indicate an event and notify the user.

Phaser A phaser is an effect that phases frequencies in and out with a certain period of repetition, again a very noticable and drastic effect but can be used for a short period of time to indicate a certain event.

(15)

3.3 Target Audience

A software tool that alters music in any of the previously described ways might be of use for a programmer that likes to listen to background music and is involved in a software engineering project that makes use of build automation. These may seem quite restrictive conditions, but in reality many of the bigger software projects are performing continuous integration and therefore use build automation tools already. Next to that many active programmers in the circle of acquantances of the writer have informally reported that they like to listen to background music on the side. With two common conditions this should still leave for quite a significant group as a target audience. Like described in the introduction, these programmers should use the tool whenever they are working on the project. They can work on the project while listening to music like they always did. Except that this time their music will change according to the results of the build automation tools.

(16)

CHAPTER 4

Implementation

Now we have a general idea about which people we will be targeting and what the tool should be doing, it is time to make design and implementation choices. As part of this research is about finding to which extent it is possible to modify music, it would be the best to work towards an initial release that supports just one combination of platforms, but keep modularity in mind at all times while coding, so that the tool can later be modified to fit other platforms as well. The programming language used here is Python 2.710, this is chosen because Python code is readabe, fast and easy to write and because the writer had the most experience with it at the time of writing. Python might be computationally slow, but the software we are going to make is not computationally intensive anyway, so this disadvantage should not be worth worrying about. Note that the examples of code given below are a very simplified version of what is actually being used in the tool itself, this is done to keep the explaination from getting too long and tedious. The actual code can be found in the repository of this project11.

4.1 Customizability

As different users with different projects will have different preferences about which parameters in their music should be changed, we shall be making a configuration file that can be easily edited to affect the behaviour of the tool. With a configuration file the user can configure which actions should be performed on the music player on which events that the build automation software can produce. This is almost mandatory when using different playlists for different events, because each user likes different types of music that can be categorised into these playlists. Besides, to actually link the tool to a build automation server we need to have some sort of configuration anyway, because the user needs to enter the address of the server so the tool knows where to get its data from.

(17)

4.2 Build automation - Tool communication

Periodically polling webpages

The first problem is how to get the information about the latest build to our tool. Some build automation tools offer webpages which automatically update after each attempted build. This might be of use as we can use packages such as Beautifulsoup12 _{that offer} an easy programming interface to let us download a certain webpage and systematically wade through its contents. This can be used to periodically poll the webpage of the build automation system to check if there are any changes. If there is a new build, we can extract the needed data from the webpage and use this data for selecting the associated effect. The downside of this is that this introduces a lot of redundant network requests and therefore can unnecessarily increase the workload on the server if the project involves larger groups of people. The frequency of the requests can be lowered to reduce the amount of unnecessary requests and the amount of server workload. However, this introduces another problem, which is delay. If the clients only poll the server every x minutes, it can take up to x minutes before the clients actually get informed about a new build in the worst case, which is when a build just finishes after a client periodically polls the server for new build results.

Connecting to Buildbot through IRC

The good news is that Buildbot (see chapter 2) has built-in support to run an IRC bot on the master server13. IRC stands for Internet Relay Chat protocol, which is a simple protocol used for sending chat messages to each other through a chosen channel on an IRC chat server. This IRC bot however, emulates a user but is in fact an automated pro-gram that informs people in the channel about the latest status of the Buildbot server. This bot has a number of different functions. It can be configured to send chat messages to the channel it is connected with when a build starts or finishes and other people on the same channel can communicate with the bot to get information about the builds through several commands. Even though this bot is normally used in a chat channel with actual human users it should also be able to be used with custom made software. Instructions on how to enable the bot properly can be found in the readme.md file in the root directory of the repository of this project11.

To connect with a server we simply create a TCP socket and connect to the server to be used. A good free IRC server is irc.freenode.org, which is also used throughout the development and testing of the tool in this thesis. To connect to this server, we need the following code in Python:

mysocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) mysocket.connect(("irc.freenode.org",6667))

(18)

Now we have to identify ourselves to the server and specify which channel we want to join by sending a few messages containing that information to the server. The syntax of these messages is determined by the IRC protocol14. Because it can be a bit heavy to wade through the whole protocol just for making a simple connection, it was useful to derive the code from a clear example found on an internet forum15_:

nick = "botclient01" channel = "mybbchannel"

mysocket.send(’NICK ’ + nick + ’\r\n’)

mysocket.send(’USER ’ + nick + ’ ’ + nick + ’ ’ + nick + ’ :Python IRC\r\n’) mysocket.send(’JOIN #’ + channel + ’\r\n’)

After sending these messages and receiving confirmation from the server, we should be connected to the IRC server irc.freenode.org and the channel #mybbchannel. If any-one in the channel sends a message to the channel, we can receive it from now on. If the Buildbot IRC Bot is configured to send messages through this channel each time a build finishes, we can receive them and then extract the name of the builder that finished and the build result from that message to. This is the important step in which we actually receive the data we want from Buildbot. The advantage of this method over periodically polling a webpage is that there are no unnecessary requests anymore, when a build finishes, the Buildbot IRC bot will inform us with the associated information. To receive a messages from a socket, we use the following code:

data = mysocket.recv(4096).decode(’cp1252’)

This statement waits for any incoming messages or returns a message that was already saved in the buffer of the socket. The decode() method is used for decoding any unex-pected characters such as an ˜O, which could otherwise make the code crash.

We can also send messages ourselves by using the IRC protocol through Python. When connected to the IRC server it is very important to respond to ping messages from the server. The server will periodically send us these ping messages to check if we did not improperly disconnect. This means it is crucial to answer these pings with a so called pong to inform the server that we are did not lose connection. If this is not done, the server will kick the simulated user by the tool, which would cause us not to receive updates from Buildbot anymore. Answering to a ping is as simple as:

if data.find(’PING’) != -1:

(19)

Extracting the important data

Now we have a stable and lasting connection with the IRC server it is time to focus on the important part: receiving data from Buildbot. When Buildbot finishes a build, the IRC bot will send a message through the channel it is connected with. A complete IRC message like this received through a Python socket looks like this:

:kbb!~kbb@pc-vlab09.science.uva.nl PRIVMSG #kbbchannel :build #51 of runtests is complete: Success [build successful]

In this example kbb is the name of the Buildbot IRC bot, pc-vlab09.science.uva.nl is the host address of the message sender and #kbbchannel is the name of the channel the message was sent through. Everything after the colon is the actual message that was sent, in which runtests is the name of the builder that just finished and build successful is its result.

We can also ask the Buildbot IRC Bot what the latest build was. This is going to be very useful to do when a user starts up the tool, so he does not have to wait for the first new build for something to happen after he launches the tool. In the example below we ask about the latest build to our Buildbot IRC bot. The two variables represent the name of the channel and the nickname of the bot:

mysocket.send(’PRIVMSG #’ + channel + ’ :’ + botnick + ’: last\r\n’) Supposing that kbb is still the nickname of the Buildbot IRC bot and #kbbchannel the name of the channel, this code sends the message "kbb: last" to #kbbchannel. The bot will respond to this message with a new message:

:kbb!~kbb@pc-vlab09.science.uva.nl PRIVMSG #kazaabuildbot :last build [runtests]: 2m53s ago: build successful

The useful information in this message can be extracted by using a regular expression. A regular expression is a hardcoded string with some wildcards left out, which in our case will be the name of the builder and the name of the result, because that data is interesting to perform actions with. A regular expression for extracting this data from a message in which the Buildbot IRC bot tells us about the last result looks like this: regex = re.compile(’^:’ + botnick + ’_*!\S+ PRIVMSG #’ + channel + ’ :last build \[(.*?)\]: \S+[ seconds]* ago: (.*?)\r\n’)

There are several different syntaxes for the bot to tell us about a newly finished build or about the result of the last build that finished ealier. Because we cannot know before-hand which message contains what, we will check for all the different types of syntaxes for as long as we do not find one.

To check every single message if it is of importance we will be making an endless loop after connecting and keep calling the recv() method to receive all the data the server sends us. Every time we receive data, we need to check if the data is of any use to us by checking if the data is a ping or if it is useful information about a build by using these regular expressions. For the full receive-loop, refer to the repository11 _{of this project.}

(20)

4.3 Tool - Music Player communication

Now we have found a way to get the name of result of the latest build and the name of the builder to our tool written in Python, we have to influent the music that is playing on the users music player. Since Spotify seems to be a music player that is still becoming increasingly more popular than it already is, it might be a good choice to use this for the initial functional release of our tool. Several professional programmers at the University of Amsterdam have also informally reported that they like to use Spotify, because it has unofficially been released for Linux16. This means the target audience for the initial release will be programmers that use Linux with Spotify.

The problem now is how to control the Spotify application through our Python soft-ware. Luckily, many of the recent music players can offer extended functionality with community-made plugins. Since it would cost too much time for the timespan of this research to build a plugin like this ourselves, we will have to find an alternative that is already available. Luckily there is a plugin available for Spotify under Linux called SpotCommander17. This plugin is actually written to control Spotify with mobile de-vices or with a webpage through POST requests. Since it is also very easy to perform HTTP requests with POST variables with Python, this plugin can be of great use for this tool if we can find out the protocol behind it.

By using a plugin called Firebug18for the Firefox browser, it is possible to open a console which logs all requests. By simply visiting the webpage hosted by the SpotCommander plugin and activating Firebug, we can see all requests that are being performed and their associated variables. Because activating different playlists seemed to be a potential method (see chapter 3) we will try to find out which POST cariables we need to send to change the playlist first. Opening up the HTML console of Firebug while selecting a custom made playlist on the webpage of SpotCommander yielded the following infor-mation:

action: play_uri

data: spotif:user:kazaakas:playlist:6bYGgYKxkb0bl1V1lBINOY

See figure 4.1 on the next page for Firebug console output that was used for retrieving this data.

It is easy to use Python to send these POST variables to the same webpage by using the urllib and urllib2 packages:

url = http://localhost/spotcommander/main.php

post = {"action": "play_uri", "data": "spotify:album:5LMGAYhn2ywaxGZdtmXGpw"} urllib2.urlopen(url, data=urllib.urlencode(post))

Sending this request through Python this way proved to execute the same action as when a playlist was clicked in the browser menu of SpotCommander. The best part is that the second parameter holds the Spotify URI of the playlist that will be played. An URI is basically a code that links to a certain collection of tracks. This is very useful because within Spotify you can right click on any album, artist or playlist and copy the URI. This would allow the user to simply right click on his playlist of choice, copy the URI and paste it into the configuration file, which will be easy to use.

(21)

Figure 4.1: Firebug showing the POST variables used for initating playback of this playlist

(22)

4.4 User configuration

It would be preferable to make an interactive graphical user interface for the tool, how-ever, since different music players with different remote control plugins are going to need different commands, we would need to make a different user interface for each dif-ferent music player and potentially for difdif-ferent platforms, because some players have platform-dependent plugins. This requires more time than available for this research, so a simple configuration file was used which will have to be edited by the user. Since our target audience consists of experienced programmers, they will be proficient with editing configuration files anyway, which makes spending time on creating an interactive user interface even less valuable.

This configuration file consists of a JSON dictionary. JSON is basically a syntax for writing plain text that can be loaded into a data structure instantly. Using this will again save us the hard work of developing our own notation for the configuration file. The JSON dictionary holds the credentials, a few boolean options and a command dic-tionary. The tool filters out any newline, tab or carriage return characters when it loads the configuration, so it can be saved as a nicely structured and indented file without making it invalid JSON.

The following options in the configuration file determine some credentials: "nick": "botclient01",

"server": "irc.freenode.org", "port": 6667,

"channel": "mybbchannel", "botnick": "kbb",

These options respectively represent the nickname of the client bot, the address of the server, the port of that server, the name of the channel to join and the nickname of the Buildbot IRC bot.

The most interesting aspect of the configuration file is the commands dictionary. This is a dictionary within the main dictionary that holds several keys that represent data that Buildbot returns. This can be the name of a result, but it can also be the name of a builder followed by the name of a result, separated by a comma. Giving only the name of the result will make the tool execute the command associated with that key if it is equal to a found result. If a name of a builder is also given, the tool wil only perform the associated command if the given builder returned the given result. For both the name of a builder and result regexes are also supported instead of just strings. This was done because some projects might have a lot of different failure events and if you just want to play the same playlist with any kind of failure event you could enter .*[Ff]ail.*, which will match any result with the substring fail or Fail in it.

(23)

The values associated with the keys that represent Buildbot results is a data structure which could be called a command dictionary. A command dictionary is a dictionary describing the attributes of the command that will be executed if the result of the build matched the associated key. Command dictionaries can have several different valid forms. As previously described, the command can be an HTTP request with POST variables, but it can also be an HTTP request with GET variables or it can be a system call, which is also implemented because some music players can be controlled through calling their executable with certain parameters. This gives the tool a lot of flexibility to be used with different music players and different remote control plugins.

Because some music players might need multiple commands in a row to perform certain wished actions such as starting playback of a new playlist, multiple sequential commands are also supported. In this case the key with the build result should link to a list of command dictionaries.

A single command dictionary that starts playback of a specific Spotify playlist looks like this: ".*[Ss]ucces.*": { "type": "http", "url": "http://localhost/spotcommander/main.php", "POST": { "action": "play_uri", "data": "spotify:album:5LMGAYhn2ywaxGZdtmXGpw" } }

The three keys in this dictionary determine the type of the command, the address to perform the request on and the POST variables respectively. For more detailed informa-tion on how to perform other types of commands or a sequence of commands, refer to the readme.md file which is found in the root directory of the repository of this project11 Lastly, there are several other boolean options:

"autorename": 1, "autoreconnect": 1, "pollonstart": 1, "ignoreduplicates": 0,

If autorename is set, the user automatically renames to a random name if his name is in use. If autoreconnect is set, the tool will attempt to reconnect after losing connection. If pollonstart is set, the tool will ask about the last build as soon as it (re)connects to a server and execute the associated command. ignoreduplicates will allow the script to activate two actions in a row even if they are the same. More about the use and reason of addition of these options can be found in the next chapter.

(24)

CHAPTER 5

Functionality validation

Normal operation of the written tool should consist of initially connecting to a server and then connecting to the appropriate channel. When connected, every time a build finishes it should execute any commands that are associated with that result itself or that result from a certain builder, depending on the configuration. The tool that has been developed which can be found in the repository of this project11 satifies this behaviour. However, there are several erroneous and controversial scenarios thinkable which can occur while using the tool. This chapter will discuss these kind of scenarios in detail. The scenarios discussed here have either been found during development and testing or through forethought.

5.1 Equal results

When a new build finishes, chances are that its result is the same as the build before it. In this case it depends on the users preferences if we should perform the associated action with the build result or not. When the user uses temporary effects at each result, he will probably want to have the effect played anyway to inform him, but if a user uses playlists he will probably not want to start at the first track of the playlist again if the build result has not changed anyway. This is why the option ignoreduplicates has been built in. If this option is set, the command associated with the build result will still be performed even when it is the same as the last result.

5.2 Restarting the tool

If a programmer calls it a day and stops working on the project, he should close the tool so it does not influence his music anymore. However, if he starts the tool again the next day to continue working, there may have been some changes to the project. If the programmer uses a configuration which uses different playlists associated with different types of results of the last build, the last result has to be asked from the server, otherwise the programmer has to wait until a new build finishes for the tool to actually start representing results of the last build by playing his playlists. This is why there is an option called pollonstart. If this option is set the tool will ask for the last build when initially connecting to a server and execute the associated command with that result.

(25)

5.3 Server-client connection lost

When the connection between the tool and the IRC server is lost there will be no way to correctly modify the music anymore to the users configuration, because there is no possibility of receiving updates about recently finished builds from the Buildbot master server. That is why there has been an option built in called autoreconnect. If this option is set, the tool will automatically attempt to reconnect to the server when the connection is lost. Since the server can be down and we do not want to send unnecessary amounts of packets, the tool will wait for five minutes if the reconnection fails after three attempts, until it eventually connects again. Note that if the initial connection gives any kind of error it will also always try to connect two more times. It will retry after that and after waiting for five minutes if the option autoreconnect is set. If all of those three attempts fail, the tool will again wait for five minutes until trying again, until it hopefully eventually connects. Because this might still result in spamming the server too much if connection gets lost frequently, this option can be disabled by the user by setting the option to 0, should he wish to do so. When reconnecting to the server after the connection was lost a recently finished build can be missed. If the user reconnects there is a possibility that there has been a new build, which is why the tool will always ask for the last build when reconnecting after a disconnect, regardless of the pollonstart setting. This has been done because this is an erroneous scenario, for which we should try to compensate. Its not to say with certainity if there has been a new build with the same results or if there has not been a new build after having lost connection for a while, this should also be taken into account. So when the last build has not changed when reconnecting, no actions will be performed. Also, when the user did not set pollonstart and there is no last build known, we can never be certain that the last build actually has been changed, which is why no actions will also not be performed then.

Because the IRC protocol requires periodically answering of pings between clients and the server (usually in intervals of around 250 seconds), we can be certain that the connection is lost when nothing is heard for 300 seconds. When nothing is heard for 300 seconds, the tool will also attempt to reconnect to the server if autoreconnect is set.

When something is going wrong with networking the Python socket might return empty strings. This is why any empty string returned by the recv() method will be counted as if nothing was sent, therefore increasing the time that nothing has been heard from the server, until the connection eventually times out after 300 seconds if something was actually wrong, in which case the tool will reconnect if the user has configured it to do so.

5.4 Server-Buildbot connection lost

When the connection between the Buildbot IRC Bot and the IRC server is lost, nobody will be able to receive any updates about builds and therefore nobody will be able to have their music modified accordingly. Luckily, the Buildbot IRC Bot automatically reconnects to the configured IRC server. This introduces another problem however, if the connection was not cleanly closed, the name of the IRC Bot will still be in use when the bot attempts to reconnect. Because of this, the bot will take on an alternative nickname, which is its configured nickname followed by an underscore character. The problem with this is that if any newly connecting users join the server and try to send the last command to the Buildbot IRC Bot, the bot will not receive this command because it has a different name than in their configuration file. The same counts for

(26)

identifying messages from the bot about recently finished builds. This last problem can be easily corrected by adding an optional underscore after the name of the bot in the regular expression that is being used. When asking for the last build however, there is no such method. This is why the tool does not only send the last command to the nickname of the Buildbot IRC bot, but also to the alternative nickname of the bot. In this case, if the bot is connected to the channel, it will always answer with information about its latest build, regardless of the fact if it is using its alternative name or not.

5.5 Name in use

The emulated user of the tool can also be disconnected improperly, which will cause his old nickname to linger around for a few more minutes and prevent further connection with the same name. That is why there has been an option built in called autorename. If this option is set, the tool will automatically reconnect to the IRC server with a randomly generated alphanumerical name that starts with the capital letter T (T for temporary). Since users only need to communicate with the Buildbot IRC bot and not with each other, they do not need to be able to identify each other, which is why we can seamlessly use this technique without any consequences. Because the name of nickname that the tool uses to connect to the server does not matter, the nickname field also has the functionality to be set to an empty string, in which case the tool will connect with a randomly generated name every time it connects.

5.6 Errors in the config

When the configuration misses certain options or when it does not load as valid JSON the user must have broken the configuration file. That is why the application will refuse to start when this happens because the user should have an understanding of what he is changing in the configuration, an understanding which he will not have if he created errors in the file.

5.7 Unexpected Errors

Even with extensive forethought, unexpected errors can always occur. This can be because a regular expression fails for any reason, when we receive a character that does not fit the used encoding or any other still unknown mistakes. Because we want the tool to be as reliable as possible we do not want the tool to quit unexpectedly. This is why there are try and except clauses around all vulnerable pieces of code. When something goes wrong in an unexpected way we would want to log the error to the console and continue regular operation afterwards. So when an unexpected error occurs during decoding, analyzing or executing, the current message or action will be discarded and the tool will restart the receive loop again to start receiving and analyzing following messages. Its not forgivable to discard an action or message this way, but its better than crashing.

5.8 Default configuration

Not everyone likes to wade through all the options of a tool when they just start to use it for the first time, because they might just want to see what the tool is like and what

(27)

it does. This is why a default configuration should be made for each supported plat-form. Again, because playlists have been thought to be the most interesting and usable parameter (chapter 3) this would be a good option for an initial configuration. Because playlists indicate states, we will set all of the boolean options except ignoreduplicates to make them work as well as possible as explained earlier in this chapter. The name is left blank because the user only needs a name because the IRC Protocol requires this, so a random name will suffice. The only thing that needs to be configured is the Build-bot IRC Build-bot its name and the server address. As for default playlist associations with build results in case of Spotify, the album Eliminator by ZZ Top and the album Perfect Strangers by Deep Purple are chosen, for no particular reason other than being personal favorites of the writer. This can be done because the user only needs to be given a rough idea about what is happening when he is using a default configuration, which two differ-ent playlists will manage to do. Afterwards he can configure his own playlists and use those. The two default playlists are linked to the regular expressions .*[Ss]ucces.* and .*[Ff]ail.* respectively, which will match any general status about the build either failing or succeeding.

5.9 Accidental muting

When programmers are using the Buildbot IRC Bot for the first time through manual interaction they might want to try out some commands to see what they do. The problem with this is that the bot also has a mute command which will completely sabotage the tool when executed, because the bot will keep completely silent after using this, so no more updates about newly finished builds will be given. As this can be a serious issue, the best way to fix this would be to completely disable the mute command of the bot, however, this is not supported through configuration files. Luckily, Buildbot is also written in Python, so the actual source code of Buildbot can be modified. To disable the mute command, head to the

/buildbot/master/buildbot/status/words.py

file in your buildbot source, which is the file that contains code for running the Buildbot IRC Bot. Now replace the body of the function called command MUTE with merely a return statement. Afterwards, if its there, remove the file called words.pyc in the same directory. This file is a pre-compiled version of the previous words.py file that we have just altered. Because we do not want to use this version of the file anymore, it should be removed. Python will automatically compile new the file again if needed. Following these steps will still leave mute as a usable command, but it will do nothing, so it will no longer be able to sabotage the tool. If the Buildbot IRC Bot is configured correctly the command notify off should not be able to create a situation in which there will be no analyzable messages sent after finishing builds, so this command is of no threat.

(28)

CHAPTER 6

Appliance evaluation

Even though acquaintances of the writer that are actively involved in programming informally reported that this tool may be fun and useful, scientific research on the real life appliance of the tool on the target audience still has to be done. Because of the limited timeframe available for studying this subject, no such research has been done yet. Since speculation about methods of measurement is one of the first steps that would have to be taken to continue this research in the future we will elaborate on this.

6.1 Qualitative Research

To qualitatively research the degree of appliance of this tool will need several people from a few different software engineering projects. Conditions for test subjects are as follows:

• The person is engaged in an active software engineering project that makes use of Buildbot.

• The person likes to listen to music on the background while programming. • The person is willing to set up the tool and provide feedback after a short period

of usage.

Depending on the music player and platform the person uses, it may be required to add support for additional music players. More diversity between test subjects will give a wider overall view after feedback is received. However, when time is a concern, it should be taken into consideration if the extra time investing in adding support for their music player is worth their feedback. If multiple potential test subjects are using the same music tool that is not supported, it will be more worthwhile to add support for their music player. If only a very limited amount of test subjects can be found it will also be more worthwhile to start supporting a music player for only one test eubject because the fewer people that are tested, the more difference one extra test subject makes. The only things that need to be changed when adding support for a new music player are the commands in the configuration file. It is required however to write an additional readme file for each different supported music player since the user can not know what plugin has been used to make the configuration file work with his player and how to configure the configuration file accordingly. A readme file should include concise steps on how to install the required plugin and which paremeters of the commands to change to alter the

(29)

chapter 3) it is recommended to make the default configuration use switching between playlists and explain the user how to add his own playlists in the readme file.

After offering and preferably personally explaining the tool to the test subjects it should be used during active programming for at least a couple of days. Afterwards a ques-tionnaire should be given to the test subjects so they can give their feedback about the actual usefulness of the tool. Possible questions on this questionnaire could be:

• Did the tool help you stay informed about the current status of the project? – If so, what functionality did you use for achieving this and how? – If not, do you have any suggestions to improve this?

• Was the tool fun to use or did it make your time programming more pleasant in any way?

– If the tool was fun to use, which features played the bigged role in this? – If the tool was not fun to use, what did you dislike about using the tool? • Did the tool help you work more effectively?

– If you worked more effectively, which features played the biggest role in this? – If you worked less effectively, what was detrimental to your working

perfor-mance?

• Was the tool easy to configure?

– If not, what would you like to see changed in a future release? • Would you recommend this tool to other software engineers?

– If not, is there anything about the tool that can be improved to change your opinion on this?

• Do you have any further suggestions or remarks?

After receiving feedback from several different people from a few different projects, the individual results will have to be interpreted and a general conclusion can be drawn by comparing the answers with each other. If the tool turns out to be not recommendable, but the test subjects have a general opinion about what can be improved, a second research can be done after revising the tool by using the feedback. If the same people are used for this second research, they can be asked about wether or not they think the tool has been improved or not and they think it is more useful and more recommendable after revision.

(30)

6.2 Quantitative Research

A quantitative research would preferably use a larger group of people. Beforehand an unanimous poll should be taken among all of the test subjects about which music players on which platforms they are willing to use while testing the tool. Through this poll, the few most popular combinations of music players and platforms that are either already supported or easy to support should be chosen to use with the research. When using quantitative research it will probably not be possible to assist everyone personally by using the tool, so a lot of effort should be put into making the readme file extra clear and understandable, possibly through example screenshots. When all of these preparations are made, all of the test subjects that said to be willing to use the combinations of music players and platforms should be contacted with the information they need to use the tool. After at least a couple of days of time, the test subjects should be contacted again with a questionnaire concerning their opinion about the tool. However, with quantita-tive research it may be useful to let the test subjects rate certain aspects of the tool on a scale of 1-5 or 1-10. This makes it easier to draw conclusions from a larger amount of data, because for each question a mean can be calculated. A possible questionnaire for this could be:

Strongly Mildly Mildly Strongly Disagree Disagree Neutral Agree Agree The tool helped to inform me about

the current status of the project It was fun to use the tool

The tool helped me to work more effectively

The tool was easy to configure I would recommend other software engineers to use this tool

Do you have any further remarks?

With a large amount of results a general mean can be calculated per question and a mean per question can be calculated for each supported combination of music players and platforms. This way a difference may be seen between users of different music players and platforms. If the specific question has a significantly different rating than the general mean that same question, this may indicate that a certain supported music player has either poor or exceptional explaination or readability, depending on which question differs and how. Additional remarks made by the testers of that specific music player and platform can be used to further investigate the possible issue.

The advantage of this method is that differences between the quality of the support of music players can be clearly seen. Also, because means and larger groups are used, this method is less susceptible to producing skewed results due to unusual outliers. The disadvantage is that qualitative research might give more of an in-depth view about each issue because of using more complex questions.

(31)

CHAPTER 7

Conclusions

Through the course of this research, music-based quality control looks promising and useful in theory. Not only does it seem to be possible that through modification of music we can use a passive thought process (the listening to music) to actively alert the programmer, but the currently playing music can also be consulted at any time as a source of information about the health of the latest build when preconfigured playlists are used. There are also several different useful parameters in the music that can be changed which will have different effects on the user, namely permanent effects that can indicate states and temporary effects that can indicate an event happening at the time. By using the Buildbot IRC Bot and music player plugins together with Python, it has been shown that Python is a sufficient programming language to implement this type of functionality in a structured way without the need of any additional packages. Community made plugins for all kinds of music players have been shown to be of great help when needing to control the music player through custom written automated software. Furthermore, it should be easy to validate the usefulness of the written software through a group of test subjects which are given the tool, the required explainations and a questionnaire afterwards through which they can provice feedback about different elements of the tool and its functionality.

7.1 Future continuation of this research

To continue on this research in the future, there are still lots of areas which can be researched further. Later in this research support was added for Foobar20008 through the foo httpcontrol plugin19. This was initially done to use with the many sound effects associated with Foobar20009_{. However, this plugin did not support enabling of these} effects and no other plugins that did could be found. It should be possible to add this functionality to existing plugins through the Foobar2000 SDK to use all of the different effects with the tool.

An interactive user interface could also be built for easier modification of the tool. Even better would be to create a standalone plugin for music players with the same functionality as the tool in order to completely replace it. Because this thesis was about discovering the possibilities of custom software for this concept, this has not been done, a more modular approach was chosen instead.

(32)

Another possible point of improvement could be made by using Buildbots web pages. These pages contain additional information about finished builds such as where it went wrong. This information could also be used to filter on more specific events.

Lastly it would be very useful to perform evaluation research among the target audience for this tool to evaluate if the tool is useful after all. This could also lead to new insights on how to revise the tool even further.

(33)

Bibliography

[1] Apache Maven Project. http://maven.apache.org/. [2] The Apache Ant Project.

http://ant.apache.org/. [3] Buildbot.

http://trac.buildbot.net/.

[4] Automated Continuous Integration and the Betabrite LED Sign.

http://www.codinghorror.com/blog/2005/03/automated-continuous-integration-and-the-betabrite-led-sign.html.

[5] Lights! Continuous Integration! Action!

http://www.rallydev.com/community/engineering/lights-continuous-integration-action.

[6] Build Indicator Lights - The Pictures.

http://www.longsteve.com/fixmybugs/?p=402. [7] Bubble, Bubble, Build’s In Trouble.

http://www.pragmaticautomation.com/cgi-bin/pragauto.cgi/Monitor/ Devices/BubbleBubbleBuildsInTrouble.rdoc.

[8] Foobar2000.

http://www.foobar2000.org/.

[9] A collection of special effects (Foobar2000 plugin).

http://www.foobar2000.org/components/view/foo_dsp_effect/. [10] Python Programming Language - Official Website.

http://www.python.org/. [11] SBEXOBZO/MusiqQC - Github.

https://github.com/SBEXOBZO/MusicQC. [12] Beautiful Soup.

http://www.crummy.com/software/BeautifulSoup/. [13] Buildbot IRC Bot.

(34)

[14] Internet Relay Chat Protocol.

http://www.irchelp.org/irchelp/rfc/rfc.html. [15] IRC bot example on the Ubuntu Forums.

http://ubuntuforums.org/showthread.php?t=1493702&p=9363159# post9363159.

[16] Spotify for Linux.

https://www.spotify.com/nl/download/previews/. [17] SpotCommander.

http://code.google.com/p/spotcommander/. [18] Firebug.

https://addons.mozilla.org/nl/firefox/addon/firebug/. [19] Foobar2000 - foo httpcontrol plugin.

Music-based quality control in software engineering

Universiteit van Amsterdam