University of Groningen Bachelor thesis

(1)

University of Groningen

Bachelor thesis

Benchmarking Tool for the Cloud

Author:

W.G. Menninga

Primary supervisor:

prof. dr. A. Lazovik Secondary supervisor:

prof. dr. G.R. Renardel de Lavalette External supervisor:

E. Feliksik

Date:

July 5, 2015

(2)

Abstract

The number of cloud providers offering computing resources as Infrastructure as a Service (IaaS) is large and ever growing. Lots of different providers offer lots of different machine

types. These machines have different performance characteristics, prices and pricing models. Users, who want to buy cloud computing resources, need information about the performance and costs of running their application on these machines in order to make a choice of machine type that fits their price and performance requirements. In this thesis, we present a tool for benchmarking a user-supplied application on the different machine

types offered in the cloud, resulting in a running time and costs metric of running the application on each of these machines. This enables the user to compare the price and

performance of machine types of the different cloud providers for their specific application and aids them in making a decision based on these metrics.

(3)

List of Figures

1 Overview of the architecture of the framework presented by Xinni Ge et al. 7

2 Virtual machines compared to containers . . . 12

3 The global architecture of the benchmark tool . . . 18

4 ORM (Object Role Modeling) schema of the relational database used in the benchmark application . . . 20

5 The page where new benchmarks can be created . . . 26

6 The overview page of all the benchmarks . . . 26

7 The table displaying the results . . . 27

8 Graphs displaying the resource usage over time . . . 27

9 The page where running time and costs of a single application’s benchmarks can be compared . . . 28

10 The results of running the two benchmarks . . . 30

11 The graphs showing CPU usage of the two benchmarks . . . 30

Listings

1 An example of a Dockerfile for an image with a program to compute primes 12 2 An example of the function called when the benchmark overview page is visited . . . 19

3 The base object for deployment methods. . . 22

4 The base cloud provider object and the Amazon object overriding the part for network bandwidth cost computation. . . 23

(6)

Glossary

API Application Programming Interface.

FIFO First-in-First-out.

IaaS Infrastructure as a Service.

PaaS Platform as a Service.

SaaS Software as a Service.

SSH Secure Shell, a protocol for communicating with a remote machine in a secure way.

(7)

1 Introduction

Cloud computing platforms have become very popular as a means to process large computing jobs. There is an increasing amount of cloud providers, all offering different types of machines and for different prices. Machines range from small, with slower CPU’s and less memory, to large with a large amount of CPU cores and memory. Specialized machine types are also available in the cloud. Think of, for example, machines with a special type of storage, allowing for very fast disk IO.

Cloud providers do not only offer different machines, but also use very different pricing models. Certain cloud providers have a simple pricing model where a user simply pays a fixed fee for a certain instance per hour. Other cloud providers however, have a more complicated pricing model, where, for example, disk storage and network IO are charged separately.

A user that wants to buy computing power in the cloud wants to choose the optimal cloud provider. This requires knowledge about how the offered machines will perform when running the user’s computationally intensive job. The specifications about these different machine types on the website of the cloud provider offer very limited information to a user and is not adequate for finding out how a computing job will perform on a machine.

One way to get better insight into the pricing and performance of cloud providers is with benchmarking. However, this is a very time-consuming process to do manually, espe- cially when benchmarking multiple cloud providers. Not only the benchmarking itself, but also the complicated pricing models make it even more time-consuming when computing the costs of running a benchmark on a particular cloud provider.

A common approach to solve this is by running representative benchmarks, or finding results of previously ran benchmarks. These are benchmarks that aim to emulate the workload of the computing job and can give a lot of helpful information into the performance of the different cloud providers. However, it is still far from ideal. Finding a good representative benchmark is often difficult and it requires detailed knowledge about the system resources (like CPU/memory) that the compute job uses.

In this thesis we introduce a tool, that allows users to upload their own application, which is then benchmarked for multiple cloud providers in the cloud. This solves the problem of finding a representative benchmark and automates the process of benchmarking a job on multiple cloud providers, saving a lot of time.

(8)

1.1 Background

We now describe a typical use case of cloud computing. A data scientist might have a lot of data which needs to be analysed. Analysing this data on a normal computer is often not feasible, because it would take a lot of time or the job might not be able to run on it, as the machine does not have enough memory. To run this job, you would require more computing power. However, the job only needs to be run once. Therefore, it is not economically feasible to buy more computers or servers. Cloud computing offers a solution to this, by offering on-demand renting of computing power for any period of time, whether it is only one hour or several years.

There are a lot of machines available in the cloud, all with very different characteristics. Machines range from cheaper low-performance machines to more expensive high- performance machines. The user, the data scientist e.g., wants to make the choice for the cloud provider and machine type which offer an optimal price and performance for the compute job. Making this choice is not trivial and depends for a large part on how the application would run on that machine. One possibility is that the job uses a lot of CPU power, but another possibility is that the job uses barely any CPU and is only writing data to disk. Depending on this, a different machine might be optimal, since renting a very expensive and powerful machine is costly and to be avoided if the job running there does not use those expensive resources in an optimal way.

Finding the optimal machine for a task, which machine is the fastest or the cheapest or a combination of that, is difficult. Simply looking at the specification on the website of the cloud provider about the machines they have, offers very limited insight into how they perform in practice. Certainly if considered that every job has different characteristics, i.e.

CPU usage, memory usage, disk IO and network IO.

One approach to figuring out how a particular job will perform on all machines would be to just run a representative sample of the job on all of these machines. However, it is not feasible to do this by hand. This would be very time-consuming to do and, additionally to running the job on all the machines, it would also be necessary to obtain insight into how much running the application on each machine costs. Large differences between the pricing models of the different cloud providers exist, making it difficult to retrieve the costs of running a job on a particular cloud provider.

Not all cloud providers compute the costs in a similar way. Other than a base amount of money paid for the usage of the machine per hour, some cloud providers also charge additional fees based on the resource usage. For example, some cloud providers charge additionally for every gigabyte of bandwidth used. This makes the computation of the expected costs more difficult, since not only the number of variables in the pricing model increases, but it also requires knowledge of the resource usage of the job.

(9)

We now describe the typical process of buying computing power in the cloud. First, a choice of cloud provider has to be made. A cloud provider is a business that sells access to resources in the cloud. In the case of computing, this usually means providing access to a virtual machine running on a larger server (usually with multiple cores and a significant amount of memory). Examples of cloud providers are AmazonEC2 and Microsoft’s Azure.

After choosing a cloud provider, the next step of running a job in the cloud is to register for an account with the cloud provider. This often includes some identity verification step, usually through a credit card pre-authorization. Once an account has been created, an instance of a machine type can be spawned using the web interface of the cloud provider or by using the available API of the cloud provider. This results in a virtual machine being created, which can be communicated with using an SSH session or sometimes using the web interface of the cloud provider as well.

A virtual machine is an emulation of a computer system. It allows cloud providers to offer the same interface as an actual machine, but offers more flexibility. For example, multiple virtual machines can be run on a single physical server, in order to allow multiple users to use the same physical server. It also offers security, because everything the users runs is run within the virtual machine.

Once access to the virtual machine is obtained, the user will be able to install the necessary software and upload the necessary data to run the job there.

(10)

1.2 Research questions

The main research question is as follows:

How can users of cloud computing get insight into the price and performance of the offered machines for their specific task?

This main research question can be divided into the following sub-questions:

• What is a practical cost model for comparing cloud providers?

As mentioned before, cloud providers use a range of very different pricing models. In what way can we generalize these into a model that allows comparing the different cloud providers?

• How can the relevant cost parameters be measured in practice?

How can the parameters, which are necessary to compute the costs (like network usage etc.), be measured and stored in a way that allows computation of the price when the compute job benchmark is done?

• How can a job deployment and execution mechanism be implemented for multiple cloud providers?

Is it possible to automate the benchmarking of specific tasks in the cloud?

Is it possible to develop a tool that deploys a compute job to different cloud providers, while measuring runtime and other metrics, and compute the cost of running this job on the different cloud providers?

In order to build this tool, a way has to be found to define a compute job. Addition- ally, instances of the selected types have to be created on the platform of the cloud providers (using e.g. API’s made available by the cloud provider) and the job has to be uploaded to and run on there.

1.3 Structure of this thesis

Section 2 of this thesis takes a look at related work. This is followed in Section 3 by the problem statement. Section 4 performs an analysis of the problem and discusses techniques towards a solution. Section 5 discusses the design and implementation of the benchmark tool. This is followed by Section 6 with a demonstration of how the tool is used and how it is useful in practice. Section 7 discusses possible extensions and improvements to the tool. The last section, Section 8, is the conclusion.

(11)

2 Related work

Some previous work has been done in the area of benchmarking in the cloud. In this section the most relevant of this previous work will be discussed.

One of the first articles describing a systematic approach to cloud benchmarking was published by E. Folkerts et al.[5]. It describes the requirements benchmarks should fulfil, divided into three groups: general requirements, implementation requirements and workload requirements. The general requirements entail that a benchmark should be simple, economical and relevant. Implementation requirements describe that a benchmark should be fair, portable, repeatable, realistic, comprehensive and configurable. The third cate- gory, workload requirements, state that benchmarks should be representative, scalable and should produce an understandable metric.

Previous research has also been done into the development of tools for running benchmarks in the cloud.

One such tool is CloudRank-D[10]. CloudRank-D is a benchmark suite for evaluating cloud computing systems, focusing mainly on computing systems that are shared and run big data applications. The article introduces two new complementary metrics: data processed per second and data processed per joule.

The article describes three main usage scenarios of CloudRank-D. First, it can be used to quantitatively measure the performance of different systems in the cloud. Secondly, the tool can be used to optimize a system. And thirdly, the results of the tool can be used to rank the different systems in the cloud.

The tool uses pre-defined scenarios with different workloads as benchmark jobs. The costs (i.e. pricing) of running the benchmark is not one of the computed metrics in this benchmark suite.

Research describing how different pricing models of cloud providers can be compared in an optimal way is scarce. Some research however, does mention differences in the way cloud providers compute the resulting price.

In 2012 B. Martens et al. in their research[11], distinguished the different pricing schemes between IaaS, PaaS and SaaS service models. They noticed that a wide variety of different pricing schemes exist within the IaaS market. They found that most providers offer hourly based rates, but that some cloud providers tried to attract customers with lower pricing, while charging ‘hidden’ costs for, e.g., inbound/outbound data transfer.

There are even cloud providers that charge for data transfer within the infrastructure of the cloud provider itself (i.e. when data is transferred between two locations in the same data center).

Additionally, their research found that not all cloud providers adhered to this more common pricing model. The cloud provider FlexiScale had introduced their own so-called

‘units’, serving as a currency for buying computing power.

(12)

While on-demand resource provisioning is an important property of the cloud, not all cloud providers charge based on usage. Some cloud providers offer cloud computing resources at a fixed price per month. This fixed-price package can then often be extended when the user requires this. An example is when the bandwidth included with the fixed- price package is exceeded and charging per extra gigabyte of bandwidth starts.

A hybrid of fixed-price and on-demand pricing exists as well, where a buyer of cloud computing resources has the option to pay a fixed fee in order to get a discounted hourly rate (Amazon’s reserved instances are an example of this).

Previous work by R. Prodan et al. [12] showed that the interval used for billing can also play a significant role in the total price of running a task in the cloud. Several benchmarks were deployed and executed in the cloud of Google App Engine and Amazon EC2. The cost estimation afterwards determined that Google App Engine was cheaper at the time, largely due to the fact that costs were computed based on the number of CPU cycles used, as opposed to Amazon’s method, where a fixed fee per hour (rounded up to the nearest hour) was computed.

A benchmark tool which takes the costs of running a benchmark into account is Cloud- Cmp[9]. CloudCmp performs measurements on the elastic computing, persistent storage and networking services offered by cloud providers. The tool can be used to get insight into the performance and costs of the major cloud providers in the market.

Three metrics are used for the comparison of the elastic compute service offered by the different cloud providers: benchmark finishing time, cost and scaling latency. The benchmark finishing time records how much time the benchmark needed in order to finish.

The cost is the monetary cost of running the benchmark on a specific cloud provider. It can be used in combination with the time metric to get insight into the price-performance relation of cloud providers.

The third metric is scaling latency. This metric records how much time it takes to spawn a new instance after the user requests it. This metric gives insight into how quickly an application can scale up on a particular cloud provider when the workload increases and requires more instances.

Similar to the other benchmarking tools described, this tool also comes with several pre-defined benchmark tasks, which generate a specific workload to emulate a particular scenario.

In 2012, Vedam and Vemulapati[14] mentioned the role of automation for benchmarking in the cloud as a possible solution to the problem of the cumbersome and time-consuming process of benchmarking for a larger number of machines. Several benefits of automation in cloud benchmarking are mentioned in this article:

• Repetition of the benchmark process is easy and not very time-consuming

• Elimination of human error

(13)

• Reduced running time of instances as a result of releasing the instances immediately after the benchmark completes

• Result reports can be automatically generated

Figure 1: Overview of the architecture of the framework presented by Xinni Ge et al.

In 2014, an article was published by Xinni Ge et al.[6] which described a benchmark framework that automates performance modeling on IaaS clouds. The article describes the architecture used for the automation of benchmarking. A crucial component for the automation in the architecture presented in that article is the Core Daemon component, which maintains a FIFO queue of all the benchmark jobs. Another component, the Job queue controller is responsible for managing and tracking the status of the benchmark jobs.

When a benchmark is to be done, orchestration (launching of instances and resources) is handled by the Coordinator Daemon. It launches the instance by communicating with another module, the Coordinator API. The Coordinator API uses two modules for the communication with the instance. The first one is the Cloud API, which is an abstraction layer on top of the APIs offered by all supported cloud providers and is used for launching the instance. After launching the instance, further communication with that instance is handled using the Communication Abstract Layer module, which is the second module used by the Coordinator API. After the benchmark is finished, the Coordinator Daemon will process the results. An overview of the architecture of this framework is visible in Figure 1. This framework uses a modular architecture in combination with APIs provided by the cloud providers to achieve automation of the benchmark deployment and execution.

However, it still uses a set of pre-defined benchmark scenarios.

One other tool is Smart Cloud Broker, developed by M.B. Chhetri et al.[1, 2]. This is a suite of software tools for evaluating and comparing the performance of different cloud providers.

(14)

Smart Cloud Broker allows the user to select a cloud provider, machine type and workload scenario for a benchmark, and then runs this benchmark on the selected cloud provider.

A range of pre-defined scenarios is available and the user can change the workload to obtain insight of how the performance of the machine changes with different workload scenarios.

The tool produces a results report, where performance and configuration information is combined with pricing information.

To illustrate how this tool is used in practice, we now describe the steps involved in a typical benchmark as described in the article.

The first step is provider and benchmark selection. Here, the user chooses a cloud provider and resource configuration from a list. After selecting a cloud providers, the user selects a representative benchmark from the list of available benchmarks. After a benchmark has been selected, the user has to specify the workload for the benchmark. After these steps, the benchmark is deployed and started. This is done by sending a request to the cloud provider’s API, which launches the VM of the required type, with a pre-built image containing the packaged benchmarking application. After the benchmark is finished, the results are collected and a report is generated and displayed to the user. The final step is instance decommissioning, where the spawned instances are removed again.

There are two key differences between the previous tools mentioned above and the one further described in this thesis.

The first difference is that the cloud benchmarking tool from this thesis focuses specif- ically on cloud computing. It focuses on tasks which are computationally intensive. For these types of tasks metrics like uptime (which is very relevant for when a website is being hosted on a server) are less relevant. Measuring metrics like response time (which could be relevant for applications running on multiple machines) are out of the scope of this benchmark tool.

The second key-difference is that the tool developed for this thesis is built to benchmark a user-supplied application. Instead of having a user select a benchmark scenario (with or without a custom workload), a user will upload his/her own application. This largely solves the difficulty of finding the right benchmark scenario (and workload) and the problem of the benchmark scenario not being a good match for the relevant job. For users, it is not trivial and can be hard to translate their own task into one of the pre-defined workloads. Most tasks do not translate cleanly into a pre-defined scenario.

(15)

3 Problem statement

The cloud has become very popular as a means to get more computing power. The variety of machines available in the cloud is vast. Those who want to buy computing power face the choice of many cloud providers with many different instance types. Every cloud provider has different characteristics. Some cloud providers offer cheap low-performance machines, some more expensive high-performance machines and others offer specialized hardware stacks. Users of cloud computing resources want to choose the cloud provider with the optimal performance and price for their specific computationally intensive job. However, the large amount of different choices makes this choice difficult.

Even if the precise resource usage of the compute job is known and the user has an idea of what kind of machine type he/she is looking for, many choices are still left to choose from. Some information about the machines offered is provided by cloud providers. This typically includes information about the hardware configuration (i.e. number of CPUs, memory available et cetera) and occasionally information about the actual hardware used in the physical machine. However, to the buyer of the computing resources, this offers almost no useful insight into how his/her specific compute job will perform on that machine.

One way to get more insight into the performance of a compute job on a particular machine is using benchmarking. However, benchmarking the performance of all relevant cloud providers is very time-consuming and expensive and often not feasible for all relevant cloud providers.

In the previous section it was discussed that several tools to aid with benchmarking in the cloud have been developed. However, while the benchmarking tools offer an extended set of pre-defined scenarios, they do not allow for benchmarking the user’s specific application. Such a tool would benefit those users that want to gain insight into the price- performance levels of cloud providers for their task.

We aim to build a tool for benchmarking a user-supplied application in the cloud that solves the following problem:

Given a user-supplied computationally intensive application and a list of instance types of cloud providers, produce cost and runtime metrics of the application for the different instance types to allow comparison.

The scope of this tool is limited to benchmarking computationally intensive jobs, as opposed to other services offered in the cloud, like storage, where other properties (like uptime) are important.

Comparing performance and price of different cloud providers is relevant for several groups of users. Companies and researchers that want to run a computationally intensive

(16)

application at a cloud provider want to know which cloud providers they should choose and therefore want to know about the pricing and performance of these cloud providers for their specific job.

Another group of users are developers who want to gain insight into how their application performs on a particular cloud provider. This information can help them in optimization of that application, which reduces the cost of running the application.

A third group of users are cloud providers themselves. Cloud providers want to compare the performance and pricing of the machines they offer in the cloud to those offered by competing cloud providers in order to adjust their own pricing to compete in the cloud computing market.

A solution to the problem would help the users with comparing the cloud providers for their computationally intensive application, based on price and performance.

(17)

4 Analysis

In this section, we describe how the specific job/application of a user can be represented in a way that allows deploying it to multiple cloud providers. Next, we describe how deployment to multiple cloud providers can be achieved. We continue with describing how the execution of the job can be monitored while measuring resource usage parameters needed to compute the price. The last part of this section describes how a generic pricing model, which approximates all the different pricing models of cloud providers, can be constructed.

4.1 Representing a user’s application

In order to benchmark a specific application of a user, this application will have to be uploaded to the cloud providers. This requires that this application can be isolated and made portable, in a way that it can be uploaded with all it’s dependencies and data included. This ‘package’, containing the application, dependencies and possibly data, has to be portable to the extend that the application in it can be run on all the cloud providers.

For the application to run as expected, it is important that it runs in the same environment on all the machines in the cloud as when it is run locally by the user. This environment should contain all the dependencies that the application needs.

A naive approach would be to define a set of instructions (like a bash script), which together install the application with dependencies on the machines. However, the biggest flaw with that approach is that it does not guarantee that the resulting environment is the same on all machines. It is possible that dependencies of the application are already present in the operating system, when they are not present in the set of instructions. This would break the application if it was moved to a different machine.

A recent development in the world of software deployment is the use of operating- system-level virtualization. Operating-system-level virtualization is a virtualization method where the kernel of an operating system allows the isolation of multiple instances of the user space of the operating system. Such an isolated instance is often called a container.

Leveraging this type of virtualization over full virtualization (i.e. hardware emulation) offers a significant performance increase, because there is no need for emulation of the hardware[15].

Figure 2 illustrates the difference between virtualization using hardware emulation and operating-system-level virtualization. As opposed to a virtual machine, a container does not have a guest operating system. Additionally, it does not need a hypervisor for emulat- ing virtual hardware and trapping privileged instructions. This saves a lot of performance overhead.

One tool that makes use of operating-system-level virtualization is Docker. Docker can

(18)

be used to package applications and dependencies in a ‘container’. Running this container using Docker will run it within a separate user space using operating-system-level virtualization.

Docker containers are instances of running Docker images. The process of creating a Docker image is relatively uncomplicated. It involves the creation of a so-called ‘Dockerfile’, which is a type of Makefile, containing instructions that are to be executed to install the application and dependencies (using the RUN instruction), and copy relevant data into the container (using ADD). Every Dockerfile extends another image (using the FROM instruction), which is either a previously created image or a base image (often containing the files of a specific operating system). See Listing 1 for an example of a Dockerfile.

1 FROM ubuntu:latest

2 MAINTAINER W.G. Menninga <W.G.Menninga@student.rug.nl>

3 RUN apt-get update

4 RUN apt-get install -y dpkg-dev debhelper build-essential libcr-dev mpich2 mpich2-doc

5 RUN mkdir /prime

6 ADD ./prime.c /prime/prime.c

7 RUN mpicc -Wall -O3 -o prime prime.c -lm

Listing 1: An example of a Dockerfile for an image with a program to compute primes

Docker supports exporting images to tar-archives, which can be imported between similar versions of Docker.

Docker can be used in several ways to allow a user to upload his/her isolated application (and data) to the benchmarker tool. First, the user can upload his Dockerfile and the files

Figure 2: Virtual machines compared to containers

(19)

needed to build the image, packaged in a tar-archive. This prevents the user from having to upload the ‘base image’ (which the Dockerfile extends).

The second method is to let the user build the image on his/her local system and upload an exported version of that image (a tar-archive) to the benchmark tool. This image has been build, does contain the files from the base image and is therefore larger. How much larger this image is depends on which base image was used.

For usability reasons, the second method was chosen for the benchmark tool. It allows a user to test the application container before uploading it. Secondly, it also prevents build errors from occurring on the server, which would mean that a user has to fix the errors and upload again.

Docker can be used to isolate the users application in a way that allows uploading it and deploying it to machines in the cloud. A user can create a docker image, with all needed dependencies and data inside, export this and upload the resulting tar-archive to the tool, from where it can be distributed to individual machines of cloud providers.

This guarantees that the application being benchmarked is the same on all the cloud providers and is also the exact same application as the user intended to benchmark.

4.2 Deployment to multiple cloud providers

This section describes how the user-supplied application can be deployed and executed on the different cloud providers. We discuss the available methods for deployment and execution of the user-supplied application.

The deployment and execution entails the creation of a virtual machine at the cloud provider of the type selected by the user. Additionally, the user-supplied application has to be uploaded and executed on the spawned virtual machine. After the execution of the benchmark has completed, the virtual machine has to be stopped and decommissioned, in order to limit the costs of the benchmark.

The way the tool deploys applications should not limit the number of cloud providers.

It has to be possible to extend the support for cloud providers in a later stage, since the cloud computing market is subject to change.

4.2.1 APIs of cloud providers

Almost all cloud providers in the market today offer an API, which can be used to pro- gramatically communicate with the cloud provider. Often, multiple versions of an API are made available by a cloud provider for different programming languages.

All of these APIs can be used to spawn new virtual machines at a cloud provider, set up an SSH session, which can be used to issue commands to the virtual machine, and (when the benchmark is done), stop and destroy the virtual machine.

(20)

The APIs offered by the cloud providers all have different interfaces. Methods are named differently and are called with different parameters. This means, that all of the APIs have to be implemented independently of each other. Not only does this require more code, but it also means more costs of maintaining the application, as these APIs are often subject to changes, which would require updating the benchmark application for it in order to remain functioning.

4.2.2 Apache Libcloud

Apache Libcloud is a Python library offering a more uniform interface to communicate with cloud providers. It is implement as a wrapper around the APIs offered by the cloud providers. Libcloud supports around 30 cloud providers.

However, while Libcloud does offer a more uniform interface than the individual APIs of the cloud providers, some differences are still present.

Moreover, Libcloud does not solve the entire deployment and execution problem. After virtual machines have been created, the user-supplied application still needs to be uploaded and the spawned machine has to be configured for executing Docker images.

4.2.3 Docker Machine

Docker Machine is a tool created to work on top of Docker. It is capable of creating new virtual machines in the cloud and automatically configures these machines to communicate with the Docker client on the host machine. After this configuration step, the user-supplied application can be imported and executed in the same manner as it would be done locally, while Docker handles the deployment to the remote virtual machine of the cloud providers automatically.

Internally, after spawning the virtual machine at the cloud provider, Docker Machine generates a unique SSH key for communication with the virtual machine. Using this SSH key, it installs Docker on the virtual machine and configures this Docker installation to accept remote connections. It will then store the SSH key locally, to enable the Docker client running there to communicate with the remote Docker client on the virtual machine.

The mayor advantage of this method is that it handles the entire deployment and execution of the Docker image. The local Docker client will handle all necessary steps to deploy and execute the docker container on the remote machine.

This approach of deployment and execution almost completely hides the differences between the cloud providers. Only the information supplied to Docker Machine for au- thentication differs for each cloud provider (some require a single token, other multiple).

For this reason, Docker machine was chosen as the main deployment method for the bench-

(21)

mark tool.

4.2.4 A modular approach

Since the added value of a benchmark tool is for a large part decided by the number of cloud providers supported, adding new cloud providers in a later stage is something the design of the benchmark tool should allow without too much effort. Therefore, a modular design, where methods of deployment (i.e. using docker machine or libcloud) are modules exposing a uniform interface to the rest of the system has been chosen.

This allows using different methods for the deployment and execution of the user- supplied application and makes it easier to add support for more cloud providers, using different methods for deployment, in a later stage.

(22)

4.3 Constructing a generic pricing model

In order to do a comparison of the different machine types of the cloud providers, it is essential to know how much money running a job on a particular machine costs. For the costs to be computed in a uniform way, the different pricing models of the different cloud providers have to be generalized.

In order to create a generic model that approximates the pricing models of all the cloud providers, we have looked at the pricing models of the larger cloud providers in the market, and three in specific: Amazon, Azure and Digital Ocean.

All three of these providers charge a fixed amount of money per machine type for every hour of usage. Additionally to this fixed amount per hour, two of these cloud providers (Amazon and Azure) charge bandwidth separately per gigabyte of bandwidth used.

Some cloud providers in the market charge for every megabyte written to disk or for every disk operation, which means that these metrics also have to be a part of the pricing equation.

Taking all these factors into account, the following generic pricing equation seems to approximate most of the cloud providers in the market today:

total price = hourly price * nr of hours + network io read bytes * network io read bytes price + network io write bytes * network io write bytes price +

disk io read ops * disk io read op price + disk io write ops * disk io write op price +

disk io read mbs * disk io read mb price + disk io write mbs * disk io write mb price

This pricing model is a generalization for the pricing models of most cloud providers, which base their pricing on an hourly rate with possible additional rates for resource usage. However, while this equation is a good approximation of the pricing models of most cloud providers, some of the prices in this equation are not fixed. The price of network IO is often based on how much network IO was used. Amazon, for example, uses different tiers to determine the price of a gigabyte of data transferred out of the datacenter: the first gigabyte is free, up to 10 TB is $0.090, 10TB - 40TB costs $0.085 per gigabyte etcetera.

In order to accommodate these different tiers and pricing models with an alternative structure, the implemented pricing model must allow overriding individual parts of this equation, or the equation as a whole, for individual cloud providers. This is necessary to make sure that the computed price is accurate enough to use it for a fair and representative comparison with the other cloud providers.

(23)

4.4 Cost parameters monitoring

During execution of the benchmark of the application, metrics about the resource usage have to be collected. These recorded metrics serve two purposes. First, they are necessary in order to compute the price, since some cloud providers charge additional fees depending on the resource usage.

The second purpose of collecting these metrics is to facilitate the generation of graphs, showing the resource usage over time. This gives the user of the benchmark tool valuable insight into the resource usage of his/her application and how this resource usage changes when the application progresses in its execution.

For example, a user could run a benchmark of his/her application on a particular machine and find that the maximum memory usage during the execution is 1 gigabyte. This information can help the user in his/her decision about which machine to choose.

The following metrics will have to be recorded during the execution of a benchmark to get insight into the most common resources used:

• CPU usage - The user CPU usage is measured to give a user information about whether the CPU is used optimally by the application or not.

• Memory usage - This is the total memory usage of the machine RAM.

• Disk usage - Disk usage should be measured in both number of operations and bytes (read and write separately).

• Network usage - The network usage should be measured in bytes written and read.

Certain cloud providers make a distinction in the price computation between traffic within the data centre and traffic leaving the data centre. This distinction is not made in the pricing model, because it would require inspecting the network traffic in a way that the destination can be reliably determined, which does not fit into the scope of this project.

For the benchmark, the resource usage of the entire virtual machine of the cloud provider has to be measured, since this is where the pricing of the cloud providers is based on.

(24)

5 Design and implementation

This section discusses the design and implementation of the benchmark tool. It discusses the used architecture and mentions the tools and techniques used to implement the benchmark tool.

5.1 Architecture of the benchmark tool

In Figure 3, the global architecture of the benchmark tool can be seen. In general the tool has been designed in a way that facilitates modularity of individual components.

This modularity property allows for certain components to be replaced without affecting the other components in the application. This allows replacement of specific modules,

Figure 3: The global architecture of the benchmark tool

(25)

like the ‘Price Engine’ (responsible for computing the costs of running a benchmark), when this is beneficial to increase accuracy of results (more accurate price computation) or extend functionality (add support for additional cloud providers).

In the next part, we discuss these individual components of the architecture.

5.2 Frontend and controller

The frontend is used by the user of the benchmarking tool to upload his/her application and create new benchmarks. This is also where the progress of the benchmarks and the results can be viewed. The application frontend is developed as a web application.

Interactions from the user are handled by the controller of the web application. The controller and frontend have been implemented in Python with use of the Flask web development microframework [4]. As a microframework, Flask makes it easy to connect URLs to certain views. A Python decorator (@app.route("URL")) can be used to assign a URL (with or without parameters) to a function. This function should return a string, which is the response of the web server when the user visits that URL. An example of this can be seen in Listing 2. In this example, the URL for the overview page of all the benchmarks (/benchmarks/view/) is returned using the function render_template(), which uses the Jinja2 template rendering engine to render a template with the passed variables.

The rendered HTML template is returned by the web server and the web browser of the user will render the HTML.

1 @app.route("/benchmarks/view")

2 def view_benchmarks():

3 benchmarks = Benchmark.query.join(Benchmark.dockerimage).filter(

DockerImage.user == current_user.id)

4 return render_template("view_benchmarks.html", benchmarks=

benchmarks)

Listing 2: An example of the function called when the benchmark overview page is visited

(26)

5.3 Relational database

The benchmark tool makes use of a database for storing relational persistent data, like user and benchmark information. The schema in Figure 4 shows the objects stored in the database and the roles between them.

Figure 4: ORM (Object Role Modeling) schema of the relational database used in the benchmark application

To communicate with the database from within the web application, an Object Rela- tional Mapping toolkit, SQLAlchemy, is used. This allows querying the database, while using Python objects, instead of having to use the PosgreSQL SQL dialect. SQLAlchemy supports several database dialects, including PostgreSQL, MySQL and Microsoft SQL Server[13]. SQLAlchemy does not only help to query the database in a more object-oriented way, but also helps in decoupling the database from the rest of the application. The Post- greSQL database could be replaced by another type of relational database supported by SQLAlchemy without affecting the rest of the application.

(27)

5.4 Orchestrator

After a user has created a benchmark using the web interface, machines have to be spawned of all the different types selected by the user and the user-uploaded application has to be benchmarked on all of these machines. This is handled by the orchestrator component.

This orchestrator component cannot be implemented as part of the normal request- response cycle of the web application, because deployment of machines and running the benchmark is a time-consuming process, which can take up to an hour or more, depending on the application and machine being benchmarked. Therefore, the orchestrator component has been implemented using a Celery task queue. After a user has created a benchmark, a task will be created and placed in a task queue. Whenever a worker thread becomes available, it will start execution of this task asynchronously. During the execution of the task, the state of the benchmark will be updated in the database. Every machine being benchmarked results in a new task. For example, running a particular benchmark on two machines, one of Amazon and one of Azure, will result in two tasks being placed in the task queue.

5.4.1 Benchmark deployment and execution

A task starts with spawning a virtual machine of the selected type at the selected cloud provider. To accomplish this a separate module (Benchmark Deployer/Runner) is used.

The Benchmark Deployer/Runner component is responsible for the creation of new machines in the cloud of the selected type, installing Docker on them, uploading the user application Docker image and running the benchmark by executing the application Docker image using Docker.

The Benchmark Deployer/Runner can support several components (called deployment methods) to deploy the benchmark to a cloud provider. Which method is used depends on the cloud provider which has been selected. By supporting different methods, different techniques can be used to deploy applications to the cloud, which means that more cloud providers can be supported by the benchmark tool. These deployment methods have been implemented as objects, exposing three functions: one for creating a new machine and configuring it to run the benchmark, one for starting the benchmark and one for stopping and deleting the spawned machine. Separation of the logic into these three functions allow the Orchestrator component to call them individually. This is needed because the Orchestrator needs to determine when the benchmark starts in order to record the start time, which is needed to compute the total execution time of the benchmark. The base object for these components can be seen in Listing 3.

If a cloud provider is not supported by one of the deployment methods, a new deployment method can be added which does support that cloud provider. This can be a deployment method using another tool, like Libcloud, or using the API offered by the cloud

(28)

provider itself.

At the moment, the benchmark tool only supports deployment by using Docker Ma- chine. Other deployment methods can be added in the future.

1 class DeploymentMethod(object):

2 @staticmethod

3 def deploy_benchmark(name, docker_image, docker_image_name,

4 cloudprovider, instancetype):

5 raise NotImplementedError()

6

7 @staticmethod

8 def start_benchmark(name, docker_image, docker_image_name,

9 cloudprovider, instancetype, command):

11

12 @staticmethod

13 def clean_benchmark(name):

Listing 3: The base object for deployment methods.

5.5 Monitoring resource usage

As discussed in Section 4.4, in order to compute the price after the benchmark is finished and to give the user insight into his/her application’s resource usage over time, resource usage metrics had to be collected during the execution of the benchmark.

To collect metrics about resource usage, a separate Docker container is deployed next to the container with the user application. This container contains collectd, a tool for collecting system performance metrics[3]. For the benchmark application, collectd was configured to collect metrics about the memory usage, CPU usage, network usage and disk usage.

Memory usage is measured in bytes, network usage is measured in bytes (transmit and receive separately), and disk usage is measured in both bytes and number of disk operations (read and write separately).

CPU usage is measured in ‘jiffies’. Basically, a ‘jiffie’ is a small period of CPU time, which the operating system’s scheduler assigns to a process. On most Linux systems there are about 100 jiffies available per second. However, depending on factors such as system load, hardware and possibly other factors, more or less jiffies can be available in a second.

Therefore, while the amount of ‘jiffies’ used in a second is an indication of the CPU per- centage used in that second, it is only a rough indication, since the amount of jiffies in a second could differ between seconds.

(29)

The metrics recorded by collectd are sent to an external server every 10 seconds. This server runs a time-series database, InfluxDB, which is a database optimized for handling time series data (rows with data indexed by a timestamp). Storing the metrics on an external server allows querying these metrics after the benchmark has finished and the machine has been destroyed.

5.6 Resource usage aggregation

After a benchmark has finished, the Orchestrator will call the Resource Usage Aggregator component to compute a summary of the resource usage using the metrics stored in In- fluxDB. The resource usage aggregator iterates over all the disks of the machine to compute the total disk usage and iterates over all the CPU cores to compute the total CPU usage.

Additionally, it also computes the total memory usage and network usage.

5.7 Price computation

After a benchmark is finished, the costs of running that benchmark are computed. The logic for this is contained in the Price Engine module.

A base cloud provider object is used, which implements the common logic of the cloud providers. It uses the generic pricing equation from Section 4.4 and has been implemented in a way that allows overriding individual parts of this equation in objects extending this base cloud provider. An example of where an individual part of this equation is overridden is the computation of the price of outgoing network traffic for Amazon EC2 instances.

Here, a fixed price is paid for every GB of bandwidth sent, but unlike in the generic base object, the first GB of usage is free. This can be seen in Listing 4.

1 class CloudProvider(object):

2 (...)

3 def compute_price(self, result, instancetype, region):

4 return self.instance_price(result, instancetype, region) + \

5 self.disk_read_op_price(result, instancetype, region) + \

6 self.disk_write_op_price(result, instancetype, region) + \

7 self.disk_read_byte_price(result, instancetype, region) + \

8 self.disk_write_byte_price(result, instancetype, region) + \

9 self.network_rx_byte_price(result, instancetype, region) + \

10 self.network_tx_byte_price(result, instancetype, region) + \

11 self.storage_byte_price(result, instancetype, region)

12

13 def network_tx_byte_price(self, result, instancetype, region):

14 return self.PRICE_NETWORK_TX_BYTE * result[’network_out’]

15

16 class AmazonEC2(CloudProvider):

17 def network_tx_byte_price(self, result, instancetype, region):

(30)

18 # first GB free of charge

19 return self.PRICE_NETWORK_TX_BYTE * max(0, result[’network_out

’] - 1000000000)

Listing 4: The base cloud provider object and the Amazon object overriding the part for network bandwidth cost computation.

(31)

6 Demonstration and results

In this section, the developed tool for benchmarking user-supplied applications in the cloud is demonstrated. The steps involved in executing a benchmark using the tool are discussed and illustrated using relevant screenshots.

1. The first step a user should take when he/she wants to run a benchmark of an application in the cloud, is using Docker to create a Docker image with that application and export this image as a tar-archive, so it can be uploaded to the web application.

2. Uploading the exported docker image to the web application is the second step. The user also has to supply the name of the Docker image, which has to be the same name as was given to the image using Docker. The uploaded docker image will be stored by the benchmark application.

3. After the user has uploaded the application, a new benchmark can be created. This involves selecting the application from a table of all uploaded applications, supplying a command to start the application and selecting all the machine types on which to run the benchmark. This benchmark creation page can be seen in the screenshot of Figure 5. Currently, the benchmark tool supports 60 different machine types spread over the cloud providers Digital Ocean, Amazon, Azure and Exoscale.

Allowing the user to supply a command to start the benchmark in this stage allows the user to run a benchmark of the same application with different input or configuration, without the need for uploading a new version of the application.

4. After the benchmark has been created, the progress of the individual benchmarks running on all the selected machine types can be followed on the benchmarks overview page (as can be seen in Figure 6). After a benchmark has finished, a link to the benchmarks result page becomes available.

5. When an individual benchmark is finished, the results of that benchmark can be viewed. The results page includes a table with a summary of the recorded resource usage metrics and the total runtime and costs of the benchmark. The page also displays graphs of the resource usage over time, in order to give the user insight into the resource usage of his/her application. This can be seen in figures 7 and 8.

6. When all the individual benchmarks are finished, the comparison page can be used to quickly compare the prices and performance of the different machines. This can be seen in Figure 9.

(32)

Figure 5: The page where new benchmarks can be created

Figure 6: The overview page of all the benchmarks

(33)

Figure 7: The table displaying the results

Figure 8: Graphs displaying the resource usage over time

(34)

Figure 9: The page where running time and costs of a single application’s benchmarks can be compared

6.1 An example use case

An example that shows the value of the tool can be illustrated using a benchmark of two machines. The first one is a Digital Ocean machine with 48GB or RAM and 16 CPU cores, costing 0.17 dollar per hour. The second one is a machine from Microsoft Azure, with 112GB or RAM and with 16 CPU cores as well, which costs 2.08 dollar per hour.

The information supplied on the website of the cloud provider does not supply information about hardware characteristics (like CPU speed etc.) on which these machines can be compared in a useful way. This is a typical scenario where benchmarking can be really useful.

Say a data scientist wants to run an computationally intensive application in the cloud.

This application produces a lot of data, which is stored in memory. Looking at the specifications and price of these two machines, one might suppose that the machine from Digital Ocean will be cheaper in the end, since it only costs a third of the Azure machine.

Running a benchmark of these two machines for this application, revealed that the Azure machine was cheaper ($0.03 as opposed to $0.06), because it needed only 45 seconds as opposed to the Digital Ocean machine, which took 5 minutes. This can be seen in Figure 10. Inspection of the resource usage graphs (shown in Figure 11) show that, while

(35)

most likely caused by a lower memory write speed of the Digital Ocean machines. All the data being produced by the application and written to memory, cause the slower memory of the Digital Ocean machine to become a bottleneck, making the entire benchmark a lot slower and therefore more expensive.

(36)

Figure 10: The results of running the two benchmarks

Figure 11: The graphs showing CPU usage of the two benchmarks

(37)

7 Future work

This section discusses the extensions and improvements which can be made to the tool.

7.1 Increase support for cloud providers

An obvious way in which the tool can be improved is to extend the support for cloud providers. At the moment, the tool supports 60 different machine types, spread across four different cloud providers. However, the number of cloud providers in the market is significantly larger. Adding support for more cloud providers to the benchmarking tool would directly increase its value.

The tool has been designed in a modular way, such that additional cloud providers can be added without too much effort. For example, by implementing a separate module, using Libcloud as a tool for deployment, the support for more cloud providers can easily be increased.

7.2 Variability of results

The performance of machines in the cloud depend on lots of factors, including possible hardware differences and how resources are divided over clients. It is very well possible that running the same benchmark configuration (application-command combination) several times on the same machine type results in different runtime and cost metrics. Previous work from Iosup et al.[8] showed that there can be large differences between the minimum, mean and maximum performance of a machine with the same workload. They also showed that the performance of machines changes over time[7].

In order to get a reliable result, a benchmark should be run multiple times. Vedam and Vemulapati noted in 2012[14] that a benchmark should be run at least three times with the same configuration.

At the moment, the responsibility of running a benchmark numerous times to get reliable results lie at the user’s end. It would be an improvement if the tool would be capable itself of running a benchmark numerous times and would be able to report the differences between the different benchmarks of the same machine, so that the user is able to use that information to get an idea of how stable the performance of a particular machine type is.

7.3 Scaling latency

Unlike CloudCMP[9], this tool does not measure the scaling latency. The scaling latency is the amount of time needed to create a new machine at a cloud provider. Knowledge about the scaling latency can help a user make a choice, when the speed at which a machine can be created is important to that user.

(38)

It would therefore be beneficial to the users of the benchmark tool, if the time between requesting a new instance at a cloud provider and the moment the cloud provider has created this new instance would be measured and displayed to the user additionally to the total runtime and cost metrics.

7.4 Running benchmarks in different regions

Most cloud providers offer users the ability to rent machines from different regions. Pricing of these machines generally differs between these regions. The benchmark tool currently does allow a user to select a region for the costs computation when a benchmark has finished, but a user cannot choose in what region the benchmark is executed. It is very well possible that not only the price, but also the performance of the same machine type in different regions is not the same. It would be an improvement if the tool would allow the user to choose the region in which to benchmark the application, allowing the user to compare the performance of a machine type between different regions.

7.5 Costs extrapolation using historic benchmark results

By using the resource utilization of a job and historic benchmark results of other jobs, it might be made possible to predict the costs of running that job on other cloud providers.

When a job is CPU bound and the amount of IO (disk and network reading/writing) is not significant, previous results might be used to extrapolate the costs/runtime of running a job on cloud providers B and C based on the resource usage measured while benchmarking the job on cloud provider A.

For example, if a job is CPU bound and has no network or disk IO and historical results of other CPU bound benchmarks show that machine A is twice as slow and expensive when compared to machine B, then the price/runtime of the job when benchmarked on machine A can be used to extrapolate the price/runtime of running that job on machine B, without actually running it there.

This would allow for a faster and less expensive comparison of the different machine types.

7.6 Multi-container deployments

The benchmarking tool is not capable of benchmarking more complicated architectures spanning multiple machines. An example of such an architecture is a pipeline architecture, where machines do specific types of computations and the result of one machine is the input of another.

It is possible to benchmark the individual parts of such architectures, but certain performance elements would not be taken into account. For example, the network latency between machines could play an important role in the performance of such an application and could depend largely on the choice of machine. Benchmarking such multi-machine

(39)

architectures in the cloud is a challenge to overcome in the future.

Additionally to architectures spanning multiple machines, architectures also often contain multiple components, where each component individually is an application in itself.

For example, an architecture could have a computational component, a database component and a message queue component.

In practice, these applications are often placed in their own containers, which are then linked, allowing communication between them. These types of applications, running on a single machine, but consisting of multiple containers are also not yet supported and could be added in the future.

7.7 Cost prediction with continued use

The pricing models used by cloud providers often include discounts, which are not appli- cable for machines running a relatively small amount of time (like most benchmarks). One example of this is the sustained use discount of Google Compute Engine, where the price of a machine per hour becomes less when a machine is used for more than 25% of the month. Another example of this is the tier-based pricing model used by Amazon for the price of data transferred out of the datacenter, where the price per gigabyte decreases if more bandwidth is used.

When the costs of a benchmark are computed, these types of discounts usually do not apply due to the relatively small time span in which a benchmark runs. However, in practice, the application of the user might run for a longer period of time, and these discount could apply.

An example would be a user benchmarking the conversion of a single video file. Running this benchmark would take approximately 15 minutes, meaning that sustained use and similar discounts do not apply. However, in practice, this user might want to convert videos non-stop, converting a new video when the previous conversion is finished. If a user would do so, sustained use would apply.

The tool could be improved by allowing the user to indicate he/she plans on continu- ously using a machine and by changing the cost computation in a way that it takes this into account.

(40)

8 Conclusion

Users of computing resources in the cloud need to have insight into the price and performance of the different offered machines in order to make a choice of cloud provider.

In this thesis, we answered how users of cloud computing can get insight into the price and performance of the machines available in the cloud.

One way of getting more insight into the price and performance of these different machines is through benchmarking. However, current tools only support benchmarking with pre-defined scenarios, which does not give the user insight into the price and performance of running his/her specific application. The answer to the problem is a benchmarking tool, allowing the user to benchmark his/her own application in the cloud, resulting in a costs and time metric.

By leveraging new technologies and tools, like Docker, we were able to implement deployment and execution of specific applications for benchmarking in a benchmarking tool.

Using a generalization of the different pricing models of cloud providers, the costs of running an application on a particular machine type could be computed.

The presented tool allows users to upload their application, which is benchmarked on all selected machine types in the cloud. After benchmarking, a user is presented with metrics about the running time and costs. A user can use these two metrics to decide which machine type to use for running his/her application.

(41)

References

[1] M.B. Chhetri et al. “Smart Cloud Broker: Finding your home in the clouds”. In:

Automated Software Engineering (ASE), 2013 IEEE/ACM 28th International Con- ference on. Nov. 2013, pp. 698–701. doi: 10.1109/ASE.2013.6693136.

[2] M.B. Chhetri et al. “Smart CloudMonitor - Providing Visibility into Performance of Black-Box Clouds”. In: Cloud Computing (CLOUD), 2014 IEEE 7th International Conference on. June 2014, pp. 777–784. doi: 10.1109/CLOUD.2014.108.

[3] collectd - The systems statistics collection daemon. url: http://collectd.org.

[4] Flask (A Python Microframework). url: flask.pocoo.org.

[5] Enno Folkerts et al. “Benchmarking in the Cloud: What It Should, Can, and Cannot Be”. English. In: Selected Topics in Performance Evaluation and Benchmarking. Ed.

by Raghunath Nambiar and Meikel Poess. Vol. 7755. Lecture Notes in Computer Science. Springer Berlin Heidelberg, 2013, pp. 173–188. isbn: 978-3-642-36726-7. doi:

10.1007/978-3-642-36727-4_12. url: http://dx.doi.org/10.1007/

978-3-642-36727-4_12.

[6] Xinni Ge et al. “Loosely-Coupled Benchmark Framework Automates Performance Modeling on IaaS Clouds”. In: Utility and Cloud Computing (UCC), 2014 IEEE/ACM 7th International Conference on. Dec. 2014, pp. 473–480. doi: 10 . 1109 / UCC . 2014.60.

[7] Alexandru Iosup, Nezih Yigitbasi, and Dick Epema. “On the performance variability of production cloud services”. In: Cluster, Cloud and Grid Computing (CCGrid), 2011 11th IEEE/ACM International Symposium on. IEEE. 2011, pp. 104–113.

[8] Alexandru Iosup et al. “Performance analysis of cloud computing services for many- tasks scientific computing”. In: Parallel and Distributed Systems, IEEE Transactions on 22.6 (2011), pp. 931–945.

[9] Ang Li et al. “CloudCmp: Comparing Public Cloud Providers”. In: Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement. IMC ’10. Mel- bourne, Australia: ACM, 2010, pp. 1–14. isbn: 978-1-4503-0483-2. doi: 10.1145/

1879141 . 1879143. url: http : / / doi . acm . org / 10 . 1145 / 1879141 . 1879143.

[10] Chunjie Luo et al. “CloudRank-D: benchmarking and ranking cloud computing systems for data processing applications”. English. In: Frontiers of Computer Science 6.4 (2012), pp. 347–362. issn: 2095-2228. doi: 10.1007/s11704-012-2118-7.

url: http://dx.doi.org/10.1007/s11704-012-2118-7.

[11] B. Martens, M. Walterbusch, and F. Teuteberg. “Costing of Cloud Computing Ser- vices: A Total Cost of Ownership Approach”. In: System Science (HICSS), 2012 45th Hawaii International Conference on. Jan. 2012, pp. 1563–1572. doi: 10.1109/

HICSS.2012.186.

(42)

[12] R. Prodan, M. Sperk, and S. Ostermann. “Evaluating High-Performance Computing on Google App Engine”. In: Software, IEEE 29.2 (Mar. 2012), pp. 52–58. issn: 0740- 7459. doi: 10.1109/MS.2011.131.

[13] SQLAlchemy 1.0 Documentation. url: http://docs.sqlalchemy.org/en/

rel_1_0/dialects/index.html.

[14] V. Vedam and J. Vemulapati. “Demystifying Cloud Benchmarking Paradigm - An in Depth View”. In: Computer Software and Applications Conference (COMPSAC), 2012 IEEE 36th Annual. July 2012, pp. 416–421. doi: 10.1109/COMPSAC.2012.

61.

[15] John Paul Walters et al. “A comparison of virtualization technologies for HPC”. In:

Advanced Information Networking and Applications, 2008. AINA 2008. 22nd Inter- national Conference on. IEEE. 2008, pp. 861–868.

University of Groningen Bachelor thesis