Utilizing WebRTC DataChannels in a server-to-peer connection

(1)

Utilizing WebRTC DataChannels

in a server-to-peer connection

July 23, 2018

Student: Hector Stenger 10398872 Supervisor: Adam Belloum Company: Crobox

(2)

Abstract

With the introduction of WebRTC the browser now has a bidirectional com-munication alternative for WebSockets. This technology comes under the name of DataChannel and introduces partial reliable and unordered messaging be-haviour to the browser. While its original intent was to serve messaging in a peer-to-peer fashion it could also utilized in a server-to-peer fashion. It turns out that with the configurable message reliability and ordering DataChannels almost always outperform WebSockets under environmental conditions of in-creased packet loss.

(3)

Acknowledgements

There are three different groups of people I have to acknowledge for the success of this thesis. First of all I would like to express my gratitude towards my col-leagues at Crobox. Sjoerd thanks for the support and the internship possibility at Crobox and Roy & Paul for helping me demystify the different details and abstractions in the WebRTC stack. On an academic level I would like to thank Adam for supporting and helping me during the course of this thesis. Last but definitely not least is the stranger on the internet that was kind enough to answer my questions through email. Jonas, the author of Bitbreeds, was incredibly generous with sharing his WebRTC knowledge. Although my emails weren’t always concise and clear I could expect a detailed answer. Without you this thesis would have been a lot harder.

(4)

1 Introduction

With the introduction of WebRTC the browser environment suddenly has an UDP like protocol available. This functionality under the name of DataChannel was originally intented to serve as a communication layer between different peers. Enabling two peers to directly send and receive data that could be configured either reliable or unreliable and without the usage of an intermediary server. Making this technology available on the server opens up a wide variety of use cases that have not been possible before.

1.1 Problem statement

Most of the communication done by a browser is done in a reliable and ordered way. This ensures that whenever a message is dropped the sender will auto-matically retry the transmission until the receiver acknowledges it received the message. The receiving side will also receive the messages in the same order as they were send. However not all applications favor this prioritization of mes-sage reliability over performance. There is a variety of applications that favor lower message latency over message reliability. Although unreliable protocols, UDP for instance, are available in every operating system this functionality has not been exposed to the browser. This fact makes it impossible for web based application to send and receive data using a protocol where reliability is configurable1_.

1.2 Motivation

Making a reliability configurable protocol available in browsers could lead to the introduction of new browser based applications. This gives developers the tools to configure the message reliability based on their current needs. Applications that could beforehand not be made possible, due to the reliable nature of the TCP protocol that the browser utilizes, are suddenly becoming possible.

This could lower the need for dedicated operating system specific clients because the browser natively supports the required protocols. Also browser ap-plications target a much larger number of users, there is no need to install a client whenever the application is running in the browser. This could result in an application migrating back to being a browser based application rather than being a native operating system client. For other use cases this could mean that WebSockets are directly replaced by DataChannels, potentially gain-ing in performance because DataChannels messaggain-ing behaviour as opposed to WebSockets can be unreliable and unordered.

While these use cases lean heavily on the unreliable aspect a DataChannel has to offer imagine the possibility where reliable DataChannel performance is identical to that of the WebSocket. In such a situation there would be no need anymore for the WebSocket technology. This would make a DataChannel preferable over a WebSocket since it offers everything a DataChannel has to offer and more, besides reliable ordered messaging DataChannels also support unordered and partial reliable or unreliable messaging.

1_{Sending UDP datagrams was possible while using Flash(ActionScript). Since Flash}

how-ever is not supported anymore due to its various security flaws the problem has not yet been solved

(7)

1.3 Research goal

The goal of this thesis is to explore whether DataChannels could potentially replace WebSockets. To determine whether this is the case the following research questions should first be answered:

1. Does a DataChannel suffer from the same throughput problems as Web-Sockets when operating on a connection with increased packet loss. 2. What are the requirements for data carried over a DataChannel to be

faster than a corresponding connection over WebSocket.

3. Can a DataChannel implementation maintain the same amount of active concurrent connections like its WebSocket counterpart.

Another goal of this thesis is to develop a performance testing tool that works in a similar way for both a WebSocket and a DataChannel.

Because SCTP, Stream Transmission Control Protocol, has a TCP, Trans-mission Control Protocol, friendly congestion algorithm[14] reliable perfor-mance should be near identical to that of the WebSocket. Because it has the ability to send data in a partial unreliable and unordered manner it has the potential to outperform a WebSocket when performance is prioritized over mes-sage reliability. This messaging characteristic also makes DataChannels a better choice for a larger variety of application. Rather than only having reliable and in-order messaging, which WebSockets offer, users can configure a DataChan-nel’s ordering and reliability based on their application’s needs. Combining these two facts leads to the hypothesis that DataChannels have the potential to replace WebSocket, because their congestion algorithm is similar but Dat-aChannel behaviour is customizable.

1.4 Related work

There has been a couple of different researches regarding the degradation of TCP over LTE2_{networks. Huang et al[12] discovered that TCP over LTE has different}

inefficiencies such as an undesired slow start. This causes the bandwidth to be under utilized. They conclude that more research is needed in creating LTE friendly transport protocols. Robert[16] studied the effects of different congestion control algorithms being exposed to LTE network conditions. While different variations in congestion control algorithm exists none of them seem to be an optimal solution for the performance loss TCP faces on mobile devices. Lui et al[13] also observe that the different congestion control algorithms used in TCP underperform under different high speed mobile networks like LTE. They propose a protocol optimization devices in order to overcome the under utilization of the available bandwidth.

The performance comparison of SCTP over TCP has also been the subject of different researches. Afzal et al[4] showed that while both TCP and SCTP show similar behaviour on a single flow, SCTP seems to be less affected by the addition of packet loss. Rajesh et al[15] also compare difference in throughput between SCTP and TCP. They conclude with the observation that SCTP can achieve a lower latency and improved throughput whenever the packet loss rate is between 0 and 25%. However they seem to be using the SCTP RFC2960[22], which was marked as obsolete with the introduction of RFC4960[19].

2_{Long Term Evolution is a term used to denote high speed wireless communication for}

(8)

Carullo et al[10] tested the performance of WebRTC over changing LTE conditions. By changing the network conditions throught the NS-3, Network Simulator, tool they measure the influence on throughput and jitter, difference in packet delay, between two different WebRTC clients. The clients in the differ-ent experimdiffer-ents are engaged through a media and audio call, not a datachannel. The paper regarding WebRTC Data Channels that is co-authored by Michael T¨uxen et al[8], also the co-author of a wide variety of Request For Changes (RFC) related to SCTP and DataChannels, gives a very detailed writeup on the current DataChannel specification. They conclude with two different obser-vations that might negatively affect the performance of a DataChannel. Cur-rent buffer display of the DataChannel does not correctly reflect the underlying SCTP send buffer level. Because SRTP, Secure Real-time Transport Protocol which is used in case of audio & video exchange, and SCTP both work through the same network path SCTP could influence the performance of SRTP. These findings are however based on theoretical evidence, the paper did not contain any evidence of an experimental setup. Eskola et al [6] did a research regarding the performance of DataChannels. All experiments performed seemed to be consisting of two different Firefox based peers, no reliability configurations are mentioned throughout the paper. The authors observe that because the SCTP window on both the Firefox and Chrome implementation is not adjusting per-formance is negatively affected.

Besides these different researches there seems to be another push regarding harnessing WebRTC to utilize the peer-to-peer advantages it possesses. Vogt et al[5] have researched whether WebRTC DataChannels could be leveraged for peer-to-peer content distribution in web browsers. They conclude that Dat-aChannels looks very promising but unfortunately still lacks a complete speci-fication. Although Vogt et al describes the usage of NodeJS3 _{and the WebRTC}

implementation contained within the Libjingle library in order to emulate the functionality on the server, no experiment results are discussed nor the usage of different reliability and ordering configurations.

Development wise a handful of different projects are focussing on harnessing the power of a WebRTC DataChannel on the server. Node-webRTC4is a project that uses the Chrome WebRTC runtime implementation5and defines a binding in Javascript. Rawrtc6 is a project developed in C, using the usrsctp7 library for the SCTP logic. OpenWebRTC by Ericsson8 is written in C and seems to be a fully implemented version of WebRTC for the backend, supporting dif-ferent operating systems like OSX & Linux. Unfortunately the project is not maintained anymore, last commit dates back to 18 October 2018. Bitbreeds9

is a Java implementation with a custom written SCTP library. Besides these implementations there seems to be a large share of projects that are mainly focussing on the audio and video exchange aspect of WebRTC.

To conclude there has been done a lot of research regarding improving trans-port layer protocol performance on high speed wireless networks. There however is very limited research available on the performance of SCTP in the context

3_{NodeJS is a framework using Chrome’s V8 Javascript runtime to enable a server to run}

Javascript without the overhead of a browser

4_{https://github.com/js-platform/node-webrtc}

5_{The Chrome runtime WebRTC implementation is a library forked of Alex Gouaillard’s}

library. Alex Gouaillard is one the author of the WebRTC Evolution testing paper[11].

6_{https://github.com/rawrtc/rawrtc} 7_{https://github.com/sctplab/usrsctp}

8_{https://github.com/EricssonResearch/openwebrtc} 9_{https://github.com/IIlllII/bitbreeds-webrtc}

(9)

of a WebRTC DataChannel. Especially in a situation where an unordered and unreliable configured DataChannel is employed in a server-to-peer connection.

1.5 Thesis structure

The structure for this thesis is defined as followed. Chapter 2 will introduce both the WebSocket and DataChannel technology, the technical details regard-ing both implementations and the related work. Chapter 4 will discuss experi-mental setup. Details about how the performance testing tool was written, what kind of architecture was used to collect the data and how different environment parameters were controlled. Chapter 3 will contain a technical write up of how the prototype was made and what kind of technical challenges had to be over-come in order to get a working prototype. Chapter 5 will reflect on the data that was gathered from the experiment and chapter 6 will conclude whether the research questions specified in the introduction were answered. The last chap-ter, chapter 7, contains a summary regarding the different points of discussion that could have affected the outcome of the experiments.

All the different tables that are linked to in chapter 5 can be found in the appendix. The most important tables and graphs will be included inline, directly visible around the text, while the remainders will be added to the appendix section.

(10)

2 Background

2.1 SCTP protocol

SCTP, Stream Control Transmission Protocol, is a message-oriented protocol like UDP that provides both reliable, unreliable, ordered and unordered messag-ing behaviour. Message oriented protocols preserves message boundaries. Each message is send exactly as the protocol servers it. Unlike TCP, a stream-oriented protocol, that might squash different messages together during transmission in order to optimize the connection.

With the addition of RFC3578[20] and RFC7496[24] SCTP supports sending messages in different reliability modes. Besides having the option of sending a message with full reliability, automatic retransmission of lost packets, it can also be configured to behave ’partial’ reliable. The latter means that an end user can define the reliability configuration parameters. These are either specified in milliseconds, being the retransmission window, or in numbers, represented by the total amount of retransmissions before failing to send the message.

As defined in RFC4960[19](Section 6.6) SCTP should support both ordered and unordered delivery of messages. This out of order delivery should enable SCTP to bypass so called head of the line blocking. Head of the line blocking is caused whenever an ordered stream of messages is held up because the first messages needs to be retransmitted. In this situation the queue of messages is blocked by the first message.

SCTP supports multi-homing. This means that multiple paths can be added and used to reach the endpoint. Whenever SCTP detects message fail-ure on one transport address, it can switch over to another active transport address[19](Section 6.4).

Because SCTP is a transport layer protocol it also has to contain path MTU logic. MTU, Maximum Transmission Unit, is the maximum amount of bytes that are able to travel to a destination without a link on that route dropping the message due to its size. Because SCTP can be multi-homed this MTU mapping has to be done for each destination it contains. Whenever SCTP detects a packet with a higher message size than the path MTU it disassembles the packet into smaller fragments, sends those fragments and finally reassembles the packet.

2.2 WebRTC

WebRTC, Web Real-Time Communications, is a combination of existing proto-cols that are introduced to the browser to natively support real time commu-nication between other browsers (peers). Peers should be able to discover and connect directly through different NAT traversal techniques, making intermedi-ate servers obsolete.

The WebRTC draft is defined by the IETF RTCWeb Working Group[3]. The different report release cycles maintained by W3C are[2](Reports Section):

1. Publication of the First Public Working Draft. 2. Publication of zero or more revised Working Drafts. 3. Publication of a Candidate Recommendation (CR). 4. Publication of a Proposed Recommendation (PR). 5. Publication as a W3C Recommendation.

(11)

6. Possibly, Publication as en Edited or Ammended Recommendation. The current status of the document is marked as a publication of a candidate requirements, meaning that some additions are needed before advancing the document to the state of Proposed Recommendation.

Figure 1: The different W3C report cycles and relations among them. According to the candidate recommendation document the following changes

are needed in order to advance this document to the state of Proposed Recommendation[3]: To go into Proposed Recommendation status, the group expects to

demonstrate implementation of each feature in at least two deployed browsers, and at least one implementation of each optional feature. Mandatory feature with only one implementation may be marked as optional in a revised Candidate Recommendation where applicable. Although the specification has not achieved the recommended status by the W3C, yet, different browser vendors have already started to incorporate these functionalities. Adhering to the different API’s that are defined within the document.

Figure 2: WebRTC PeerConnection API supportability among different browsers.

Querying a service like ’caniuse’ 10_{for webrtc supportability is displayed in}

figure 2. From the figure it is visible that most of the major browser vendors included the functionality in their browser. The exception of Internet Explorer, which is only displayed for backwards compatibility reasons as it is discontinued, Opera Mini & UC Browser for Android.

10_{https://caniuse.com/#search=webrtc. caniuse.com maintains a list of all supported}

tech-nology per browser version and vendor. This enables developers to quickly determine whether a drafted technology is supported by a specific browser version without having to test for that functionality themselves.

(12)

The communication done in the WebRTC framework can globally be split into two different groups: media(audo & video) and non-media data[14]. All the data regarding media is sent over SRTP(Secure Real-Time Transport), the non-media data is transmitted over SCTP(Stream Control Transmission Protocol). While SRTP enables peers to send and receive media & audio tracks in a peer-to-peer fashion it is not investigated nor discussed any further in this thesis. The only WebRTC aspect of interest for the scope of this thesis is the Datachannel since that allows to send and receive non-media data in a fashion similar to that of a WebSocket.

2.2.1 Signalling

Signalling is the process of exchanging connection descriptions between two peers. From this connection description it should be clear what kind of media codeces the peer supports, the different ICE candidates and possible precon-ditions the peer has. The IETF specification[18](Page 2) formulates it in the following way:

In this model, one participant in the session generates an SDP mes-sage that constitutes the offer - the set of media streams and codecs the offerer wishes to use, along with the IP addresses and ports the offerer would like to use to receive the media. The offer is conveyed to the other participant, called the answerer. The answerer generates an answer, which is an SDP message that responds to the offer pro-vided by the offerer. The answer has a matching media stream for each stream in the offer, indicating whether the stream is accepted or not, along with the codecs that will be used and the IP addresses and ports that the answerer wants to use to receive media.

The answer that results from this exchange represents a connection condition upon which both sides have consented to. The SDP also contains the public IP of the issuer, so both sides know each others publicly accessible IP after the signalling is done.

While the content of the SDP is fully specified in dept in IETF draft[7], and the usage of SDP in handshaking in spec[18]. specification, the WebRTC draft[3](Introduction Section) formulates nothing about how the SDP should be exchanged between peers. The WebRTC draft[3] states both Websocket and XmlHTTP requests as options, however every method that facilitates reliable transmission could be used.

2.2.2 DataChannel

The bidirectional communication channel that can be opened after negotiating a session between two peers in WebRTC is called a DataChannel. This channel is used to exchange non-media information between peers. What differentiates a DataChannel from WebSocket is that the connection behaviour can be con-figured upon initialization because a DataChannel utilises the SCTP protocol and not TCP. This enables a DataChannel to support functionality like timed retransmission of data and unordered data transmission.

However there are more protocols involved other then SCTP to satisfy the different requirements the DataChannels needs to fulfill. To ensure data confi-dentiality, source authentication and NAT traversal functionality SCTP is build on top of DTLS, Datagram Transport Layer Security, that is on top on a com-bination of ICE and UDP.

(13)

The DTLS layers add encryption and thus provides the source authentication and data confidentiality while ICE enables NAT11_{traversal abilities. Whenever}

an SCTP packet is provided to the DTLS layer, the whole SCTP packet is pro-cessed by the DTLS layers before being passed back to the SCTP[23](Section 3) layer which then transmits it over UDP. The only payload of DTLS ever is SCTP, meaning that binding request produced by ICE are essentially multi-plexed over UDP.

Because the top layer of the protocols are ICE and UDP this means that ICE connectivity checks are send over UDP. The connectivity checks are send in the form of binding requests and have to be answered with a binding response. When these requests are left unanswered the connection closes down in a similar fashion like when a connection timeout occurs.

The DataChannel API is also designed so that it closely mirrors the web-socket API[14](Section 6.4). Because UDP itself employs no path MTU, Max-imum Transmission Unit, functionality SCTP is used to satisfy this role. In contrary to a Websocket a DataChannel connection does not break when an IP address changes during a session[14](Section 5). Similar to that is the con-gestion requirement stated in the WebRTC draft. According to the WebRTC Data Channels draft[14] to channel should employ a “Usage of a TCP-friendly congestion control”.

As soon as the DTLS handshaking is done a SCTP open channel message will be send. In order to open the DataChannel the receiving side has to respond to this open message with an acknowledgement message. These messages are sent over the same stream as the user messages. In this open message the initiating side has also encoded the reliabiblity and ordering parameter of that specific stream that is being opened[1](Section 5.1).

The combination of these different techniques makes it possible to harness the SCTP protocol in the browser. Giving either the developer or the user the ability to tune protocol behaviour to the desired situation.

2.2.3 Interactive Connectivity Establishment

ICE is used as a solution to overcome difficult situations that occur when de-vices are connected through a NAT. A NAT is mostly used in situation where different groups of computers are connected to the internet through a device. This devices, the NAT, is then responsible for redirecting the inward and out-ward traffic to the correct nodes that are residing in its own network. ICE, which mostly consists of two other protocols called STUN (Session Traversal Utilities for NAT) & TURN (Traversal Using Relays Around NAT), is in its purest form an extension of the offer & answer model. Both peers exchange candidates, these candidates are a combination of ip addresses and ports upon which they are accessible[17] to the world outside their NAT. The candidates are then tested for connectivity in a peer-to-peer fashion to see which candidate is suitable for maintaining an active connection. STUN in this case is used as a technique to ask a different server for an accessible IP. The devices in question asks a STUN server outside the NAT to answer the request with the source IP address that issued the request. TURN is essentially the same only it uses an intermediate server, a relay, to send the packets to the receiving side of the channel. It uses this intermediate server when direct communication of nodes behind NATs is not available.

(14)

ICE is used in WebRTC since most of the devices that will be utilizing WebRTC are mobile devices. These devices tend to be behind complex and strict NATs most of the time. Strict means that the NAT has a very strict filtering policy in which traffic it allows and which it blocks. In some cases a connection initiation request will never arrive at the node due to the fact that the NAT automatically blocks requests using protocol x. In order to get these devices connected with each other the ICE protocol is used. Before exchanging SDP a devices gathers ICE candidates. These candidates are the IP address and port which the other peer can use to connect this peer. The ICE candidates could consists of either ’plain’ local ethernet address or an address derived through the usage of the STUN or TURN protocol.

2.3 Websocket

The WebSocket Protocol RFC[9] is a solution to the recurring problem in which a bidirectional communication layer was needed. In some situations a browser and a server need to exchange more than just one or two messages. The old fashioned solution would contain sending plain HTTP requests. This could be done in a long polling manner, meaning the client would keep requesting information while the server only returns the response whenever there was new information, or a regular polling manner, the server would immediately reply even when there was no new information.

Besides the fact that HTTP requests were not designed with this function-ality in mind it adds overhead. Every request contains a lot of HTTP headers, which adds more bytes being send over the wire, that are not critical but can not be left out. As an solution to this a bidirectional communication channel was designed in the form of RFC6455[9]. This enables the developer to setup a connection with the server in which both sides could send and receive messages. Not only would this remove a majority of the overhead of the single HTTP requests, the handshaking and negotiation of the DataChannel has to be done only a single time. As described in the specification “Basically it is intended to be as close to just exposing raw TCP to script as possible given the constraints of the Web”. Because the initialization is done over a regular HTTP requests both HTTP and WebSockets run over the same port while both utilizing TCP. Both fragmentation, path MTU calculation and retransmission are carried out by TCP. This forces the WebSocket to always behave reliable and ordered.

(15)

3 Prototype

3.1 Signalling

Signalling in the prototype is done over a WebSocket. The initiating side, the client, first creates an offer, this also starts the gathering of different ICE can-didates. As soon as the gathering of the ICE candidates is completed they are added to the offer and the offer is then ready to be sent to the server.

Upon arrival at the server the SDP is parsed from the messages. Three different attributes from this SDP are then used to create a response:

• u-frag: Username12

• u-pwd: Password. • mid: Media id.

As soon as the response is ready it is forwarded back to the client. The SDP is then set as the remote description, which finalizes the DataChannel initialization, and an UDP tunnel is opened from the client to the server.

3.2 DTLS

Most of the signalling logic has already been done by a different project called bitbreeds13_{. Bitbreeds uses BouncyCastle}14 _{for a large part of the encryption}

that is used in order to achieve DTLS over UDP. Fortunately a majority of DTLS logic in the bitbreeds project could be re-used in the current prototype.

A certificate on the server is used to create an sha-256 fingerprint. This is also included in the answer that was send back to the client. This fingerprint is then used to initialize DTLS over UDP.

3.3 ICE

The ICE implementation could also mostly be re-used from Bitbreeds. While both peers, server and client, exchange their offer and answer through signalling the connectivity checks are only send by one side. The ICE specification lists three types of candidates[17][Section 2.1]:

1. Host candidate: public address obtained directly from a local interface. 2. Server reflexive candidates.

3. Relayed candidates: throught the usage of TURN servers.

Because the server peer resides on the public network, it is omnipotent of its own IP address and its publically accessible IP address does not change often, a different setup can be used. In this situation an ICE Lite client is used on the server side. This means that the connectivity checks are only send from the client to the server. The server needs to respond to these STUN binding requests in order to keep the connection alive. The full client, the web client node in this case, takes the role of the controlling agent[17][Section 2.7].

While the web client still gathers its ICE candidates from either a STUN or TURN server, the server only sends one candidate, the host candidate, that consists of its own publically accessible IP address and UDP port.

12_{Both the username and password are used by ICE for message integrity.} 13_{https://github.com/IIlllII/bitbreeds-webrtc}

(16)

3.4 SCTP

The SCTP library used in the prototype originates from the libjitsi 15 _team.

Their work regarding SCTP16_{, which is called sctp4j, is mostly done by writing}

a wrapper around a C based library.

How this works is by defining a wrapper file, in java these are called JNI (Java Native Interface), which create a binding between Java and the native library. This enables developers to use a library that is outside their ecosystem with minimal effort while maintaining decent performance.

The wrapped library is usrsctp17_{which is an userland stack implementation}

of SCTP in the C language that runs over UDP and IP18_{. Unfortunately the}

wrapper written by libjitsi does not expose the unreliablity configuration that is defined within the usrsctp library. This means that even though the underlying library supports this functionality there is no way of setting this configuration without altering the JNI.

Through Lennart Grahl19over a github ticket20 directions were received on how to expand the JNI so messages could be send in a partial reliable manner. The original libjitsi repository has been forked21 _{and to this fork the}

ex-tended functionality has been added. RFC6458[21] has been used to determine the different partial reliability parameters and policy values that had to be used. After recompiling the adapted JNI wrapper and rebuilding libjitsi a new func-tion would be visible for the Java client that made use of the newly exposed functionality.

Upon recieving the open message from the initiating side both the reliability and ordering parameter are parsed from the SCTP message. This information is then saved in the object before responding with an acknowledgement. Sending this response will then open the DataChannel.

15_{https://desktop.jitsi.org/Projects/LibJitsi}

16_{https://github.com/jitsi/libjitsi/tree/master/src/org/jitsi/sctp4j} 17_{https://github.com/sctplab/usrsctp}

18_{Surprisingly the main contributor of usrsctp is Micheal Tuexen which is also one of the}

editors for both the IETF WebRTC DataChannels draft [14], the SCTP partial reliablity extension[20] and many more drafts regarding SCTP.

19_{https://github.com/lgrahl}

20_{https://github.com/sctplab/usrsctp/issues/233} 21_{https://github.com/hjastenger/libjitsi}

(17)

4 Experiment

4.1 Performance testing tool

Performance testing was done using Puppeteer. Puppeteer is a Node library that makes it possible to control a Chrome process by using the DevTools22

protocol23_{. Using Chrome has three other advantages:}

• The Javascript runtime in Chrome, V8, is one of the most used on the market24, making it ideally to test since it represents a large share of the market.

• Puppeteer exposes the DOM25 _{making it possible to dynamically change}

variables and execute functions during runtime in the DOM of the web-page.

• Because WebSockets are also specified in a RFC26 _{by the IETF,}

Inter-net Engineering Task Force, this mean that both WebSockets and Dat-aChannels are supported by the V8 runtime. Making it ideally to test both technologies . Using the same runtime for both experiments makes the experiments more reliable, both implementations are run through the same runtime under different conditions. The overhead of for instance Chrome is nullified since both experiments suffer exactly the same from this overhead.

There is however a major disadvantage to using Puppeteer, which boils down to the usage of a headless instance. There are a lot more resources used in comparison to tools that use standalone technology to performance test. While a WebSocket performance tool only needs to open a WebSocket, and thus only need to deploy WebSocket based channel technology, a DataChannel is in need of a runtime because this facilitates the usage of the WebRTC technology.

To successfully and correctly test performance under different traffic condi-tions the tool should have the option for configuring the messaging done over the channel. The tool developed has the following CLI27_{configurations exposed:}

• Interval. Change the time between sending the messages.

• Messages. The total amount of messages that is send over either the WebSocket or the DataChannel.

• Retransmission. In case of the DataChannel the retransmission param-eter should be configurable. This is either doen through specifying the retransmission window or the total amount of retransmissions.

• Concurrency. Specifying the amount of concurrent connections will re-sult in the headless Chrome instance opening up more pages and thus increasing the amount of concurrent active connections that are sending and receiving messages at the same time.

22_{Allows for third party tools to interact with Chrome. Making it possible to debug &}

inspect for instance the DOM.

23_{https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#overview} 24_{http://gs.statcounter.com/browser-market-share}

25_{Document Object Model is the representation of the webpage that makes it possible to}

interact with it.

26_{https://tools.ietf.org/html/rfc6455} 27_{Command Line Interface}

(18)

• Message ordering. The ordering of messages, this is also related to the configuration of a DataChannel. Setting this to true will result in an underlying DataChannel to be configured with in-order messaging. • Repeat. The total amount of runs the experiment has to be done before

exiting. The length of an experiment run can be calculated by multiplying the amount of messages times the interval. To get a higher sample size the repeat parameter enables the developer to repeat the experiment an x amount of times using the same parameters.

• Message size. Calling the library with this parameter adds a certain size to the message that is being send on the given interval.

• Protocol. Specify whether the experiment is using the WebSocket or the DataChannel protocol. This allows Puppeteer to determine which connection is being tested, either a WebSocket or a DataChannel. • url. Points to the url where the web page is found.

• WebSocket url. Specify over which url the WebSocket channel should be opened.

• optional. The optional string is used to give an unique description to a set of experiments.

Before starting the experiment the library initiates a warmup round. For the duration of sixty seconds the library starts a single connection using the specified protocol and message size. This enables the prototyping application to warm the caches and do all the initialization before the actual experiment starts.

With all these different configurations options it should be less time consum-ing to configure different traffic profiles. Should the experiment simulate a slow paced high message size traffic then this could be configured by the CLI, vice vice for a profile regarding fast paced high concurreny traffic.

node index.js -u http://localhost:9000/empty -w ws://localhost:9000/WebSocket –messages=50 –interval=500 – ordered=true –concurrent=1 –protocol=DataChannel –repeat=1 –optional=CLIExample

Figure 3: Example invoking the CLI script from a terminal and passing traffic parameters.

First, depending on the protocol parameter, the connection is initialized. Puppeteer is called with the amount of concurrent connections that it needs to maintain. For every concurrent connection a new page is opened within Puppeteer and this page is pointed to the url specified in the url parameter. As soon as this page is done loading the Javascript code will be loaded by Puppeteer, this contains either the WebSocket or the DataChannel performance testing logic depending on the parameter.

By first loading the page before starting either a WebSocket or a DataChan-nel connection no CORS28 rules are violated. The lower the size of the initial

28_{Cross Origin Resource Sharing by default does not allow a WebSocket to connect to a site}

other than the current site. This prevents bonafide websites from loading malafide code from a different url.

(19)

page the faster the WebSocket or DataChannel logic can be inserted by Pup-peteer. Doing it this way makes it easier to performance test on existing appli-cations, no CORS exceptions should have to be added in order to whitelist a domain.

In the case of DataChannel performance testing the signalling is done over a WebSocket. Before initializing the DataChannel the WebSocket url is pointed to create a signalling channel. After sending the offer and receiving the answer the WebSocket is closed and the DataChannel is initialized with the received SDP, also known as the offer.

As soon as the channel is succesfully initialized it starts to send messages, this is based on the interval en amount that was specified on the CLI. In order to do latency profiling on either the DataChannel or the WebSocket two timestamps are taken during the lifetime of a message. The first timestamp is added right before sending the message, while the second one is created right after receiving the message. Taking the difference between these two timestamps yields the number of milliseconds the round trip took.

After receiving the last message the channel closes and all the message times-tamps are passed back to the Puppeteer scope. One major edge case in this situation is the unreliable nature of the DataChannel. Waiting before all send messages are received can cause a never ending process, there exists a change that messages are lost. In order to fix this a timeout has been added. After receiving a message this timeout is refreshed to a certain value. Whenever a message is lost the process does not keep waiting indefinitely, it waits until the timeout expires before exiting.

This logic ensures that when such a thing happens upon the first message the connection will not immediately timeout. In this case the timeout will reset after receiving the second message. Would such a thing happen on the last message than a maximum of thirty seconds is paused before failing that message. In these cases the duration of the experiment is extended by a maximum of thirty seconds. The number of lost messages is added to the connection meta data and returned to the main thread after exiting.

4.2 Metrics gathering

Two different types of metrics are collected during the experiments. Besides storing system statistics the message latency is stored in a database for fur-ther analysis later on in the process. All the metrics are inserted in a time series based database called InfluxDB29_{. InfluxDB is then visually exposed by}

Grafana30_{. Grafana enables developers to display different data source, like}

In-fluxDB, through a server. This all comes included with tooling like an editor, time window selection and graph styling. By supplying an SQL like language the developer is able to query the datasets for a (sub)set that Grafana than displays in a visual representation.

29_{https://www.influxdata.com/} 30_{https://www.grafana.com/}

(20)

Figure 4: Example of Grafana interface where CPU usage is displayed in a real-time manner

4.2.1 System statistics

Metrics regarding system statistics are collected using the tool called ’collectd’31. Collectd has supported integration options for a wide variety of application, including InfluxDB. Every second all the different metrics data are posted to the InfluxDB server which are than queried and displayed by Grafana.

4.2.2 Connection statistics

The data generated by the different Puppeteer pages are also inserted into In-fluxDB. At the end of every last experiment repeat the resulting data is pro-cessed into an query and posted to InfluxDB. All the different configurations that were supplied when calling the script are also passed into this query, mak-ing it possible to determine what data is related to which experiment. This is an important aspect because not supplying the correct and adequate tags to a batch of insert statements will result in not being able to distinguish between different experiments.

latency,concurrent connections=1,messages=50,interval=500,ordered=true, repeat=1,protocol=datachannel,connection nr=1,optional=CLIExample value=1 1531226479906

Figure 5: Example of inserted query string into InfluxDB

Figure 5 is an example of a query string that contains one datapoint that is inserted into InfluxDB. The last value denotes the timestamp in milliseconds in which the event with the outcome of value 1 happened. The data is then inserted in a table where the attributes that describe the datapoint, prepended with an equals sign, are mapped to a column.

The query string displayed in figure 5 is the result of the script call from figure 3. As visible all the important parameters that are supplied upon invocation are also used to insert the data in InfluxDB.

(21)

Figure 6: Example of Grafana interface where average latency is displayed in a real-time manner

Inserting all the data on the timestamp upon which they occured would make it impossible to plot the different graphs on top of eachother. Unless two different timelines could be shown in the same graph, something which Grafana is not capable of. In order to show different times eries on top of each other all the results are inserted in two different ways.

• Values are inserted on the timestamp upon which they occured. • Values are inserted on a timeshifted timestamp.

The second option is achieved by adding a timestamp of the start of the experiment and choosing a certain point in time. This point in milliseconds is going to be used as the starting point of all the other data points for that experiment. By calculating the point in time of the message, difference between the timestamp of the message and starting point of the experiment, it is known at how much milliseconds into the experiment this message happened. Applying that difference in milliseconds to the chosen point in time shifts the event to a point in the past. Doing that to every point makes all the data fit into the same time frame.

4.3 Adding artificial packet loss

The artificial percentage of packet loss is added to network connection by us-ing a toolus-ing called Netem32_{. Netem is already installed on a large majority}

of Unix distribution. Besides adding packet loss adding an x amount of la-tency in milliseconds is also included in one of the configurations. Using Netem both outbound and inbound network interfaces are subjected to dropping an x percentage of the total packets.

4.4 Infrastructure

The infrastructure used in this thesis can be split into three different nodes: Puppeteer node, Prototyping node & Monitoring node. All these nodes reside in different virtual machines hosted by the same provider in the same datacenter.

(22)

Figure 7: Relations between different nodes in the infrastructure.

4.4.1 Puppeteer node

The Puppeteer has two functions within this infrastructure. Besides running the performance testing tool it has the function to reset and initialize the pro-totyping node after every consecutive experiment. This enables the propro-totyping node to remain in near identical state at the start of every round of experiments. After a round of experiments the Puppeteer node reboots the Prototyping server. One minute later it sets the appropriate levels of packet loss and finally it starts the prototyping application. All this logic resides in different bash files. These bash files also serve as a source of interaction that has taken place between the Puppeteer node and the Prototyping node. Looking back in retrospective at these file will tell whether a command was issued or which parameters were used during a specific rounds of experiments.

4.4.2 Prototyping node

The Prototyping node main functionality is to serve as a node in which the prototype is run. Besides sending health data to the Monitoring node its only interaction is with the Puppeteer node. This interaction involves receiving and sending data back and forth to the Puppeteer node.

Both the Puppeteer and Prototype node reside on the same private network hosted on the same region. Private Networking is a feature introduced by Digital Ocean33 _{so droplets}34 _{that are hosted in the same datacenter can have a direct}

private connection with each other. This greatly reduces latency between nodes and allows for a private network interface to be made on both machines. This network interface is then used for the sole purpose of communicating between nodes.

All the communication that is done between the Prototyping and Puppeteer host is done over this network interface, this also includes all the traffic generated by the performance testing tool. Only applying packet loss to this private networking interface greatly isolates the impact on only that specific part of the architecture, leaving the other network traffic untouched.

33_{Hosting provider on which the nodes are run.} 34_{Virtual Machines}

(23)

4.4.3 Monitoring node

The monitoring node has two different purposes: to serve as a central point for other nodes to post their data and to visualise this data. The data varies from health data to message latency data. By isolating this functionality on a specific node no extra resources are needed on other nodes, which could also affect performance on other nodes. Both InfluxDB and Grafana run on the monitoring node.

(24)

5 Results

All the results collected during the experiments are displayed in a table in the appendix. Every paragraph will contain a reference to the appropriate label in the appendix where the specific table is located.

The results from that specific section are compared to that of the WebSocket and the reliable configured DataChannels. The first section will contain the data from the experiments where the amount of retransmission tries are variable, the second section will contain the data originating from experiments where the retransmission timeout window was changed. Both of these sections will have a sub variance in which either the message interval, message size or concurrency was changed. This will amount to a total of six unique experiment environments. Each of these sections will contain three different groups of packet loss. The only exception to this rule is the experiment in which there was no added packet loss, 0.0%. The reason was that because no packets were dropped there is no motivation to either add a retransmission tries or a retransmission timeout window.

The configurations for ordered unreliable DataChannels is chosen not to be included in the experiments. Early tests showed that in a majority of the cases the reliable ordered configuration was nowhere near as fast as the unordered part. Adding packet loss would only widen this performance gap.

The results will first be evaluated in their own context before changing the scope to a broader view. Combining these findings should answer the ques-tion if there exists a condiques-tion under which the DataChannel achieves a better performance than the WebSocket.

5.1 Retransmission tries configuration

The results in the next section were obtained by changing the amount of retrans-mission tries. Three different thresholds were chosen to divide the experiment groups in: 0, 1 & 2.

Having a 0 as retransmission try configuration means that no extra attempts are performed after the initial attempt has failed. While having 2 as retrans-mission configuration specifies that up to two extra attempts are tried before dropping the packet.

5.1.1 Message interval

Starting with a message interval of 75ms35_{, the table 2 in the appendix on page}

50 shows no clear advantage of using a reliable DataChannel over a WebSocket when no packet loss is added to the environment. In both cases, either reliable or unreliable, the DataChannel has a higher average latency than the WebSocket. Both the average latency and the 95th percentile of the ordered and unordered variant show no improvement over the WebSocket.

Adding packet loss to the environment results in minor improvement in the performance of the DataChannel. Both the unreliable datachannel and WebSocket clearly show performance regression, however it is noticeable that the regression of the WebSocket is worse compared to that of the DataChannel. While the average latency of the WebSocket is still lower, the 95th percentile of both the unreliable DataChannel as the WebSocket are on par. It is safe to say

35_{A message interval of 75 milliseconds means that between each message 75 milliseconds}

(25)

that the performance of a reliable DataChannel is still nowhere near that of the WebSocket.

Further increasing the percentage of packet loss results in an even better sce-nario for the DataChannel. Maintaining a packet loss percentage of 10.0% dis-plays the potential advantage that an unreliable DataChannel possesses. While the performance of the DataChannel is almost identical to that of the previous conditions, packet loss of 5.0%, the performance of the WebSocket has further degraded to a point below the DataChannel.

The average latency per message of a WebSocket connection at this stage is well above that of the DataChannel. This same principles applies to the regres-sion of the 95th percentile. For an interval of 75ms it seems to be the case that increasing the percentage of packet loss causes different regression models for both the WebSocket as the unreliable DataChannel. Adding packet loss results in a higher latency for the WebSocket, while the DataChannel prioritizes latency over packets dropped thus dropping more packets the higher the percentage of packet loss.

However when the data is plotted in a graph a totally different conclusion is drawn from the results.

Figure 8: Difference in message performance between WebSocket and unreliable DataChannel on an interval of 75ms

As visible in figure 8 the performance of both the unreliable DataChan-nel configurations is unprecedented. The average latency of both unreliable DataChannel configurations remain constant throughout the whole experiment. With the exception of the start, the DataChannel configurations show no peri-ods of performance loss like both WebSocket configurations show.

Lowering the interval rate to 125ms shows a similar trend. While the packet loss is increased the performance regression of the WebSocket is worse than the DataChannel. Results are visible in table 3 in the appendix on page 51. While the numbers for the average latency are in favor of the WebSocket, the table has turned when looking at the 95th percentile.

(26)

Figure 9: Difference in message performance between WebSocket and unreliable DataChannel on an interval of 125ms

Plotting the data of the unreliable unordered DataChannels with zero re-transmission tries against the WebSocket under different percentages of packet loss shows a clear distinction in performance, figure 9. While both WebSocket configuration are heavily affected by the packet loss, the DataChannels latency remains constant. No major performance drop is noticed at the start.

Doubling the interval to 250ms results in the table 4 in the appendix on page 51. Again the same trend is visible. While WebSocket performance is superior over an unreliable and reliable DataChannel, adding packet loss degrades the performance to a point below the unreliable unordered DataChannel. In the current situation of 250ms interval that point is already reached after adding the initial 5.0% packet loss. As visible in table 4 both the 95th percentile as the average of the WebSocket are worse than that of unreliable unordered DataChannel.

Figure 10: Difference in message performance between WebSocket and unreli-able DataChannel on an interval of 250ms

(27)

Figure 10 shows the data plotted in a graph. Compared to the previous two graphs, figure 8 and 9, the data originating from the DataChannel seems to be much more volatile in reaction to the added packet loss.

There are two major exceptions visible when comparing the results of 250ms interval, table 4, to 75ms, table 2, and 125ms, table 3:

• The latency regression is far worse for the DataChannel on an interval of 250ms. Under 10.0% packet loss it almost tripled in comparison to the conditions of 5.0% packet loss. In previous results this ratio was atmost a twofold, while now it’s almost a trifold.

• The general performance of both the WebSocket and the DataChannel has been worse in comparison to the previous experiments. In essence this means that increasing the time between messages has a (negative) effect on the latency. Which is counterintuitive, less messages send in the same period should result in less processing power needed which should result in an overall lower latency.

Further doubling that interval rate to 500ms has again a dramatic effect on the overall performance. Both the WebSocket and DataChannel show significant performance drops. In general the performance of the DataChannel is under no condition better than the WebSocket, with exception of the 95th percentile when total percentage of packet loss is 10.0%. The total amount of messages dropped by the unreliable connection under both packet loss conditions is also high. Looking at table 5 in the appendix, more than 75% of the packets is dropped when maintaining zero retransmission tries.

Figure 11 displays that under 5.0% packet loss the unordered unreliable DataChannel configuration performs worse than the WebSocket. Under 5.0% added packet loss the DataChannel configuration performs at certain time points better than the WebSocket, with exception of the start and the end which show two similar spikes.

Changing the interval a last time to 1000ms shows regression according to earlier seen trend. Adding packet loss significantly worsens the performance

(28)

of the WebSocket, while the DataChannel performance stays the same. The performance of the WebSocket already drops below the performance after the first initial addition of 5.0% packet loss.

Plotting the data on a graph, displayed in figure 12 , shows a similar trend as seen in figure 8 and 9. While the performance of the DataChannel stays constant during periods of added packet loss the performance of the WebSocket becomes highly volatile.

Almost all different unreliable DataChannel configurations scored better than the WebSocket parts when the environment suffered from packet loss, with the exception of an interval of 500ms. At 5.0% packet loss the websocket outperformed the DataChannel. It is safe to say that the latency regression in a DataChannel when increasing packet loss is lower than the WebSocket. This phenomena is related to the fact than an unreliable DataChannel favors latency over packet loss, while WebSockets prioritize no packet loss over latency.

A second thing to conclude from the data is that under no conditions reli-able, ordered or unordered, DataChannels scored better than WebSockets while unordered reliable DataChannels in general performed better than ordered Dat-aChannels.

Although unreliable unordered DataChannel performance seems better in overal performance there is a very noticeable difference at the start. All Dat-aChannel graphs suffer from a major performance drop right at the start of the experiment. In most cases, all graphs except for figure 11, this performance degradation lasts till five seconds into the duration of the experiment.

5.1.2 Message size

With no additional added message size the DataChannel performance looks su-perior in both the 5.0 and 10.0 % added packet loss groups. As visible in table 7 the unreliable unordered performance is far better than the WebSocket config-uration. Both the ordered and unordered reliable DataChannel configurations are far less performant than the WebSocket.

(29)

Increasing the message size to 250 bytes appears to negatively affect the DataChannel more than the WebSocket. As visible in table 8 all the different configurations of the DataChannel, whether this is reliable, unreliable, ordered or unordered, score lower than the WebSocket. In addition the amount of pack-ets dropped is unbelievable high. In both packet loss conditions almost 60% of the packets are dropped.

Figure 13: Difference in message performance between WebSocket and unreli-able unordered DataChannel on a message size of 250.

As visible in figure 13 both the zero retransmission tries unreliable unordered DataChannel performs substantially worse than the WebSocket. Both Dat-aChannel graphs show no sign of constant performance

Doubling the message size to 500 bytes as seen in table 9 has a more negative impact in the performance of the WebSocket compared to the DataChannel. Be-sides both connection performances being on par after adding the initial packet loss threshold, unreliable unordered DataChannel performance seems superior when 10.0% is added. Again like previous message size results a large majority of the packets is dropped.

(30)

Figure 14 shows both unreliable unordered DataChannel and WebSocket average message performance in the same graph. It is noticable that the Web-Socket under the condition of 10.0% added packet loss is the least performant configuration. It also shows that despite the unreliable configuration both Dat-aChannels appear to behave more unstable, displaying frequent and higher la-tency drops.

Raising the additional size to 2500 extra bytes per messages and we get a combination of trends, visible in table 10. As for the performance when the initial 5.0% packet loss is added WebSocket performances seems to be identical to that of the unordered reliable DataChannel, with an exception of the 95th percentile which seems to be a tenth of the WebSocket’s 95th percentile. Adding on top of that packet loss to a total of 10.0% and the DataChannel performances on message latency seems superior. Take into consideration that the amount of packet loss is again unbelievable high, a lot higher than the percentage packet loss that was added artificially.

(31)

Looking at the graph in table 15 it remains clear that under 10.0% added packet loss the WebSocket is the least performant while all others, with the exception of the two different spikes both DataChannels show, show identical performance. Like the previous graph, figure 9, it seems that the different Dat-aChannel configurations perform less stable, shows more fluctuations in message latency during the experiments.

Finally doubling the extra bytes for a total of 5000 bytes paints a totally different picture. Looking at table 11 it is visible that almost all the packets get dropped when the packet loss is added. In addition to this, the experiment was retried for a total of ten times before aborting the test runs. In some cases none of the packets that were send made a full trip to the server and back, rendering the experiment useless since no meaningful data could be collected. Again the data confirms the decrease in performance, none of the DataChannel configurations has a latency that is anywhere near that of the WebSocket.

(32)

Combining all the results from the increased size under different retransmis-sion tries condenses the following trends:

• In all cases the unordered DataChannel was faster than the ordered Dat-aChannel. While their performance was identicial when no percentage of packet loss was added, the unordered emerged as the faster configuration when packet loss was added to the environment

• Gradually increasing packet loss causes a factor more packet loss whenever a DataChannel is configured to be unreliable.

• All graphs showed similar behaviour like the experiments done under sec-tion 5.1.1. Performance for unordered unreliable DataChannels is the worst for the first five seconds of the experiments. After this point the message latency seems to even out.

(33)

Figure 17: Difference in unreliable unordered DataChannel performance under 5.0% added packet loss when changing message size

Figure 18: Difference in unreliable unordered DataChannel performance under 10.0% added packet loss when changing message size

Plotting all the different message size experiments grouped by 5.0% added packet loss in the same graph results in figure 17. Displaying in figure 18 the results under 10.0% added packet loss shows a similarity between the two different graphs.

The only condition under which the DataChannel performance seems con-stant in both graph is when there is no added message size. At an added message size of zero both figure 17 and 18 show a horizontal line. It seems to be the case that when the message size grows past a specific point the average latency starts to degrade.

(34)

Figure 19: Outbound and inbound latency differences.

To further investigate this situation an experiment has been setup. Instead of raising both the outbound and inbound packet loss to a certain value on each experiment only one of these is set. This results in either having packet loss on the inbound connection, application to server, or on the outbound connection, server application. Packet loss under these conditions will never be on both sides. If both sides would be performing identical the results should be showing identical numbers.

For the condition of having no additional memory added to the message this is exactly the case. Like figure 19 shows both the inbound and outbound cases display a constant line, stacked on top of eachother at the bottom of the graph. However the graph for the experiment with increased outbound packet loss shows a performance drop right at the start, while the experiments with only inbound packet loss show a constant line. This means that the outbound connection has different messaging behaviour than the inbound and explains all observed performance penalties at the start in the previous graphs.

Environment Size Latency 95th Percentile Packets dropped

Inbound - 2.80 ms 5 ms 91

Outbound - 4.05 ms 4 ms 102

Inbound 250 17.19 ms 8 ms 1299 Outbound 250 5.59 ms 5 ms 1732

Table 1: Outbound and inbound latency difference for varying message sizes. However for the increased message size of 250 bytes a totally different result is visible. The graphs might look similar but the numbers translate different as seen in table 1. While both the inbound and outbound have different perfor-mance for a window of 1.5 seconds right after the start, with increased message size both outbound and inbound seem to degrade in a similar fashion.

It remains unclear what is causing this major performance gap. It could be a variety of reasons like an unoptimized library, or wrongly reacting to different acks by either the sending or receiving side. It is however clear that without these performance differences the test results would show a different outcome, one more in favor of unreliable unordered DataChannels and no peformance

(35)

penalty right at the start. 5.1.3 Concurrency

Increasing the concurrency to two active connections shows good performance on behalf of the DataChannel. As visible in table 12 the average latency and 95th percentile for both unreliable unordered DataChannels under added packet loss are lower than the WebSocket.

Running the experiment with a total of five active concurrent connections shows a similar trend as seen in the experiment with two active concurrent connections. Results are visible in table 13. From the data we can conclude that indeed the DataChannel has a better overall performance when configured unreliable unordered. Both the average performance and 95th percentile score better for the DataChannel.

Increasing the active concurrent connections from two to five does not have a significant effect on the WebSocket results. As visible in table 12 and 13, the average latency and 95th percentile stay almost identical for the WebSocket. A similar fashion is noticeable when comparing the results for the unreliable unordered DataChannel. The results for five active concurrent connections show no significant performance drops in comparison to the result data from the experiment where the active concurent connections was two.

Doubling the concurrency parameter to a total of ten concurrent active con-nections shows no significant regression towards unreliable DataChannel perfor-mance. As visible in table 14 the average message latency has decreased in the 5.0% added packet loss bracket, while increased in the 10.0% packet loss when compared to the conditions in which a total of 5 concurrent connections were active.

The average message latency and 95th percentile in both packet loss scenar-ios of the unreliable unordered 1m retransmission window configuration is still unmet by the WebSocket. While dropping a certain percentage of packets, the DataChannel still manages to achieve a lower latency per message.

Increasing the concurrency in general has no effect in the latency per mes-sage. From these results it shows that the DataChannels scales just as well as the WebSocket does. In all of the different test experiments that were run the unordered reliable DataChannel with a retransmission window of 1ms was proven to achieve the lowest average latency. Sacrificing packets in favor of la-tency gains showed under no circumstances to be excessive or disproportionate amounts.

(36)

Figure 20: Concurrency performance differences between WebSocket and unre-liable unordered DataChannel on 5.0% added packet loss.

Figure 20 shows the difference in message latency on 5.0% added packet loss while concurrency changes. The average latency of the WebSocket looks more volatile and in general higher in comparison than the performance of the unordered unreliable DataChannel.

Figure 21: Concurrency performance differences between WebSocket and unre-liable unordered DataChannel on 10.0% added packet loss.

Plotting the data related to 10.0% added packet loss, visible in figure 21, shows a similar trend like in figure 20 with the exception of one major difference. All the data points show that the unreliable unordered DataChannel in general has a lower message latency, however all DataChannel graphs show a major performance dip right at the start of the connection.

(37)

5.2 Retransmission timeout configuration

The following results were obtained by changing the retransmission timeout on the DataChannel configuration. The experiment results will be displayed in a table similar to that one of the previous result section where the retransmission tries parameter has been changed by the retransmission timeout parameter. The different retransmission timeout parameters used are 1, 10.0 and 1000ms.

Picking 1ms as the lowest retransmission window over 0ms has been a delib-erately choice. Investigating differences between outbound and inbound packet loss performance showed a major latency drop. This means that while the in-bound connection, client to server, interprets the 0ms retransmission timeout configurations as unreliable the outbound, server to client, does the complete opposite, making in fact the connection reliable. It is unsure, even after opening a ticket36 on Github asking whether this behaviour is according to the speci-fication. Setting the retransmission timeout configuration to 1ms results in unreliable behaviour in both directions.

5.2.1 Message interval

Starting with the lowest message interval of 75ms displays no clear advantage of choosing a DataChannel over a WebSocket. None of the different configurations for the DataChannel shows a better performance compared to the WebSocket. Even when the DataChannel is allowed to drop packets the performance is still not equivalent to that of the WebSocket. In the highest added packet loss group an unreliable unordered DataChannel achieves a lower 95th percentile, but this goes accompanied by a fair share of packet loss as visible in table 17.

Figure 22 shows the data from the experiment in a graph. As visible both the unreliable unordered DataChannels seem unaffected by the increased per-centage of packet loss. The only exception to this trend seems to be the start in which the unordered DataChannels emits a major performance dip.

(38)

socket performance however seems to fluctuate more heavily under situations were dropped packets are more common.

Increasing the message interval to 125ms shows a better progression towards DataChannel performance, table 18. For both the 0 and 5.0% packet loss thresh-olds the DataChannel, no matter the configuration, is still not as performant as the WebSocket. Setting the total packet loss to 10.0% shows first signs of a better performance for the DataChannel. In this configuration the unreli-able unordered DataChannel outperforms the WebSocket on both the average latency and 95th percentile.

Figure 23 shows the results for the unreliable unordered DataChannels con-figurations with a retransmission window of 1ms versus WebSockets under added packet loss of either 5.0% or 10.0%. Under these circumstances it is very clear that the performance of both unordered unreliable DataChannel configurations remains unaffected under periods of increased packet loss.

Doubling that message interval to 250ms shows again an improvement for DataChannel performance. In both the added packet loss situations the unreli-able unordered DataChannel is clearly superior over the WebSocket, achieving both a lower average latency and 95th percentile. In table 19 the WebSocket latency compared to that of the unreliable unordered DataChannel is almost a factor four higher in all environments were packet loss is added.

Utilizing WebRTC DataChannels in a server-to-peer connection