Indoor Positioning using Location Fingerprinting

(1)

Indoor Positioning using Location

Fingerprinting

Sebastian Hantich

sebastian.hantich@gmail.com

August 26, 2015, 49 pages

Supervisor: Dr. Vadim Zaytsev

Host organisation: bunq B.V.,http://www.bunq.com

Universiteit van Amsterdam

Faculteit der Natuurwetenschappen, Wiskunde en Informatica Master Software Engineering

(2)

3.1.1 Data Collection. . . 13 3.2 Environment . . . 13 3.2.1 Virtual Grids . . . 13 3.2.2 Existing Hardware . . . 14 3.2.3 Test Locations . . . 14 3.3 Experiments. . . 14 3.3.1 Training time . . . 14 3.3.2 Classification Time. . . 14 3.3.3 Collection Time . . . 15 3.3.4 Combination of Signals . . . 15 3.3.5 Accuracy . . . 15 3.3.6 Comparison to GPS . . . 15 3.3.7 Impact of Filtering . . . 16

3.3.8 Impact of Environmental Changes . . . 16

3.3.9 Floor Detection. . . 16 4 Prototype 17 4.1 Requirements . . . 17 4.2 Technologies. . . 18 4.3 Architecture. . . 18 4.3.1 Core . . . 19

(3)

4.3.2 Android . . . 22 4.3.3 Common . . . 23 5 Results 24 5.1 Accuracy . . . 24 5.1.1 Unfiltered Dataset . . . 24 5.1.2 Filtered Dataset . . . 26 5.1.3 Summary . . . 27 5.2 Performance. . . 27 5.2.1 Training Time . . . 27 5.2.2 Classification Time. . . 28

5.2.3 Device Discovery Time. . . 28

5.2.4 Summary . . . 29

5.3 Reliability . . . 29

5.3.1 Impact of Crowded Environments . . . 29

5.3.2 Impact of Heterogeneous Hardware. . . 30

5.3.3 Impact of Network Infrastructure Changes. . . 30

5.3.4 Floor Detection. . . 31 5.3.5 Summary . . . 32 5.4 Data Reference . . . 32 6 Conclusions 33 6.1 Threats to Validity . . . 34 6.2 Future Work . . . 35

6.2.1 Crowd-Sourced Radio Maps . . . 35

6.2.2 Geo-Fenced Environments . . . 35

6.2.3 Magnetic Fields. . . 35

Bibliography 36 A Unfiltered Accuracy 39 A.1 K-Nearest-Neighbour . . . 39

A.2 Naive Bayes . . . 40

A.3 Random Forest . . . 42

B Filtered Accuracy 44 B.1 K-Nearest-Neighbour . . . 44

B.2 Naive Bayes . . . 45

B.3 Random Forest . . . 47

(4)

Abstract

Location based services (LBS) add context-awareness to mobile applications and offer new possibilities for innovative technologies. Accurate location estimation is a key criterion for any LBS system. The Global Positioning System (GPS) provides reliable estimations for outdoor environments but loses accuracy in indoor environments through signal attenuation.

Within this thesis we investigate Location Fingerprinting (LF), a technique for indoor location es-timation. LF leverages existing network infrastructure like Wireless local area networks (WLAN) to record network characteristics that get compared to real-time measurements for location prediction. Network characteristics are typically received signal strengths (RSS) and base station identifiers (BSI). We implement a prototype consisting of two Android applications responsible for data collection and location determination that are connected to a Representational State Transfer (REST) backend responsible for data management and location prediction. We apply three different classification al-gorithms from three open-source machine learning libraries and use the RSS of Wifi, Bluetooth and GSM signals.

Based on predefined experiments we analyse the different implementations of each algorithm towards accuracy, performance and reliability for all possible signal type combinations. We show that the Naive Bayes implementation of the JSAT framework achieves the best overall results and establish that Wifi signals are best suitable for LF.

(5)

Chapter 1

Introduction

position (noun) the place where somebody or something is located or has been put. – Oxford Advanced Learner’s Dictionary

Todays high density of smartphone usage and the possibilities of cloud computing allow for inno-vative technologies and services. Indoor positioning is an increasingly important technology in this field, since it provides context-awareness to applications in indoor environments. Possible use cases for future applications include navigation, context related advertisement and location based payments. All of these require reasonable accuracy, reliability and performance.

The Global Positioning System (GPS) provides these requirements for outdoor environments. How-ever, GPS signals get disrupted through walls, roofs and other environmental properties. Therefore new technologies for Indoor Positioning Systems (IPS) have been proposed [10,16]. The most promis-ing technique due to its high accuracy is the Location Fpromis-ingerprintpromis-ing (LF) approach [16].

Existing solutions for indoor positioning systems in industrial environments require the deployment of additional hardware. LF uses already existing network infrastructures, like Wifi, GSM or Bluetooth which makes it very cost efficient since no additional hardware is needed. To determine a phones position, network characteristics of certain locations called fingerprints get recorded and compared to real-time measurements through classification algorithms. These characteristics are typically base station identifiers and received signal strengths.

1.1 Problem Statement

There are various approaches in the domain of LF. Some use Bluetooth- [23], some use Wifi-signals [1] and some even rely on the magnetic field of the Earth [5]. Yet no approach has reached the ubiquity for indoor positioning that GPS has reached for outdoor positioning. The biggest challenges of LF are its dependence on many physical factors, like attenuation through the human body or unreliable network infrastructures due to constant environmental changes. Even the used hardware influences the received signal strength of a transmitter, which makes it hard to implement a cross-platform solution for all existing Smartphone types.

Further, different implementations of classification algorithms achieve different results regarding accuracy, reliability and performance. In general classification algorithms can be divided into two cat-egories: Deterministic and probabilistic. Each category consists of multiple classification algorithms, like K-Nearest-Neighbour within the former and Naive Bayes within the latter category. Other re-search has shown that probabilistic approaches have a slightly higher accuracy than deterministic ones [16]. Both approaches heavily rely on the quality of their training datasets which rely on reliable and sufficient amounts of data.

(6)

With this master thesis we will investigate the feasibility of three different open-source machine learning libraries that provide multiple algorithms for classification problems. We have chosen three different algorithms from both categories that are supported by all libraries and we will compare them based on predefined experiments. Further, we will examine the influence of filtering as well as the impact of environmental changes.

1.2 Research Questions

By carrying out this project and implementing the different techniques mentioned above, we will explore the feasibility of LF from the perspective of the following research questions:

Q1 Which set-up of framework, algorithm and signal type combination is most suitable for Location Fingerprinting in terms of accuracy, performance and reliability?

Q2 Can we improve accuracy by filtering data?

Q3 Is Location Fingerprinting feasible for commercial applications? SQ1 Do environmental changes influence accuracy and precision? SQ2 Can we detect different floors?

1.3 Solution Outline

Our solution uses the Java open-source libraries WEKA1_{, JavaML}2 _{and JSAT}3_{. We have chosen the}

K-Nearest-Neighbour algorithm for the deterministic approach and Naive Bayes as well as Random Forest for the probabilistic approach. We will implement a proof of concept application consisting of a REST-service and two Android Apps. The REST-service provides access to the training and clas-sification functionalities of the different frameworks and the clients are responsible for collecting data and locating the user. The collected data gets stored in a database next to statistical information, like performance and accuracy gathered during training and classification.

We use Wifi-, Bluetooth- and GSM-signals for our fingerprints and test which combination achieves the best results. For this purpose we will implement an automated test module, that runs all ex-periments with different configurations. Additionally, we check the impact of filtering for Wifi- and Bluetooth-signals. We use a basic approach that filters unknown access points and Bluetooth trans-mitters to receive a reliable infrastructure.

1.4 Research Method

Before attempting to answer the research questions we conducted a literature study on indoor posi-tioning systems and existing LF techniques to get a theoretical overview of currently used approaches within the field of interest. Additionally, we explored the API references of the chosen libraries as a basis for implementation. The results of this literature study are presented in Section1.6.

Prototype. We implemented a prototypical application that uses all three mentioned libraries and algorithms, which serves as the basis for this research.

Empirical Research. We evaluated the results of the experiments using an empirical research method.

1

http://www.cs.waikato.ac.nz/ml/weka/

2_{http://java-ml.sourceforge.net}

(7)

1.5 Contribution

Contribution #1: Comparing open-source libraries. The basis for our research are three dif-ferent open-source libraries that support all three types of algorithms. We compare all of these implementations based on performance, accuracy and reliability.

Contribution #2: Combination of different signal types. Most available research either focus on a single signal type for fingerprint creation [1,18,23] or use a hybrid approach [2] with Wifi and Bluetooth. We haven’t found a source that tests the combination of more than two signals. We tested all possible combinations of the three signal types we are focusing on.

Contribution #3: Filtering on known devices. To reduce noise we filtered the collected finger-print data based on trusted devices that are part of the company network infrastructure. We simulated Bluetooth beacons to achieve a reliable result for all signal types.

1.6 Related Work

There are numerous approaches towards the realisation of indoor positioning systems. One differen-tiation criterion among these approaches is the underlying hardware technology. There are systems based on GPS [24], GSM [18], Bluetooth [23], Wifi [1] and some hybrid approaches that combine Bluetooth and Wifi [2]. In this work, we will combine Wifi, Bluetooth and GSM signals in all possible permutations.

Finkel et. al [9] developed an indoor positioning system based on Wifi signals which got deployed in the Ian Potter Museum of modern Art in Melbourne. The system was implemented as a Client / Server application consisting of an Android App and a REST-service. Their main goal was to compare different prediction and data pre-processing techniques. WASP and Random Forest were the overall best performing algorithms with nearly 90 % accuracy. However, Finkel et. al didn’t use a virtual grid to position their measurement points. Therefore they couldn’t calculate the accuracy in meters. Instead they placed a measurement point in front of every artwork to analyse which piece of art visitors were currently facing. In this case the accuracy percentage describes correctly identified locations. We want to measure the accuracy based on meters and therefore use a virtual grid with equal distances between measurement points.

Zhao et. al [28] also developed an indoor positioning system based on Wifi signals. They applied two different positioning algorithms in their work: K-Nearest-Neighbour and Naive Bayes. They achieved an average accuracy of around 2.4 meters with a precision of around 80% ≤ 3 meters and 95% ≤ 5 meters for both algorithms. The implemented system ran on a Laptop equipped with Win-dows XP and was used for collecting and locating. We will apply a Client / Server architecture with Android-based Clients.

Baniukevic et. al [2] proposed a hybrid indoor positioning system combining Wifi and Bluetooth signals. They achieve and accuracy of 2 – 3 meters. However, Bluetooth is only used to partition the indoor space by applying the proximity method described in Chapter2, thus no signal strengths are recorded. We will apply the location fingerprinting paradigm also to Bluetooth and measure the signal strengths.

Chung et. al [5] suggested a system using location fingerprinting that doesn’t use radio signal strengths but the disturbances of the Earth’s magnetic field. With this they were able to reach an accuracy of 1.64 meters with 90% precision. This shows the flexibility of location fingerprinting and that it is not limited to radio signal strengths. However, within this work we will focus on radio signals.

(8)

Chapter 2

Background

Indoor positioning is a complex engineering problem. Various methods for ambient signal processing are possible [13], including proximity, lateration, angulation, pattern recognition and dead reckoning as shown in Figure2.2. In the following, we will first give an overview about these possibilities and elaborate further on Location Fingerprinting as central method of this thesis.

Figure 2.1: Indoor Positioning Approaches [13]

Proximity methods estimate a position based on a single, fixed transmitter. As soon as a Smart-phone comes close to a transmitter the position will be returned.

Lateration approaches estimate a position based on distance measurements. There are two main types of lateration methods which calculate the absolute and relative distance. The former describes the distance between a Phone and several fixed transmitters, which results in a circle of possible positions around each transmitter. The estimated position then lies on the intersection of these circles. The relative distance method uses the relation between two distances to form hyperbolas of possible positions. The estimated position again lies on the intersection between those.

(9)

Angulation methods use the arrival angle of a signal which gives a straight line of possible posi-tions. Multiple transmitters allow prediction of the current position based on the intersection of possible positions.

Pattern Recognition techniques try to discover patterns in position-related characteristics. These characteristics have to be available in some encoding and need to be mapped to physical posi-tions. Based on known mappings, machine learning techniques can be applied which will return the most likely position.

Dead Reckoning methods track movement through speed, direction and elapsed time. The current position is estimated based on the taken path through an indoor environment.

2.1 Location Fingerprinting

LF can be categorised as a Pattern Recognition approach. This method requires knowledge of position-related characteristics before attempting to predict a location. Therefore LF consists of two phases, denoted as offline and online phase [1,27]. During the offline phase a Radio Map gets created that contains mappings from fingerprints to positions, which get recorded prior to the online phase.

In the field of signal processing, the following signal characteristics have been used [12], individually or combined, to form a fingerprint: Base Station Identifier (BSI), Received Signal Strength (RSS), Signal-to-Noise Ratio (SNR), Link Quality Indicator (LQI), Power Level (PL) and Response Rate (RR). In the context of this thesis we limit our measurements to BSI and RSS for Radio Map creation. Figure2.2illustrates the basic elements needed for LF based on RSS and BSI. An indoor area gets divided into labeled grids where each cell represents a location. Existing network infrastructure, such as Wifi Access Points (AP) are used as transmitters for RSS values. During the offline phase a signal strength vector ~si gets recorded in location i. Each vector contains RSS values for each reachable

AP {si1, si2, si3, si4} in that position. AP’s are identified through their BSI. Finally, all recorded

fingerprints form the Radio Map, exemplary illustrated in Table2.1.

Position AP1 AP2 AP3 AP4

(1,1) -34 -56 -10 -32 (1,2) -23 -12 -9 -67 (1,3) -18 -22 -46 -76 (1,4) -45 -33 -7 -35 (1,5) -31 -29 -38 -46 (1,6) -11 -3 -39 -73 (1,7) -55 -48 -37 -51

Figure 2.2: Basic Elements of analysis based on RSS [14]

Table 2.1: Sample Radio Map

Location coordinates are encoded in cell labels, thus during the online phase we want to predict a label y for a given fingerprint x from all mappings (xi, yi), where i ∈ N > 0. In this case labels

are unordered variables that can be denoted as a class. Therefore location prediction is known as a Classification Problem. To solve this, various location estimation methods can be applied from the field of machine learning, refer to [25] for a list of techniques.

(10)

2.2 Location Estimation Methods

Since location prediction is a Classification Problem, location estimation methods are so called Clas-sifiers that get trained on a given set of training examples with corresponding class labels (the Radio Map). The Classifier then takes an unlabeled example (the fingerprint) and assigns a class to it. For this, Classifier analyse a given example E through a discriminant function f (E) applied to each class. The chosen class Ck is the maximum for which

fk(E) > fi(E) ∀ i 6= k (2.1)

Krishnakumar et al. [14] divide location estimation methods in deterministic and probabilistic ap-proaches, which apply different discriminant functions. Deterministic methods base their position predictions only on values, for example the minimum distance in the signal-strength vector space [1]. Probabilistic approaches focus on discrete probability distribution [21] based on the Radio Map, for instance through Bayesian analysis [27].

For the purpose of location prediction within this thesis, we picked the K-Nearest-Neighbour (KNN) algorithm as deterministic approach and the Naive Bayes (NB) as well as the Random Forest (RF) algorithm as probabilistic approaches. In a variety of researches [1,9,27, 28], these approaches have proven high accuracy within the range of 1 – 3 meters, which we aim to reproduce.

2.2.1 K-Nearest-Neighbour

KNN is one of the most famous algorithms for fingerprint-based location determination [11]. It sim-ply calculates the distance between each RSS vector ~ri present in the Radio Map and the currently

measured RSS fingerprint. The fingerprint vector ~v consists of observed RSS values {x1, x2, ..., xn},

where n is the total number of transmitters. Unheard transmitters are substituted with zeros. A set of K closest neighbours is chosen by calculating the distance d from each vector ~r to ~v. A commonly used distance function for this computation is the Euclidean Distance. We derive the distance to the ith sample vector by 2.2, where xj denotes the jth out of n RSS values within the

vector. di= v u u t n X j=1 (xij− xj)2 (2.2)

After this, a majority vote is used for prediction where the class label with the highest occurrence amongst all closest neighbours gets chosen. Suppose li is the sum of classes from the set of K closest

neighbours, grouped by the class label i. Then the discriminant function can be described as fi(E) = j, where

lj

K = arg maxi

li

K (2.3)

To limit the risk of a draw an odd number should be chosen for K where K ∈ N > 0. Former research [15] has shown that 5 is an ideal number for K which we will therefore apply in our prototype. Additionally, an imbalanced number of samples per class can highly influence KNN, since the majority would likely consist of the highest occurring class. To avoid this, one can either introduce weighing or simply use an even distribution of classes. We have chosen the latter and use a fixed amount of fingerprints per location.

2.2.2 Naive Bayes

Like KNN, Bayesian Classifiers are relatively simple but effective. In general, they assign a class C to a given fingerprint X where the posterior probability P (C|X) is maximum. This yields the discriminant function

(11)

Bayes theorem is used to compute this posterior probability. It combines a class conditional prob-ability distribution (likelihood) P (X|C) of a fingerprint vector X given a class C with the prior class probability P (C) and the prior predictor probability P (X) as follows:

P (Ci|X) =

P (X|Ci)P (Ci)

P (X) (2.5)

Since P (X) is identical for all classes [20] it can be left out, thus the posterior probability can be computed through

P (Ci|X) = P (X|Ci)P (Ci) (2.6)

However, direct estimation of P (X|C) from a given set of training examples is hard when the fea-ture space is high-dimensional. Training such classifiers can be simplified by assuming that feafea-tures are independent given the class [20]. This simplification of a bayesian classifier is called Naive Bayes. It is naive in the sense that the assumption itself seems unrealistic, since in reality attributes are seldom independent [7]. Nevertheless, the Naive Bayes classifier can successfully compete with more sophisticated algorithms [7].

Through this so called class conditional independence assumption the probability distribution can easily be computed based on the Radio Map. A commonly used technique for this is Gaussian distribution [15] which we will also apply in our prototype. The overall likelihood of a class C can be calculated by directly multiplying the likelihoods of all features xj within the fingerprint vector X.

P (X|C) = P (x1|C) × P (x2|C) × ... × P (xn|C) (2.7)

With the naive assumption and Bayes theorem, the discriminant function to compute the posterior probability becomes fi(X) = P (Ci) n Y j=1 P (xj|Ci) (2.8)

from which the maximum is returned as predicted location.

2.2.3 Random Forest

Random Forest is one of the more sophisticated algorithms for location estimation. It creates K deci-sion trees of randomly chosen RSS vectors from the Radio Map, called bootstrap samples. Each tree gets trained on a random subset of features from these samples. To classify an unknown fingerprint each individual tree predicts the class. The results are combined using a technique called bootstrap aggregation or Bagging [3], which is simply a majority vote [9] as in KNN.

Decision trees in general use an induction algorithm for the training phase to partition the features based on a splitting criterion. Each partition averages the contained classes which is the basis for pre-diction. Various kinds of induction algorithms are possible, amongst which ID3 is designed for huge data sets [19]. It is also the standard induction algorithm for decision trees in all three frameworks and will therefore be applied in the prototype.

A single decision tree is likely to have high variance, since it is optimised for the training data. Random Forest tries to reduce the variance through random feature selection and Bagging. Therefore we favoured Random Forest over simple decision trees. However, introducing randomness might have influence on the results, since decision trees will differ after each training. To cope with this, we will measure the differences in classification results for multiple training sessions.

(12)

2.3 Radio Signals

Various kinds of radio signals are applicable for LF, out of which we have chosen WLAN, Bluetooth and GSM for this research. WLAN and GSM have a high coverage in urban environments, whereas Bluetooth is a less frequently used technology for radio networks. However, the new iBeacon standard1 promises a higher density of Bluetooth transmitters in urban environments in the future and makes this technology interesting for LF.

2.3.1 WLAN

We focus on the IEEE 802.11n standard for wireless local area networks (WLAN), since it is the most recent and supports downward compatibility to older standards like 802.11g, 802.11b and 802.11a. It operates in the 2,4 – 2,4835 GHz and 5,15 – 5,725 GHz frequency band and covers a range between 30 – 100 meters.

The standard has a power ratio between 13 – 16 dBm which corresponds to 20 – 40 mW. However, the measured RSS on a phone will be significantly lower through attenuation. Additionally the wireless chip on the phone will have an impact on the RSS value.

2.3.2 Bluetooth

Bluetooth operates in a licence free band between 2,402 GHz and 2,480 GHz, which enables usage without admission all over the world. Licence free ranges sometimes overlap with other bands which results in interference on certain frequencies. Bluetooth transmitters bypass this by dynamically changing frequencies. Though, this leads to an extended device discovery process, since all possible frequencies have to be scanned [4].

The Bluetooth standard is divided into three classes that cover different ranges through varying power ratios as illustrated in Table 2.2. Due to battery drain, smartphones are usually of class 2, whereas iBeacons can be configured for any class.

Class Max. Power Max. Power Ratio Range

Class 1 100 mW 20 dBm ca. 100 m

Class 2 2,5 mW 4 dBm ca. 10 m

Class 3 1 mW 0 dBm ca. 1 m

Table 2.2: Bluetooth Classes

2.3.3 GSM

The Global System for Mobile Communications (GSM) standard is a protocol for second-generation (2G) cellular networks. As of 2014 it has become the global standard for mobile communications2_.

In Europe GSM operates in the 900 MHz or 1800 MHz bands. GSM cell towers control the power output to keep it on a sufficient level and maintain a reasonable signal to noise ratio. The power output ranges between 0 and 39 dBm.

1_{http://www.ibeacon.com/what-is-ibeacon-a-guide-to-beacons/} 2_{https://en.wikipedia.org/wiki/GSM}

(13)

Chapter 3

Experiment Setup

An essential instrument in our effort to answer the research questions described in section1.2, is well defined and executed experiments. The basis for all experiments is a radio frequency map that relates location fingerprints to individual measurement points. Since no suitable frequency maps are publicly available, we will build and use our own. For this, we will use the office hallway as a test environment. Next to content and motivation for each experiment, this chapter will introduce the structure of the radio frequency map and the test environment.

3.1 Radio Frequency Map

The radio frequency map contains location fingerprints for each measurement point. A single location fingerprint consists of multiple mappings from base station identifiers to signal strengths. Figure3.1

illustrates our database design for the frequency map.

Figure 3.1: Database design of the radio frequency map

Location represents a single measurement point in the hallway. Since we also want to compare our results to those of the GPS sensor (see section 3.3.6) latitude and longitude values are stored alongside a label and the floor number. Additionally, accuracy calculations require awareness of the physical distance between measurement points as outlined in section 3.3.5. Therefore we apply the following naming scheme to each label: location name rowNumber columnNumber. The usage of row and column numbers is described in 3.2.1.

Location Fingerprint describes a single feature vector that is mapped to a measurement point. One vector consists of multiple features.

Fingerprint marks a single feature. A feature maps an RSS value to a base station identifier. The base station identifier is contained in the transmitter, thus the fingerprint relates to it.

Transmitter combines information of a single Wifi access point, Bluetooth beacon or GSM cell tower. Each have a Service Set Identifier (SSID) and a Basic Service Set Identifier (BSSID). The former is a self-chosen human-readable string, whereas the latter is a unique hardware address.

(14)

3.1.1 Data Collection

Over a period of six days location fingerprints are collected in each measurement point resulting in 1300 unique location fingerprints, obtained through the Collector App described in section4.3.2. To reduce noise, we held the phone in different orientations, since the human body absorbs radio signals and could potentially influence the outcome of the experiment. All measurements were taken on an HTC Desire 510 Android Smartphone.

3.2 Environment

Our experiments rely on a reasonably large radio frequency map consisting of Wifi-, Bluetooth- and GSM-Signals. Hence, the physical environment needs to provide sufficient coverage of all three signal types. Filtering out unknown transmitters as needed for the experiment in section 3.3.7 requires knowledge of the Wifi- and Bluetooth infrastructure within the office. The known access points and beacons are indicated in Figure3.2and described in section3.2.2.

3.2.1 Virtual Grids

Accurate measurements and a reliable frequency map depend on fixed measurement points through-out the whole hallway. Former researches have chosen scattered spots with varying distances to each other among an environment space [9]. Accuracy calculations require a more structured solution. One possible method is a virtual grid over the indoor space, placing the measurement points at the centre of each cell as illustrated in Figure3.2.

We have chosen a cell-size of 1m2 _{for our virtual grid, since LF won’t achieve centimetre accuracy.}

Physically marking the centre of each cell as measurement point enabled us to record data accurately. To calculate the accuracy we apply the euclidean distance function (3.1), using the row and column numbers of each point P as coordinates for x and y.

d =p(x1− x2)2+ (y1− y2)2 (3.1)

(15)

3.2.2 Existing Hardware

Several Wifi access points are deployed in the office space and plenty GSM cell towers can be reached from within the office, resulting in a satisfactory coverage for these types of radio signals. However, there is no reliable Bluetooth infrastructure since all found devices are movable. Therefore, we will simulate fixed Bluetooth beacons through deploying Smartphones at chosen positions in the hallway, resulting in the following list of known hardware per signal type:

Signal Type Hardware Amount

Wifi Apple AirPort Extreme1 7

Bluetooth Samsung S6500 Galaxy Mini 2 1

Acer Glow E330 1

Sony Xperia Sola 1

GSM Cell Tower 16

Table 3.1: Known Hardware

3.2.3 Test Locations

Next to a reasonably large radio frequency map, the automatic execution of all experiments requires a dataset of sample location fingerprints that are used for classification instead of training. To cope floor detection, environmental changes and heterogenous hardware as needed for the experiments in

3.3.8and3.3.9we collected data with different set-ups. Table 3.2shows the division of this data.

Environment Phone Amount of Samples

Normal HTC Desire 510 100

Normal Nexus 5 10

Different Floor HTC Desire 510 25

Crowded HTC Desire 510 15

150 Table 3.2: Division of test locations

3.3 Experiments

3.3.1 Training time

Due to environmental changes network infrastructures constantly evolve. Especially Wifi access points and Bluetooth beacons get added, replaced or simply removed. An IPS must react to these changes in a dynamic manner to retain its accuracy, which requires constant retraining of classifiers. Training time is therefore a crucial indicator for whether an algorithm is feasible for commercial systems or not. Each algorithm and framework internally stores the training data in a different way, which varies in efficiency and therefore in time. In this experiment we will measure the time needed until a certain classifier is completely trained and ready to use for classification.

3.3.2 Classification Time

Modern, high-scalable web applications request computational expensive data from cloud environ-ments through REST-services [8]. For location prediction that means sending the current location fingerprint to a server for classification and returning the result. From a client perspective the clas-sification time describes the duration from the second a request got send until a response is received. This duration should be reasonably fast for a good user experience and is therefore an important

(16)

indicator for a feasible system.

The response time depends on various factors, i.e. network latency and connection speed [6], which are impossible to mimic nor to predict within this study. Therefore we describe classification time as the duration an algorithm needs to classify an instance solely on the server.

3.3.3 Collection Time

The needed data for training and classification get collected by the individual clients that use an IPS. Combined with the classification time this is major factor for user experience and therefore important for real-time systems. Due to the underlying technologies and protocols of the different signal types we assume that collection times vary per radio signal. In this experiment we measure how long the discovery per radio signal takes on the used phone.

3.3.4 Combination of Signals

We assume that the more different signals we are using the higher the accuracy gets. To prove this assumption we will investigate the accuracy based on different combinations of Wifi, Bluetooth and GSM. This will provide insights on whether it is necessary to collect every kind of radio signal or not. In combination with the collection time of individual signals this experiment helps to decide if the hypothetical improvement in accuracy outweighs the additional collection time. In this experiment we will successively train each classifier with the following combinations of signals and eventually classify the instances of our test locations:

ID Wifi Bluetooth GSM WBG x x x WB x x WG x x BG x x W x B x G x

Table 3.3: Signal Type Combinations

3.3.5 Accuracy

Next to the performance consisting of classification and collection time as mentioned above, context-aware services require a reasonable accuracy. Former research [1,9] achieved an accuracy around 2 - 3 meters using Wifi signals. We assume that we can achieve the same or even a slightly higher accuracy using the signal combination.

3.3.6 Comparison to GPS

Our efforts rely on the assumption that location fingerprinting outperforms GPS in indoor environ-ments. This is the key assumption for the success of any IPS and therefore crucial for the value of this study. GPS describes a position in latitude and longitude values. For an accurate measurement our IPS therefore needs to map indoor locations to these values. We will use the latlong-service2 for the purpose of this mapping.

(17)

3.3.7 Impact of Filtering

Not all captured devices during collection time have a fixed position within the indoor space and therefore produce so called noise. This especially accounts for Bluetooth devices since smartphones, headphones and other devices emit signals that move around in the area of interest. We assume that filtering the collected data on known devices might increase the accuracy and will reduce the noise.

For this experiment we will filter out all unknown Bluetooth beacons and Wifi access points and train the classifiers with the output. We will apply the same filtering to the test location fingerprints and locate them. This enables us to compare the accuracy between filtered and unfiltered data and evaluate the necessity of filtering.

3.3.8 Impact of Environmental Changes

Environmental changes can either be network infrastructure changes or different client hardware, but also physical changes like movement of furniture or different amounts of people within an indoor area. An IPS should not be influenced by these changes and remain its accuracy and performance. This is a very likely scenario in real-world systems. We assume that there will be an impact if indoor conditions change. To prove this we will investigate three types of variations.

Network infrastructure changes. We will simulate this by setting the measured RSS values of certain access points and Bluetooth beacons to 0. This would be the result if an access point gets turned off or removed, respectively a Bluetooth beacon.

Heterogeneous Hardware. To observe the effect of different Smartphone types, we will use two different phones for the online phase of the experiments.

Physical changes. Since it is not possible to move furniture in the office hallway, we will ask em-ployees to squatter all over the hallway and collect location fingerprints used as test locations.

3.3.9 Floor Detection

Additionally to the assumed disadvantages of GPS regarding signal disruption, GPS also lacks the capability to detect altitude. Real IPS’s will be deployed in multi-storey buildings and therefore require floor detection. Next to accuracy and performance this is a crucial feature for commercially used IPS’s and therefore needs to be evaluated.

For this experiment we collected location fingerprints in the hallway one storey below our office floor. The architectural conditions are exactly the same as on our floor and therefore the measurement points of the virtual grid also apply here. Since we are focusing on floor detection in this experiment, we will only capture data at two measurement points. Latitude and longitude values for those are similar as well. We expect to be able to detect the current floor based on the differences in signal strength’s.

(18)

Chapter 4

Prototype

Executing all aforementioned experiments requires the development of a prototype as basis for en-forcement. This chapter will describe the underlying requirements, core implementations and main design decisions successively.

4.1 Requirements

The following list introduces the most important (high level) requirements for our prototype.

Real-world environment

This research focuses on the feasibility of LF in real-world environments. Therefore our proto-type has to adhere to the requirements and constraints of these proto-type of applications in order to produce reliable results. A key criteria in commercial applications is to move computation expensive processes, like classification problems to cloud-like environments and expose them through web-services which can be consumed by any client.

Separation of data collection and phone localisation

Radio frequency map creation and phone localisation are two independent tasks. The former doesn’t involve any end-user interaction and is solely used during the offline phase, whereas the latter is only used during the online phase. This separation needs to be apparent in the implementation.

Flexibility towards different machine learning implementations

Executing experiments with different setups requires a dynamic solution for runtime configu-ration. The system needs to be able to swap the underlying machine learning frameworks and additional parameters, like the currently used algorithm with little effort.

Automatic experiment execution

All experiments will be executed with each framework and each algorithm using different com-binations of signal types as described in section3.3.4. This sums up to 63 unique setups, which makes manual execution extremely inefficient and thus requires an automated solution. Data filtering mechanism

The experiments in section3.3.7and3.3.8depend on a sufficient mechanism to filter, respectively manipulate the raw data.

Statistics reporting

Training-, classification- and collection times as described in sections 3.3.1, 3.3.2 and 3.3.3

are essential statistics for this research. Training- and classification times are gathered during experiment execution, but the collection time per signal type can only be captured during the offline phase of the LF process.

(19)

4.2 Technologies

This research is based on the Waikato Environment for Knowledge Analysis (WEKA)1, the Java Ma-chine Learning Library (Java-ML)2 and the Java Statistical Analysis Tool (JSAT)3 as providers of machine learning functionalities. Each of the mentioned libraries is open-source and written in Java, which makes it the language of choice for the implementation.

As stated in the requirements above, classification tasks should be exposed through web-services and consumed by clients. This type of interaction can be classified as machine-to-machine communi-cation, which is usually realised through Representational state transfer (REST) services. The most commonly used Java framework in this context, even in commercial applications is Spring4_{. Therefore}

we will use this library and its RESTful web service capabilities.

Additionally, the Spring Data project provides a consistent access layer to various data storage technologies, such as relational, non-relational or graph databases. For the purpose of the radio fre-quency map described in section3.1a relational database is sufficient. We have chosen the free DBMS MySQL5as database provider and consequently Spring Data JPA.

Spring depends on many other open-source libraries that need to be present in the classpath of a project. For efficient dependency management we will use Apache Maven6. It automatically resolves and downloads all needed dependencies from its online repositories. Another benefit of Maven is that its repositories contain all of the above mentioned machine learning libraries.

Java web applications need to be executed in so called servlet containers. One of the most popular is Tomcat7 _{by Apache. Since it is free and it nicely integrates with our Eclipse Integrated Development}

Environment (IDE), we have chosen this servlet container.

Next to the server side of the application, our prototype needs two separate Smartphone Apps that exchange data with the REST-service. In favour of a homogenous environment in terms of programming languages we decided to use Android, since it is also Java-based. To simplify the development effort we will use Robospice8 to handle communication with the REST-service and AndroidAnnotations9to ensure clean and readable code.

4.3 Architecture

The architecture of the prototype consists of four main components resulting from the requirements. We have build two separate Android Apps: Locator and Collector. As their names suggest, the former is responsible for locating a phone whereas the latter is used to capture data for the radio frequency map. Next to these main responsibilities both Apps measure and send statistical data along each request to the server.

The server, referred to as Core manages training and classification of each algorithm. It is directly connected to the MySQL database which holds the raw data collected by the clients and is used for the creation of the radio frequency map. Smartphone Apps and server communicate over HTTP. Figure4.1illustrates the connection between all components.

1_{http://www.cs.waikato.ac.nz/ml/weka/} 2_{http://java-ml.sourceforge.net} 3_{https://github.com/EdwardRaff/JSAT/tree/master} 4_{https://spring.io} 5_{http://www.mysql.com} 6_{https://maven.apache.org} 7_{http://tomcat.apache.org} 8_{https://github.com/stephanenicolas/robospice} 9_{http://androidannotations.org/}

(20)

Figure 4.1: Physical Architecture

4.3.1 Core

The Core is divided into three layers: Controller, Business Logic and Data Access. Each layer is further broken down into separate modules with different responsibilities. Figure4.2 illustrates this design, which we have chosen to preserve a clear separation of concerns.

Each Controller represents a REST-Endpoint with a unique URL. Every Endpoint handles requests by extracting possible parameters or data objects and delegating to or using the appropriate module in the lower Business Logic layer. This layer contains the following four modules:

Classification offers access to all machine learning functionalities. Filtering accommodates filtering and data manipulation mechanisms.

Experiments contains functionalities for automatic experiment execution and statistic capturing. Data Services provides a consistent interface to the Data Access layer.

(21)

Classification

Every classifier of each individual framework has to be loaded dynamically to support automatic experiment execution. Therefore instances are created and managed by a ClassificationFactory. It holds an abstract IPSClassifier that contains the explicit Classifier defined in a Config object, which maintains the desired framework and algorithm. The IPSClassifier exposes only two methods train and locate and thus abstracts from the underlying implementation as indicated in Figure4.3. It au-tomatically serialises the explicit Classifier after training and deserialises it when requested without the expensive creation of a new object.

Figure 4.3: Class diagram classifier

Training requires the data to be written in a special file format, which is supported by all frameworks and got invented by WEKA. It is called attribute-relation file format (ARFF10) and is structured in three parts: The relation name, the attributes and the data. The data is separated by commas. Each row represents a feature vector of which the last statement indicates the class value. Each comma separated value is mapped successively to an attribute of the former section as shown in Listing4.1. We wrote our own ARFFHandler to output our database in this format.

1 @RELATION relation name 2

3 @ATTRIBUTE attr1 NUMERIC

6 @ATTRIBUTE class {class1,class2} 7 8 @DATA 9 1.0,2.0,3.0, class1 10 1.1,2.2,3.3, class1 11 0.1,0.2,0.3, class2 12 0.11,0.22,0.33, class2

Listing 4.1: Example ARFF file

(22)

Filtering

Creating individual ARFF-files for each experiment setup requires data filtering based on two param-eters, namely signal type and MAC-address. The former is used to create datasets that only contain specified signal type combinations, whereas the latter is needed to exclude unknown transmitters. It becomes apparent that both filters have to be applied in some experiments, thus they need to be chained. Additionally, we use the filtering mechanism to manipulate RSSI values for environmental change simulation. To achieve this, we applied the Filter Pattern as shown in Figure4.4.

Figure 4.4: Filter Pattern Implementation

As you can see, BaseFilter defines certain hooks for the explicit filters to overwrite. That way it is effortless to add new filters for different purposes. Additionally, defining Filter as an interface enables us to chain various types of implementations through FilterChain.

Experiments

Experiment execution is realised through the ExperimentRunner of the Experiments module. It uses the filtering mechanism and obtains an IPSClassifier through the ClassificationFactory for each exper-iment setup. Additionally, it contains a PerformanceListener interface for statistical data collection. Explicit listener can be registered with each Classifier and get informed on certain events, such as training started. The ExperimentRunner serves as listener itself as indicated in Figure4.5.

Figure 4.5: ExperimentRunner

The ExperimentController triggers the execution by calling the run(...) method of the Experi-mentRunner, which returns an ExperimentResult. These results get returned in JSON format by the REST-Endpoint. Initially we planned to use the Javascript library chart.js11 to generate a visual representation of the results. Due to time limitations we now store the plain data in our database.

(23)

4.3.2 Android

Android comes with an extensive Software Development Kit (SDK) that provides consistent access to various features of a Phone, such as Sensors, UI-elements or Hardware Adapters. Our main inten-tion behind the creainten-tion of Android Apps is the measurement of Wifi-, Bluetooth- and GSM signal strengths. Android handles these signals through so called Managers, which belong to the Hardware Adapter category. To obtain device lists for all signal types, necessary managers are WifiManager, BluetoothAdapter and TelephonyManager.

Device discovery can be time consuming, therefore the Android SDK only allows you to register special Receiver for each signal type. These will be called by the Framework once the asynchronous discovery task finished. An exception from this approach is GSM, since the TelephonyManager doesn’t define any Receiver. To prevent blocking, the Android SDK still restricts you from accessing the GSM devices list within the main thread. Therefore we needed to wrap the call into an AsyncTask also provided by the SDK.

Activity’s handle user interaction in Android. Both Collector and Locator start all discovery tasks through an onClick event of a button within the MainActivity. We defined a Handler per signal type that only exposes a startDiscovery() method and registers the needed receivers, respectively starts an AsyncTask. Once finished, a BroadcastListener receives the complete device list per signal type. A specific implementation is defined within each App. Both wait until all tasks are finished, merge the lists and send the data to a certain Endpoint using Robospice. This design enabled us to deal with an asynchronous and event driven paradigm. An overview is shown in Figure4.6.

(24)

Collector

During the offline phase the Collector App is used to obtain signal strengths for each measurement point. The user has to physically stand on the measurement point and choose the specific label for that location from a dropdown menu. By clicking the Collect button the procedure described above gets started and the user is asked to wait until it is finished while a progress indicator is shown.

The Collector extends the above mentioned design and defines a CollectingController, which im-plements the BroadcastListener interface illustrated in Figure4.7. Once all tasks are finished and the lists got merged, it maps the location fingerprint to the chosen measurement point label and sends it to the server.

REST calls are asynchronous themselves and thus also need event based handling. Robospice de-fines two interfaces that are especially designed for Spring REST-services: SpringAndroidSpiceRequest and RequestListener. The former wraps the request and executes it, whereas the latter gets informed once it receives a response. This leads to the following extension for the Collector App:

Figure 4.7: Extensions within the Collector App

Locator

Locating a phone is separated from the data collection process in an own App. The procedure is similar to the one described above, besides that the user doesn’t have to choose a measurement point label. He will receive the predicted label after submission of a fingerprint to the Locating Endpoint. For this, a LocatingController instead of a CollectingController is defined that creates a LocatingRe-quest.

Next to testing purposes the Locator App has another responsibility, which is sample collection. Therefore we created a configuration screen that enables the user to activate sample collection and set needed parameters, like actual measurement point label, environment type or phone type. If enabled, the server will save the current request as sample location separated from the training data.

4.3.3 Common

All three applications share a Common module that contains commonly used classes and configuration files. Since the Jackson library is used to automatically serialise and deserialise plain old Java Objects (POJO’s) into JSON format these POJO’s are for example located in this module. JSON is the preferred format for communication through the REST-service in our case and therefore needed among all three applications. Additionally, Endpoint addresses are stored here that are used throughout the entire application.

(25)

Chapter 5

Results

Using the prototype described above, we collected 1305 fingerprints containing 32345 RSS values from 183 different transmitters. This data defines our Radio Map and is the basis for all conducted experiments. The results are obtained through the ExperimentRunner described in section 4.3.1. Within this chapter we will present and analyse these results in terms of accuracy, performance and reliability.

5.1 Accuracy

To determine the accuracy, we simply trained a classifier with the Radio Map and predicted the loca-tion of all test localoca-tions. We ran this experiment 63 times with different settings regarding framework, algorithm and signal type combination, first on an unfiltered and afterwards on a filtered dataset (see section3.3.7). Due to the high number of individual results we will only present the best five for the filtered and unfiltered dataset respectively. A complete list is included in AppendixA. We base our ranking on the results for accuracy and precision.

We obtain the accuracy for each test location by computing the Euclidean Distance to the predicted location as described in3.2.1. We refer to a high accuracy if this distance is low. Precision is described as the amount of relevant items. Relevant in our case are predicted locations beneath a certain distance. We want to analyse the precision under a threshold of three and five meters.

5.1.1 Unfiltered Dataset

The unfiltered dataset includes all transmitters. Table5.1illustrates the five experiment settings with the lowest average distance to the predicted location. As you can see, the Random Forest algorithm of the JSAT framework achieves the highest average accuracy of about three meters using all three signal types. It is remarkable that the Random Forest algorithm appears in four out of five settings. It emphasises that this algorithm can handle noise better than other algorithms, probably due to training on randomly chosen feature subsets as explained in Section2.2.3. But this carries the risk of deviating results after retraining. To evaluate this impact, we ran setting A five times and observed a deviation in average accuracy between 0.2 and 0.4 meters.

A closer look to the complete list reveals that the best 20 settings contain the K-Nearest-Neighbour algorithm only twice (as 14th and 20th best) and the Java-ML framework occurs only three times (as 7th, 11th and 18th best). Further, the Naive Bayes implementation of the Java-ML framework takes the last seven ranks regardless of the signal type combination. This is an indicator that the K-Nearest-Neighbour algorithm as well as the Java-ML framework are prone to noisy datasets.

Additionally, it becomes apparent that Wifi signals are part of every signal type combination within the top five. The results show that adding Bluetooth and GSM signals increases the accuracy only

(26)

by orders of magnitude. Also, using only GSM produces unfavourable results for all frameworks and algorithms, showing a maximum average accuracy of 7,2 meters.

ID Accuracy in Meter Framework Algorithm Signal Type Combination A 3,0657492465 JSAT Random Forest Wifi, Bluetooth and GSM

B 3,1632954022 JSAT Naive Bayes Wifi

C 3,1712740113 WEKA Random Forest Wifi, Bluetooth

D 3,190412685 JSAT Random Forest Wifi

E 3,2005336296 WEKA Random Forest Wifi

Table 5.1: Top five experiment settings in terms of highest accuracy for unfiltered data The relatively small difference in average accuracy between the shown configurations requires a closer look into data distributions to evaluate the achieved precision. Figure 5.1 illustrates these through a boxplot for each setting of Table 5.1. You can see that D is more unequally distributed than all other settings and contains the most outliers. Further, the figure reveals that most data points in C lie above its median which will lower the precision.

Figure 5.1: Top five experiment settings in terms of highest accuracy for unfiltered data A detailed overview about the precision is given in Table 5.2 for an accuracy below or equal to three meters and in Table 5.3 for data points with an accuracy below or equal to five meters. You can clearly see that the JSAT framework has the highest precision in both categories for the Random Forest followed by the Naive Bayes algorithm, which mirrors our results for accuracy.

ID Precision FW ALG COM

D 74.50% JSAT RF W A 72.55% JSAT RF WBG B 72.55% JSAT NB W 72.55% Java-ML RF WBG 70.59% JSAT RF WG E 68.63% WEKA RF W 68.63% JSAT NB WB 68.63% JSAT RF WB 66.66% WEKA RF B C 66.66% WEKA RF WB Table 5.2: Precision ≤ 3m

A 90.20% JSAT RF WBG 88.24% JSAT RF WG B 84.31% JSAT NB W D 84.31% JSAT RF W E 84.31% WEKA RF W 84.31% WEKA NB W 84.31% WEKA NB WG C 82.35% WEKA RF WB 82.35% JSAT NB WBG 82.35% Java-ML RF WBG Table 5.3: Precision ≤ 5m

Additionally, it becomes apparent that although C has a higher average accuracy its precision is worst compared to all other settings in Table 5.1. The reason for that is the amount of correctly

(27)

predicted locations which is well above all others and lowers the average accuracy. Therefore it is necessary to analyse the precision next to accuracy.

5.1.2 Filtered Dataset

The filtered dataset consist only of known Wifi Access Points, simulated Bluetooth beacons and GSM towers as described in Section 3.3.7. Table5.4 shows the five experiment settings with the highest average accuracy. Here the dominant signal type combination is Wifi and Bluetooth which achieves a higher accuracy by 0,45 meter compared to the unfiltered dataset. Especially the reduced noise of Bluetooth devices is probably an explanation for this, since the number of Bluetooth transmitters in the dataset dropped from 118 mostly movable to 3 fixed devices. As explained in Section2.3.2these devices emit signals with a limited range of 10 meters. This divides our indoor area in three sections which seems to create a more unique fingerprint in these sections compared to an unfiltered dataset. Like before, Wifi appears in all signal combinations and only JSAT as well as WEKA occur as frameworks. Contrary to the unfiltered dataset the majority of algorithms are Naive Bayes which shows that random feature selection might not be as suitable for a filtered dataset. Additionally, it is worth mentioning that setting D and E of Table5.4appear among the best settings for both datasets.

ID Accuracy in meter Framework Algorithm Signal Type Combination

A 2,5580529548 JSAT Naive Bayes Wifi, Bluetooth

B 2,6106009325 WEKA Naive Bayes Wifi, Bluetooth

C 2,7161193257 JSAT K-Nearest-Neighbour Wifi, Bluetooth

D 2,967654858 WEKA Random Forest Wifi, Bluetooth

E 3,0371010778 JSAT Naive Bayes Wifi

Table 5.4: Top five experiment settings in terms of highest accuracy for filtered data

Figure 5.2 illustrates the data distribution for all settings in Table 5.4. You can clearly see that the number of outliers drastically decreased compared to the unfiltered dataset. Next to this, C has a lower precision than D and E, although it has a higher average accuracy.

Figure 5.2: Top five experiment settings in terms of highest accuracy for filtered data The precision below or equal to three meters is shown in Table5.5, respectively in Table5.6for a precision below or equal to five meters. As you can see, next to highest accuracy, settings A to E also achieve the highest precision for an accuracy up to and including three meters, although in a different order. A, B and D score the exact same precision rate as setting D from Table5.1with 74,50%. Only the precision for five meter accuracy can be slightly increased from 90,2% to 92,16%.

(28)

ID Precision FW ALG COM B 74.50% WEKA NB WB A 74.50% JSAT NB WB D 74.50% WEKA RF WB E 72.55% JSAT NB W C 72.55% JSAT KNN WB 70.59% JSAT NB WBG 68.63% WEKA RF WBG 66.66% WEKA RF W 66.66% WEKA KNN W 66.66% Java-ML RF WB Table 5.5: Precision ≤ 3m

A 92.16% JSAT NB WB B 90.20% WEKA NB WB 90.20% JSAT NB WBG 88.24% JSAT NB WG C 88.24% JSAT KNN WB E 86.27% JSAT NB W D 86.27% WEKA RF WB 86.27% WEKA NB WBG 86.27% WEKA NB W 84.31% WEKA KNN WB Table 5.6: Precision ≤ 5m

5.1.3 Summary

Our experiments have shown that the frameworks JSAT and WEKA achieve the highest scores for accuracy and precision regardless of the dataset. The best performing algorithms are Random Forest and Naive Bayes. The former seems to handle noise better than Naive Bayes, probably due to ran-domly selected features but this causes deviations in results. Naive Bayes on the other hand works slightly better on our filtered dataset by reducing the amount of outliers and thus increasing the precision.

Additionally, the results show the importance of Wifi signals. Every shown signal type combination either contains Wifi or consists only of it. After filtering, Bluetooth was the main driving force for increased average accuracy through elimination of the very high noise ratio for Bluetooth. GSM signals seem to have the least impact on accuracy and precision and perform even worse on their own. This shows that the differences in signal strengths for GSM might not be relevant enough.

5.2 Performance

During experiment execution we have collected performance measurements for training, classification and device discovery times. Due to the results for accuracy we will only present the overall performance for Random Forest and Naive Bayes of the JSAT and WEKA frameworks, next to device discovery times for all signal types.

5.2.1 Training Time

We want to investigate the changes in training times for datasets with increasing sizes. Since we didn’t have the time to run a dedicated experiment with a fixed interval of increasing feature vectors, we will use the fact that different signal type combinations produce datasets of varying sizes. GSM (G) produces the smallest dataset, followed by Wifi (W) and Bluetooth (B). Their respective combinations create larger datasets. Therefore we will observe the differences based on the average training time per algorithm and framework for these combinations.

Figure 5.3illustrates the training times for JSAT and Figure 5.4 for WEKA. You can clearly see that Naive Bayes is superior over Random Forest and has a linear growth for increasing dataset sizes. Also the individual training times are nearly identical for both frameworks.

The Random Forest implementation of WEKA shows an exponential growth for increasing dataset sizes whereas the JSAT implementation exposes a more linear growth. Additionally, JSAT outper-forms WEKA in terms of training times as its curve lies well above JSAT. This shows that Naive Bayes might be the better choice for large training sets, especially if the algorithm gets frequently retrained, for instance to cope with environmental changes.

(29)

Figure 5.3: Training times for JSAT Figure 5.4: Training times for WEKA

5.2.2 Classification Time

Similar to training time, we want to investigate the influence of an increasing dataset size on clas-sification times using the same approach as described above. Figure 5.5 illustrates the results for JSAT and Figure5.6for WEKA. As you can see, classification times lie in the millisecond range and thus are extremely fast. Although the differences are really low, the graphs give the impression that the WEKA Random Forest implementation growths slightly more exponential than the others which seem to grow linear. This might be a problem for extremely large datasets, but can’t be analysed during this thesis.

Figure 5.5: Classification times for JSAT Figure 5.6: Classification times for WEKA

5.2.3 Device Discovery Time

During data collection we measured the discovery time per signal type from discovery start until completion of the device list retrieval. As you can see in Table5.7 GSM performs best, followed by Wifi and Bluetooth. It is striking that the average discovery time for Bluetooth devices lies far above

(30)

those of Wifi and especially GSM. With an average 32.07 seconds, Bluetooth is not feasible for any real-time commercial application. The reason for that might be that Bluetooth has to scan multiple frequencies several times to ensure that all devices are found.

GSM Wifi Bluetooth

0.022 seconds 2.98 seconds 32.07 seconds Table 5.7: Average device discovery time per signal type

However, the new iBeacon1 _{standard by Apple might change that, since these beacons send push}

notifications to phones in proximity. The send packages don’t contain RSS values though, but it is possible to combine the proximity method described in Chapter 2 with our location fingerprinting approach. This would give you the benefits of fixed Bluetooth beacons which can increase the accuracy as shown in Section5.1.2.

5.2.4 Summary

Our experiments have shown that the Naive Bayes implementation of both frameworks is superior in terms of training times. Therefore it should be the algorithm of choice for large datasets in real-time applications with a high need for retraining. Classification times are nearly identical with a slight disadvantage for the Random Forest implementation of WEKA. It gives the impression of exponential growth and would be slower on larger datasets. The most important finding is that Bluetooth discovery is infeasible for commercial applications with an average discovery time above 30 seconds.

5.3 Reliability

Lastly, we want to investigate the reliability towards environmental changes such as crowded indoor areas, heterogeneous phone types and network infrastructure changes. Additionally, we want to ex-plore the capabilities of floor detection.

Again we won’t present results for all possible settings but use the aforementioned insights to fo-cus on the most interesting ones. These are listed in Table 5.8and will be used for all experiments throughout this section.

ID Framework Algorithm Signal Type Combination

A JSAT Naive Bayes Wifi

B JSAT Naive Bayes Wifi, Bluetooth

C WEKA Naive Bayes Wifi, Bluetooth

Table 5.8: Average device discovery time per signal type

5.3.1 Impact of Crowded Environments

For this experiment we asked 21 people to scatter across the office hallway. Due to time limitations during the collection phase the sample size for this experiment is relatively small with only 15 test locations. Although this affects the validity, the achieved results are still an indicator for reliability.

Table 5.9 illustrates the accuracy and precision for the unfiltered dataset and Table 5.10 for the filtered dataset. You can observe a loss of precision below or equal to three for A and C. Additionally, C achieves a lower accuracy for the unfiltered dataset whereas A scores worse for the filtered dataset.

(31)

B roughly maintains its results. We still get the impression that the attenuation of signals through a crowd is rather small and doesn’t have a significant impact on the reliability. This can be due to the small sample size, but most likely the amount of people was just too little for an indoor area of about 96m2 to have a serious impact. Therefore we rate the results of this experiment as an indicator for medium sized crowds.

ID Acc. Prec. ≤ 3 Prec. ≤ 5

A 2.75m 64.29% 92.86%

B 2.92m 71.43% 85.71%

C 3.38m 64.29% 85.71%

Table 5.9: Unfiltered Dataset

A 3.52m 57.14% 92.86%

B 2.83m 71.43% 92.86%

C 2.77m 64.29% 85.71%

Table 5.10: Filtered Dataset

5.3.2 Impact of Heterogeneous Hardware

Within this experiment we used a Nexus 5 phone to collect samples. Again the sample size of 10 test locations is rather small due to time limitations. Nevertheless, Table5.11and Table 5.12emphasise that hardware heterogeneity has a significant impact particularly on C. Similar to the crowded envi-ronment accuracy and precision are higher for the unfiltered dataset and especially for JSAT. This suggests that JSAT can compensate device heterogeneity to a higher extent using a larger dataset.

A 3.57m 50% 80%

B 2.58m 70% 90%

C 7.43m 30% 40%

Table 5.11: Unfiltered Dataset

A 3.55m 60% 80%

B 3.92m 60% 70%

C 5.04m 30% 60%

Table 5.12: Filtered Dataset

5.3.3 Impact of Network Infrastructure Changes

We will simulate network infrastructure changes by successively setting measured RSS values of Wifi and Bluetooth transmitters to zero and training all algorithms of Table5.8with the resulting datasets. This will have the same effect as disabling a Wifi access point or a Bluetooth beacon in reality. Table

5.13indicates the number and type of disabled transmitters.

ID Number and type of disabled transmitters

S1 1 Wifi

S2 2 Wifi

S3 3 Wifi

S4 3 Wifi, 1 Bluetooth

S5 3 Wifi, 2 Bluetooth

Table 5.13: Number of disabled transmitters

The deviations for each case are shown in Figure5.7for the unfiltered dataset and in Figure5.8for the filtered dataset. You can clearly see that the impact of network infrastructure changes are less apparent in larger datasets. If the dataset gets smaller the impact is significantly higher. Here the benefits of multiple signal types are clearly visible. B and C can better compensate the loss of a Wifi access point through additional Bluetooth signals, as long as they are available.

(32)

This suggests that if you have a reasonably large network infrastructure you can easily compensate the loss of a certain percentage of transmitters, which we also expected in beforehand. The signal type combination seems unimportant as long as it increases the overall amount of transmitters in the training samples.

Figure 5.7: Unfiltered Dataset Figure 5.8: Filtered Dataset

5.3.4 Floor Detection

For this experiment we have placed five measurement points and 25 test locations one floor below our office floor. Since this experiment is not about accuracy, we will only focus on correctly detected floors while trying to locate ourselves. We will show this for the Naive Bayes implementation of JSAT. Table5.14illustrates the percentage of correctly detected floors for all signal type combinations.

It is striking that whenever Wifi is involved floors get correctly detected in 100% of the cases. The main reason for this is probably, that very few Bluetooth signals were found and that our fixed beacons and movable devices weren’t reachable. Additionally, GSM has already shown that the differences in signal strengths are too low for a high accuracy and the same applies to floor detection. Wifi access points on the other hand were easily reachable and the differences in signal strength were significant enough. We can conclude that Wifi is the signal type of choice for floor detection.

Framework Algorithm Signal Type Combination Percentage of correctly detected floors

JSAT Naive Bayes Bluetooth 0%

JSAT Naive Bayes Bluetooth, GSM 0%

JSAT Naive Bayes GSM 0%

JSAT Naive Bayes Wifi 100%

JSAT Naive Bayes Wifi, Bluetooth 100%

JSAT Naive Bayes Wifi, Bluetooth, GSM 100%

JSAT Naive Bayes Wifi, GSM 100%

(33)

5.3.5 Summary

We have seen that our best performing frameworks and algorithms are somewhat resistant against environmental changes. Due to small sample sizes we could only get a slight impression about the impact of (medium size) crowded environments and device heterogeneity. We can say that the latter probably has a higher impact than crowded environments especially on the WEKA framework which showed a significant loss of accuracy and precision.

Network infrastructure changes can be compensated by large training datasets. The results have clearly shown the smaller the dataset size, the higher the impact on accuracy. The JSAT framework using only Wifi maintained its accuracy best for the unfiltered dataset and showed the same difficulties as the others for the filtered training samples.

Lastly, the floor detection experiment has clearly demonstrated the importance of Wifi for indoor positioning. Whenever Wifi appeared in a signal type combination the percentage of correctly detected floors reached 100%. This makes Wifi indispensable for location fingerprinting.

5.4 Data Reference

Due to the limited possibility of leveraging the complete results, we published the obtained data at https://github.com/shantich/master-thesis. For privacy reasons we anonymised the data by removing all references to BSSID’s and SSID’s. Further, we used the programming language R from the R-project for statistical computing2to visualize our data and included the needed scripts to create all graphs. However, AppendixAandBpartially present these visualisations. Additionally in AppendixCwe exemplary show an R-script to create a Boxplot.

Indoor Positioning using Location Fingerprinting