Analysing visitor flow using a Bluetooth positioning system

(1)

Bachelor Informatica

Analysing visitor flow using a

Bluetooth positioning system

Pieter van den Ham

June 11, 2019

Supervisor(s): dr. B. Bredeweg & prof. M.E.J. Raijmakers

Inf

orma

tica

—

Universiteit

v

an

Ams

terd

am

(2)

(3)

Abstract

Bluetooth-based positioning systems offer improved accuracy over traditional Wi-Fi-based positioning systems. This paper will compare the status-quo of Bluetooth proximity systems and propose a Bluetooth fingerprinting based system that can be used to analyze participant movement within a public space. The data collected by the system is able to provide metrics such as time spent in a location and movement patterns. Two experiments, both focusing of different aspects of the proposed Bluetooth positioning system, were con-ducted to analyse its performance. Finally, several suggestions will be provided on how this system may be improved.

(4)

(5)

Introduction

Indoor localization is of great interest for many diverse applications, including asset tracking (retail, warehouse), people tracking (airports, shopping malls, healthcare) and indoor navigation. Therefore, several techniques have been developed over the past few decades. These systems use radio frequencies (cellular, Wi-Fi, Bluetooth), ultrasound or infrared [16]. An example of a technology that uses ultrasound and triangulation for localization is the Active Bat system developed by AT&T, which is accurate to 3 centimetres but requires a grid of ultrasound receivers to be set up [24].

Analysis of visitor behaviour in a museum environment has proved to be very valuable to curators and directors. Having a way to visually inspect how visitors react to certain changes in exhibits opens the door to research aimed at improving the overall visitor experience in the museum [26].

Outdoor localization techniques such as GPS, which have been in use for multiple decades, are rendered useless when applied to an indoor environment because of obstacles (roofs, walls, human bodies) and their relative inaccuracy [27].

Ease of installation, cost and the ubiquity of Bluetooth-enabled smartphones (as of 2019 100% of smartphones ship with a Bluetooth receiver [18]) were the primary motivators for choosing a Bluetooth-based system over other technologies. While not as accurate as the Active Bat system, Bluetooth localization systems merely require Bluetooth beacons: compact, standalone devices that broadcast Bluetooth packets at a configurable interval. These packets can be received by a Bluetooth receiver, commonly a smartphone.

Previous research predominantly focuses on analysing and/or improving the status-quo in relatively small, well-controlled environments [11][14]. Little research has been conducted on usage in live, relatively large environments. This research will add to the existing body of knowledge by implementing and analysing a Bluetooth positioning system in a public and noisy environment, addressing the question of how to construct a Bluetooth positioning system so that it is accurate enough for use in visitor flow analysis in a public place.

(8)

(9)

CHAPTER 2

Theoretical background

2.1 Indoor Localization: Wi-Fi to Bluetooth Low Energy

Until Bluetooth 4.0 was introduced in 2010, the de-facto standard for indoor localization was a technique known as “Wi-Fi fingerprinting”, an algorithm that used the received signals and their strengths to map the user to a known, pre-recorded location. While Wi-Fi fingerprinting is more accurate than indoor GPS [15], mainly due to the universality of the technology, its accuracy is suboptimal [11]. Moreover, Wi-Fi fingerprinting suffers from lengthy scan-times and inadequate access to the necessary API on development platforms [11]. However, at the time, Bluetooth did not provide any significant improvement over Wi-Fi fingerprinting. Bluetooth Classic (pre-4.0) has similar drawbacks: its lengthy scan times (sometimes up to 15 seconds, a window in which a person can walk about 30 meters at a leisurely pace) and high battery usage prevented it from being widely adopted as an alternative [11].

Bluetooth 4.0 specifies a subsystem known as Bluetooth Low Energy, a protocol that was built specifically for usage in the context of Internet of Things. It has none of the drawbacks of Bluetooth Classic and Wi-Fi mentioned earlier, and therefore seems to be a promising candidate for indoor localization. Bluetooth Low Energy uses the 2.4GHz band, as does Wi-Fi. Its messages can be classified as either data packages or advertisement packages. Advertisement packages are intentionally compact packages (to save battery usage) that are analogous to a Wi-Fi network broadcasting its Service Set Identifier (more commonly known as SSID, or “network name”). Wi-Fi networks nowadays have over 50 bands, and with a reasonable advertising interval of 10Hz scanning for all in-range Wi-Fi access points can take up to multiple seconds. In contrast, Bluetooth Low Energy has 3 dedicated advertisement channels (channels 37, 38 and 39), which circumvents the necessity to traverse 40 channels and allows for scan times to be primarily limited by the advertising frequency of beacons [11][13].

2.2 Bluetooth positioning systems

Techniques that implement indoor localization using Bluetooth can be divided into proximity-based [4], trilateration-proximity-based [8][23], fingerprint-proximity-based [5][11], or any combination of the afore-mentioned [15].

2.2.1 Proximity-based localization

Proximity-based localization techniques predominantly rely on a dense network of beacons, where the user’s location is derived from the distance to the nearest beacon. Therefore, the accuracy of this technique is directly reliant on the density of the beacon-network. Additionally, the variable nature of Bluetooth radio signals makes it especially hard, albeit not impossible, to convert signal strength into a precise distance[14][30].

(10)

2.2.2 Trilateration-based localization

Trilateration-based localization (figure 2.1) is a technique where the distance to three beacons is used to construct three radii, each centred around a beacon, that will intersect at the approximate location of the user. Similarly to proximity-based localization, it relies on a reasonable estimation of the user’s distance to a beacon based on the received signal strength and is therefore subject to the same accuracy limitations.

Figure 2.1: Trilateration. The device’s position is calculated by measuring its distance to nearby beacons. (Texas Instruments, accessed on 28-05-2019) Retrieved from: https://dev.ti.com/

2.2.3 Fingerprinting-based localization

Similarly to Wi-Fi fingerprinting, Bluetooth fingerprinting works by observing incoming signals and classifying this signal-vector (fingerprint ) to known fingerprints at reference locations. In ideal conditions (low interference, 1 beacon per 30m2), an accuracy of fewer than 2.5 meters can be achieved 90% of the time, which is a significant improvement over Wi-Fi fingerprinting (8.5 meters 95% of the time)[11].

For example, given the following 6-dimensional feature vector consisting of the RSSI (Received Signal Strength Indicator) values received by 6 different beacons:

~

u = h−50, −42, −70, −54, −23, −56i

We can use a machine learning algorithm such that classify(~u) yields a location, ideally alongside a probability, that most likely contains ~u. However, all supervised machine learning algorithms must first be trained using manually-labelled RSSI vectors in order for it to make accurate estimations.

Therefore, fingerprinting-based systems must be split into two distinct phases: a training phase, during which a database is constructed of labelled training data to be fed to a machine learning algorithm (classifier), and a “live” phase, during which that trained classifier is used to provide “live” probability estimations for a given RSSI vector.

It should be noted that while fingerprinting is reasonably accurate, it is also a time-consuming system to implement because of the previously mentioned training phase [11]. Assuming a time window of two seconds per fingerprint, it will take an hour to map a 400m2_{floor consisting of 14}

areas of 30m2_{. Moreover, this database must be reconstructed from the ground up the moment}

a change is made to the environment that may influence the fingerprints, such as the addition or removal of a beacon, a change in transmission power or the addition/removal of obstacles.

Nevertheless, RSSI fingerprinting seems to be the most promising technique because of its high theoretical accuracy [11]. RSSI fingerprinting localization systems differ mainly in what classifiers they use to classify RSSI vectors.

(11)

2.3 Classifiers

The accuracy of the fingerprint-based positioning system is dependent on the classifier used as well as the quality and noisiness of the training data [14][28].

Next, three commonly used classifiers will be discussed: k-Nearest Neighbors, Naive Bayes and Support Vector Machines.

2.3.1 k-Nearest Neighbors

The Nearest Neighbors algorithm is defined as

classify(~u) = arg min

i=1,...,Nk~u − ~xik

i.e. the Euclidean distance between the input vector and each vector in the training set is calculated and the class of the vector that is closest to our input vector is selected. k-Nearest Neighbors is a variation of the above algorithm where the k closest neighbors determine the class of the input vector by majority vote.

One drawback of the k-NN classifier is that each neighbor has an equal vote, e.g. with k = 3, two relatively distant neighbors will always win over a single, very close neighbor. A weighted variation of k-NN addresses this problem by weighting a neighbor’s vote by their distance.

Another drawback of the k-NN classifier is that it is computationally expensive. For each classification task, the whole training set is traversed to locate the nearest neighbors of a given input vector. However, in practice, most k-Nearest Neighbors implementation use search trees to reduce this problem to the O(log n) space.

k-Nearest Neighbors has been applied successfully in numerous Bluetooth positioning sys-tems[2][21].

2.3.2 Naive Bayes Classifier

The Naive Bayes Classifier uses the Bayes theorem to determine P (x | C) (the probability of x belonging to class C):

P (C | x) = P (C)

P (x)P (x | C)

Where P (A) can be any likelihood estimator (commonly used are Gaussian and Bernoulli dis-tributions). This classifier then makes the naive assumption that P (x | C) =Q

iP (xi | C), i.e.

each of the elements of vector x are all independent of each other. Usually, P (x) is left out of the equation, as it merely serves to normalize the result. This simplifies the equation to:

P (x | C) = P (C)Y

i

P (xi| C)

The class that x belongs to is then determined by picking the class for which the probability is deemed to be the highest (the Maximum A Priori estimator).

When applied to fingerprints, this translates to the chance of RSSI-vector x belonging to location C. The naive assumption that this classifier makes poses a problem, as it is unclear whether measured RSSI values are truly independent. Notwithstanding, this classifier makes for a promising alternative to k-NN[19][29].

2.3.3 Support Vector Machine

A Support Vector Machine (figure 2.2) aims to find a hyperplane in an n-dimensional space such that it separates clustered vectors. Therefore, it can be used for classification by utilising a one-vs-one or one-vs-all classification paradigm.

In a one-vs-one SVM, N × 1₂(N − 1) different classifiers are constructed, where N is the number of classes (possible locations). Each classifier is then trained for two different classes and class membership is determined by majority vote.

(12)

Figure 2.2: A 2-dimensional SVM separating two point clusters by a hyperplane (scikit-learn developers, BSD License, accessed 28-05-2019).

Retrieved from: https://scikit-learn.org/stable/modules/svm.html

Support Vector Machines are known to work well in high-dimensional spaces[20]. Support Vector Machines reportedly outperform Neural Networks and achieve similar performance to k-NN when used for RSSI fingerprint classification[28][6]. However, it is computationally expensive to calculate class probabilities, as they require a resource-intensive five-fold cross validation[20].

2.4 Interference and noise

Several studies noted Bluetooth is highly susceptible to signal interference[11][29][14][8][23]. The following factors may impact how RSSI signals behave:

1. Covering the receiver’s antenna (accidentally or purposely) 2. Natural variations in absorption of signals in the environment 3. Nearby human bodies and other obstacles

4. Phone orientation (Angle of Arrival)

5. The orientation at which the beacon is mounted (Angle of Departure) 6. The height at which the beacon is mounted

While factor 5 and 6 are relatively static, and thus should be factored in automatically during the training phase of the classification algorithm, factors 1-4 pose a problem when using classification. As spurious misclassifications must be avoided, there needs to be some sort of noise filter to minimize the impact of these dynamic variables. Kouyoumdjieva et al. propose to smooth RSSI signal variations by applying an exponential moving average for each incoming signal: si= (mi− si−1)w + si−1, where siis the smoothed value to be calculated, mi is an RSSI

value, si−1 is the previously calculated value and w is a weighted constant [14] . The authors

chose w = 2/11 = 0.1818. This value, however, may vary between deployments and should be determined empirically.

(13)

CHAPTER 3

Design & Implementation

3.1 Estimote beacons & Android application

In the current research, Estimote’s Proximity Beacons (figure 3.1) were used as the system’s Bluetooth beacons. Official specifications note that the Proximity Beacons have a battery life of approximately 3 years and a range of up to 70 meters. This will, however, depend on the configuration and environment [22].

Figure 3.1: One Estimote Proximity beacon (first generation, hardware revision D). It weighs around 30g.

These BLE beacons are compact devices that emit Bluetooth packets at a certain interval us-ing a beacon protocol. There are multiple different beacon protocols currently available: Apple’s iBeacon, Google’s EddyStone and Estimote’s Telemetry (among others). The protocols differ in packet format, stored information and complexity. For example, while the EddyStone UID pro-tocol only transfers a unique beacon identifier[9], the Estimote Telemetry propro-tocol additionally broadcasts metrics such as sensor data, battery level and firmware version[10].

These beacons are configurable using a mobile app developed by Estimote and support all of the previously mentioned beacon protocols. The only important factor when choosing a protocol in this case is its maximum configurable broadcast rate. As previous research has shown, a broadcast rate of at least 2Hz (i.e. 2 packets are broadcast each second) is desirable, with diminishing returns on accuracy when that frequency is increased[11]. The system uses the Estimote Location protocol, which allows a broadcast rate of 5Hz to be configured.

One advertisement packet of the Estimote Location protocol contains the following informa-tion:

• byte 0 Estimote Location protocol version number

• byte 1-16 Identifier of the device that advertises the packet (beacon ID). Often represented as a 32 characters long hexadecimal string.

• byte 17 Measured power level of the advertising beacon at 1m. This is a factory-calibrated, read-only constant.

(14)

• byte 18 Bluetooth channel on which this packet was broadcast (37, 38 or 39) Additionally, the following metrics are added by the client device upon packet arrival: • The receiving device’s MAC address

• The Received Signal Strength Indicator (RSSI): an estimated measure of the signal strength that the client device is receiving from a beacon

• Timestamp in nanoseconds when the packet was scanned by the receiving device.

Android introduced support for Bluetooth Low Energy in version 4.3 (API level 18, July 2013) [1]. Built on top of this native support, Estimote provides a free-to-use library, the Estimote Proximity SDK, that handles setting up the Android Bluetooth subsystem and parses incoming packets.

The Android application was intentionally kept light-weight. Only minimal changes are made to the collected data: this allows for analysis of the collected raw data in various ways. It features two different modes that correspond to the “training” and “live” phase of the classifiers. These two modes differ in the way that incoming packets are processed and sent to the central data collection API.

Figure 3.2: Left: the app main screen. Middle: the “training” phase screen. Right: the “live” phase screen (with a device unique identifier at the top).

The “training” phase screen collects fingerprints and uploads them to the central data col-lection API. These fingerprints created by collecting incoming packets for a configurable time window, grouping them by beacon and averaging the RSSI.

The “live” phase screen is even simpler: it merely forwards all received packets to the data collection server without any averaging or processing.

(15)

3.2 Data collection API

A central, public-facing server acts as a central database that allows devices to upload the packets they receive for analysis. It is written in PHP and backed by a MySQL database. The MySQL database has two tables: one that stores fingerprints collected during the training phase (table 3.1), and a table that stores the raw packets collected during the live phase. (table 3.2).

Column Type Comment

id int(11) Auto Increment Row identifier created at datetime Row creation time

location varchar(255) Location (e.g. ”NEMO Science Museum”) section varchar(255) Section (e.g. ”Exhibit A”)

packets json The packets receive during the fingerprint time window Table 3.1: fingerprints table design

Column Type Comment

id int(11) Auto Increment Row identifier

created at datetime Row creation time

device identifier varchar(255) UUID of device that received package

beacon identifier varchar(255) unique identifier of beacon that broadcast this packet packet received at datetime Time at which the packet was received by the device

rssi int(11) RSSI of packet

raw data json Raw packet encoded as a JSON object Table 3.2: telemetry packets table design

The API exposes 5 endpoints:

• GET /api/ping Returns API status.

• POST /api/fingerprints Create new fingerprints. The fingerprints to be created must be provided in the JSON-encoded request body.

• GET /api/fingerprints?location=<location> Retrieve recorded fingerprints for a par-ticular location.

• POST /api/packets Store new packets received during the “Live” phase.

• GET /api/packets?seconds=<n seconds> Retrieve packets of the last N seconds (seconds=infinite will return all stored packets).

3.3 Constructing the classification pipeline

The next concern is building a classifier that can accurately translate RSSI-vectors into locations. Scikit Learn (a Machine Learning Python library; https://www.scikit-learn.org/) provides this functionality in the form of pipelines. A pipeline, in a Scikit Learn context, comprises a series of preprocessing and classification operations, combined into one entity. A pipeline has to be fitted to training data before it can transform unlabelled test data into locations.

A diagram representing the pipeline used for RSSI classification can be found in figure 3.3.

3.3.1 Scaling

Some preprocessing is required before the feature matrix is usable. Both the SVM and KNN classifiers benefit from scaling features before usage. Additionally, as discussed in chapter 2, the two types of missing values need to be encoded into the dataset.

The former can be accomplished by applying min-max normalization column-wise: x0 = x − min(x)

(16)

Min-Max Scaling Missing Indicator Classiﬁer Impute missing values Feature Union Pipeline Feature vector Location

Figure 3.3: The RSSI vector classification pipeline.

where x is the original value, x0 is the new value, mean(x) is the mean of the column, max(x) is the maximum value in the column and min(x) is the minimum value in the column. This transformation will clamp all values in a column to the range [0, 1].

3.3.2 Imputing missing values

Interference, noise or a misaligned time window may cause packets to be missed or dropped altogether, resulting in incomplete fingerprints[11].

This is especially problematic during the training phase, as some algorithms (such as kNN) are not able to handle missing values. It should be noted that there are two reasons that could cause a value to be missing:

• No packet was received during a particular time window because of interference and/or dropped packets, but the beacon is in range, as is evident from other samples, and thus this is an anomaly.

• The beacon is actually out of range

The first case can be solved by applying the previously mentioned exponential moving average. The second case, however, is more complicated in that it is not desirable to lose information that could be utilised by the classifiers. Replacing these missing values by some sort of average would be incorrect, because these missing values possess a distinct meaning (the beacon is not in range vs. no packet was received during this specific time window). Solving this problem requires adding N more binary features to the feature matrix, where N is the original number of features. A value of 1 indicates that at least 1 packet was received by a beacon, and a value of 0 indicates that the beacon is out of range. After that, the missing value can be safely replaced by some constant C (see figure 3.4).

Because all features will be scaled to the [0, 1] range after the previous min-max scaling step, either C = 0 or C = 1 would suffice, since the value of C does not affect the classification outcome due to the added binary columns.

~

u = h−50, −42, NaN, −54, −23, NaNi (3.1) ~v = h−50, −42, C, −54, −23, C, 1, 1, 0, 1, 1, 0i (3.2) Figure 3.4: The fingerprint in 3.1 contains two missing values. To replace these variables with a numeric value without losing information, the vector is extended with 6 binary features indicating whether the value was originally present, resulting in 3.2. Additionally, the NaN (missing) values are replaced by a constant value of C.

(17)

3.3.3 Parameter fitting

Choosing which classifier (kNN, Naive Bayes or SVM) to use is done dynamically. Determining the best classifier for a given dataset will be explained in this section.

The pipeline has several configurable parameters:

• What classifier to use: k-Nearest Neighbors, SVM or Naive Bayes • kNN: k, or the k in k-Nearest

• kNN: whether to use a weighted Nearest Neighbors classifier

• SVM (C-Support Vector Classification): C (penalty parameter or error term), gamma (kernel coefficient)

This gives us the following parameter space: • classifier: [’kNN’, ’SVM’, ’Naive Bayes’] • k: [4, 5, 6, 7, 8, 9, 10]

• weight: [’none’, ’distance’] • C: [1, 10, 100, 1000]

• gamma: [1, 0.1, 0.01, 0.001, 0.0001]

Searching for the optimal parameter configuration is tedious, if not impossible, to do by hand. Therefore Scikit’s parameter grid search functionality (GridSearchCV) is utilised instead.

Pipeline GridSearchCV k-fold CV Parameterizes Veriﬁes Parameter Space Optimal Parameters

Figure 3.5: Nested Cross Validation

GridSearchCV performs a grid search on the parameter space and calculates the parameter-ized pipeline’s accuracy using k-fold cross-validation. In addition, nested cross-validation is used to prevent model bias due to cross-validation being used to both train and validate the model [3][17]. A schematic diagram of this process can be found in figure 3.5.

(18)

(19)

CHAPTER 4

Experiments

A crucial step in validating the system design is performing experiments and analysing the results. Two experiments were conducted: a first, tentative experiment and a second, more thorough experiment. A third experiment that will include test subjects is yet to be performed.

4.1 Experiment #1: Accuracy of classifier on training data

This experiment is designed to verify that the system outlined in the previous chapter is indeed usable in a real environment. As the entire system is highly dependent on the accuracy of the pipeline, the main focus for this initial experiment is collecting a dataset for training and cross validation. Finally, this experiment will attempt to evaluate the different classifiers by comparing their mean accuracy where “accuracy” is defined by the number of correct predictions divided by the total number of predictions.

Classification of unlabelled fingerprints and path visualization will be included in the next experiment.

4.1.1 Method

During this experiment, 9 Estimote beacons were spread across an open area of approximately 200m2_{. This area contains multiple exhibits. Each exhibit within the area defines a “section”:}

a location that can be matched with fingerprints. The sections differ in size and proximity to other sections. Exact placement of the beacons and exhibits can be found in figure 4.1.

The beacons were mounted at a height of 3 meters. A previous study focusing on analysing RSSI interference found that RSSI fluctuations will depend slightly on the height at which a beacon is mounted: the most optimal height being 1 meter. However, that same study also notes that interference caused by human bodies have a significant impact on RSSI readings [14]. By mounting beacons on the ceiling, the system hopes to mitigate the effects of this type of interference when used during opening hours.

Note that exhibits 4, 5, 6, 7 and 12 are relatively close to each other, while exhibit 1 occupies a larger area and is comparatively isolated. Also note that some exhibits have a beacon directly above them, while others do not.

As mentioned earlier, the primary focus is building a dataset that can be used to test the pipeline design and accuracy. To build this dataset, fingerprints were collected over a period of 90 seconds for each section, although there was no timer used. Each fingerprint contains the average RSSI per beacon in range, measured in a time window of 3 seconds. During fingerprinting within a section the receiving device (a Nexus 5X running Android 8.1) was held in a static position, in the same manner as a phone is held when in use.

After the collection phase, the fingerprints within a single section are smoothed by applying an exponential moving average (w = 2/11) and thereby exploiting the time-interdependency of the fingerprints to eliminate noise.

(20)

Figure 4.1: Beacon placement. Each red square represents a Bluetooth beacon. Green circles indicate an exhibition (section). 1. colour filters 2. colour mixers 3. drop stopper 4. static table 5. lightning spiral 6. ball blower 7. jumping ring 8. spinning liquid 9. coin crush 10. pulley seats 11. vortex 12. lightning globe 13. spinning magnet

With extreme variance removed from the dataset, GridSearchCV is run with the parameter space defined in section 3.3.3. The resulting pipeline is then scored by performing k-fold cross validation (k = 5).

Finally, to analyse what beacons contribute the most to the classifier accuracy (feature im-portance), the algorithm proposed by Fisher et al. can be used [12]:

• Determine the base accuracy of the classifier accorig

• for each feature in the feature matrix:

– Permutate all values in the feature column. This will cause the feature to be “discon-nected” from the other features.

– Calculate the new accuracy anew

– Calculate the feature importance i = accnew/accorig

• Sort features by descending importance

4.1.2 Results

During the experiment a total of 393 samples (fingerprints) were collected. As was expected based on previous studies [11][14][7], the RSSI fluctuates quite a bit with an average variance of 25 (σ = 4.9, µ = −69), even though the receiving device is held still. By applying an exponential

(21)

0

50 Time (s)

80

60

40 RSSI

0

50 Time (s)

80

60

40 RSSI

0

50 Time (s)

80

60

40 RSSI

0

50 Time (s)

80

60

40 RSSI

0

50 Time (s)

80

60

40 RSSI

0

50 Time (s)

80

60

40 RSSI

0

50 Time (s)

80

60

40 RSSI

0

50 Time (s)

80

60

40 RSSI

0

50 Time (s)

80

60

40 RSSI

0

50 Time (s)

80

60

40 RSSI

Figure 4.2: Recorded RSSI values for the “jumping ring” section over time. The top figure is the combined signal fingerprint plotted over time, composed of the bottom 9 figures. For each bottom figure, the grey line represents the original raw RSSI, while the coloured line is the result of applying an exponential moving average (w = 2/11). Each vertical line indicates the time at which a fingerprint was recorded.

configured classifier. Both the SVM and weighted kNN are given an average accuracy of 95.2%. It should be kept in mind, however, that the size of the dataset (393 samples) is comparatively small. Therefore, results in this table must be treated as a rough estimate. The standard errors reflect this sentiment.

The accuracy differs per section as well. Table 4.2 contains the accuracy per section. The weighted kNN classifier was used to generate this report.

Applying the algorithm proposed by [12] shows that beacons contribute almost equally to the classification process. An overview of importance per beacon is shown in table 4.3. Beacon 46A contributes the least, but still a significant portion.

4.1.3 Discussion

The classifiers performed very similarly on the dataset, as is evident from 4.1, with the Gaussian Naive Bayes classifier being an exception. The probabilistic approach this classifier uses does not seem to fit the dataset as much as the other classifiers, although the Naive Bayes classifier still achieves an accuracy of 90%. A possible explanation is that the probability distribution of RSSI signals is not Gaussian.

The accuracy of 95% of the other classifiers seems promising. It can be safely concluded that the pipeline is indeed able to correctly classify the dataset collected during the training phase.

(22)

Classifier Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Avg. Std. dev. SE

SVM 85.2% 100% 100% 100% 90.7% 95.2% 6.16% 0.31%

kNN 85.2% 95% 97.5% 98.7% 97.3% 94.7% 4.93% 0.25%

kNN (weighted) 85.2% 96.3% 98.7% 98.7% 97.3% 95.2% 5.11% 0.26% Gaussian NB 79.0% 95% 96.2% 97.4% 81.3% 89.8% 7.93% 0.40% Table 4.1: Cross validation performed on 4 different classifiers. For the SVM, C=10, Gamma=1 and kernel=rbf. For both kNN classifiers, k = 5. Fold 1-5 indicates the accuracy of the kth fold performed by the k-fold cross validation algorithm.

Section Fingerprints Precision Recall F1-score

1 colour filters 30 0.97 1.00 0.98 2 colour mixer 30 1.00 1.00 1.00 3 drop stopper 30 1.00 0.90 0.95 4 static table 29 0.92 0.76 0.83 5 lightning spiral 24 0.77 0.96 0.85 6 ball blower 33 1.00 1.00 1.00 7 jumping ring 30 0.80 0.93 0.86 8 spinning liquid 32 1.00 0.94 0.97 9 coin crush 30 0.97 1.00 0.98 10 pulley seats 30 1.00 1.00 1.00 11 vortex 34 1.00 1.00 1.00 12 lightning globe 30 0.96 0.87 0.91 13 spinning magnet 31 1.00 1.00 1.00

Total: 393 Avg.: 0.95 Avg.: 0.95 Avg.: 0.95

Table 4.2: Statistics on accuracy per section. Precision is defined as tp/(tp+fp) where tp is the number of true positives and fp is the number of false positives. Recall is defined as tp/(tp + fn) where fn equals false negatives. The F1-score is a weighted average of the precision and recall.

Beacon identifier Importance

46A 0.906 A34 0.936 E16 0.940 013 0.949 1D2 0.959 CF1 0.960 D36 0.962 A95 0.964 0CD 0.967

Table 4.3: Feature importance: the impact of a single beacon on the accuracy of location classi-fications. The beacon identifier displayed here is the first 3 characters of the 32 characters long

(23)

While the accuracy of 95% refers to the accuracy on average, the accuracy fluctuates per section as well. The classifier achieves a significantly lower accuracy on sections that are relatively clustered together, such as section 4, 5, 6, 7 and 12. This is a difficult to solve issue, as the natural RSSI fluctuations will cause common misclassifications of these sections specifically.

However, it must be noted that it is still unclear how the classifiers will perform on live data. There are a few differences between the dataset collected during this experiment and data collected during the live phase. For example, the training dataset was collected using a device that was held in a static position, while it is likely a visitor would be moving around constantly. It is hard to collect a training dataset that is a true representation of how a visitor would behave, since the dataset must be labelled manually, and tagging the data with the correct section is impossible to do after the experiment.

The feature importance analysis offers an interesting insight into how much a beacon influ-ences the classification process. It can be used to determine which beacons could be relocated for optimal distribution. During this experiment, the exact names and locations of the beacons were not tracked, therefore the location of the least important beacon cannot be pinpointed. This would be an interesting analytic to apply in the real world, however.

In conclusion, this experiment proves that the system proposed in chapter 3 does indeed work for classifying fingerprints collected during the training phase. The next step is to verify the accuracy of the system when applied to a real visitor path.

4.2 Experiment #2: Accuracy of classifier on live data

This second experiment focuses on analysing the performance of the pipeline when used to classify unlabelled, live data.

4.2.1 Method

The setting is largely the same as in experiment one, with a few exceptions. Firstly, the beacon placement has been changed to a more grid-like distribution (see figure 4.3) to allow for more space in between beacons. 14 sections were mapped compared to previous experiment’s 13 sections. On average, the distance between a beacon and its direct neighbors was 7 meters.

For each section, 180 seconds were spent collecting Bluetooth packets.

After collecting a training dataset, a test subject was given a device with the “live” phase data collection screen active. The test subject then proceeded to interact with several exhibitions while keeping the device in her hand. The exact path taken by the test subject can be seen in figure 4.6. This path was then compared to the predicted path as generated by the classifier. As a final step, the predicted path was smoothed by removing any spurious misclassifications on the assumption that the visitor would stay at one exhibit for a duration of at least 10 seconds. The value of “10 seconds” is a variable that can and should be changed based on empirical knowledge. For the sake of this experiment, 10 seconds was deemed to be a good compromise because the actual path of the test participant including time duration spent at each section was known.

4.2.2 Results

During the experiment, 27372 packets were collected in total. From those packets, 1170 fin-gerprints could be constructed. This is significantly less than the expected 14 × 180 = 2520 fingerprints. It was observed that packets were received in bursts at a random interval, instead of a steady stream. This behaviour was unnoticed in the previous experiment, but it is reason-able to assume that this phenomenon might have affected the previous experiment as well, as the method has not changed in between. A visualization of this can be found in figure 4.4. Any time window in between bursts would generate a unusable fingerprint, because no packets would be registered.

Another interesting observation is that while 9 beacons were mounted during this experiment, all fingerprints only contained the packets of 8 beacons at most. This suggests that the battery of one beacon had died during the experiment.

(24)

Figure 4.3: Beacon placement for the second experiment. Each red square represents a Bluetooth beacon. Green circles indicate an exhibition (section). 1. colour filters 2. colour mixers 3. drop stopper 4. static table 5. lightning spiral 6. ball blower 7. jumping ring 8. coin crush 9. vortex 10. chaotic table 11. lightning globe 12. spinning chair 13. spinning magnet 14. listening dishes

It is interesting to see whether changing the locations of the beacons to a more evenly-spaced “grid-like” distribution affects the feature importance of the beacons as well. The feature importance of each beacon during this experiment is given in table 4.4.

Beacon identifier Importance

0CD 0.959 46A 0.972 A95 0.973 E16 0.974 013 0.980 1D2 0.982 A34 0.988 CF1 0.991

Table 4.4: Feature importance of the beacons during experiment #2.

Moving on to the unlabelled data collected by the test subject, figure 4.5 suggests that this data suffers from the same “bursting” issue as is evident from the fact that the cumulative packets received line is step-shaped instead of linear.

Even though only 160 non-empty fingerprints were collected by the test subject’s device, any gaps were filled adequately after applying the exponential moving average, which resulted in 285

(25)

0

10

20

30 Time (s)

0

200

400

600

800 1000

Packets received

Theoretical

Actual

Figure 4.4: This graph shows a 30 second time period during which packets were collected. Packets were received in bursts, instead of a steady stream. The theoretical amount of packets received in 30 seconds is calculated from the number of beacons in range and the configured advertisement rate. Note that at multiple instances in this 30 second time period, no packets were received at all, resulting in unusable fingerprints.

of fingerprints, were removed (those with a duration of less than 10 seconds). This resulted in a smoothed predicted path. Both paths are visualized in figure 4.6. This visualization represents the movement in the physical space accurately, but does not convey any information on the duration of interaction. Figure 4.7 shows a time-discrete visualization of the predicted path.

4.2.3 Discussion

The issue of packets being received in bursts is suspected to be due to signal interference of other devices on the same frequency, as the issue is not present when the system is tested in a different environment with less Wi-Fi networks and Bluetooth-enabled devices nearby. However, to ensure that signal interference is in fact responsible, more in-depth testing will have to be done.

It seems that changing the beacon locations to a grid affected the feature importance as well, as the beacons contribute more equally to the classification process compared to the previous experiment.

The path as predicted by the classifier is reasonably accurate. Nevertheless, because of Bluetooth’s subjectivity to interference, movement and antenna orientation, spurious misclassifi-cations will occur frequently [14]. This is solved partially by removing classifimisclassifi-cations that do not last for more than a couple of seconds. However, this is by no means a perfect solution. Another approach to solving this problem would be to take into account the distances between sections, and removing any transitions between sections deemed “impossible”.

Whether this system is more accurate than just using one proximity beacon per section is another area that must be investigated more thoroughly, although installing one beacon per exhibition may be considered impractical compared to the proposed system. No assumptions can be made about exact errors when using the proposed system, because no exact positioning system for reference is available.

(26)

0

50

100

150

200

250 Time (s)

90

80

70

60

50

40 RSSI

0

500 1000

1500

2000

2500

3000

Packets received

Figure 4.5: RSSI signals captured by the test subject’s device, after applying an exponential moving average. Each line represents the signal strength received by one beacon. The black line indicates the cumulative number of packets received.

(27)

Actual path

Predicted path

Predicted path (smoothed)

Figure 4.6: Top: actual path in the physical space. Bottom: predicted and smoothed path visualized in the physical space.

(28)

0

50 100 150 200 250 300

Time (s)

vortex

jumping ring

coin crush

colour filters

colour mixer

drop stopper

spinning magnet

static table

listening dishes

lightning spiral

ball blower

lightning globe

Section

Prediction

Prediction (smoothed)

Actual

(29)

CHAPTER 5

Conclusions

5.1 Applications in research

The proposed system will aid in conducting experiments aimed at analysing movements through a physical space.

The following metrics can be deducted from data collected by this system: • Path in physical space

• Time spent in a location • Number of sections visited

These metrics can be used in a variety of ways such as path optimization, exhibition popu-larity, A/B tests, heatmaps and network graphs.

5.2 Limitations

The system design has several limitations, both in hardware and in software.

Firstly, it is prone to interference. This is a frequent and largely unsolved problem with Bluetooth-based positioning systems [11][14][28][15]. As can be seen in experiment 2, interference may cause dropped packets and a limited amount of usable fingerprints as a result.

Furthermore, the system currently only uses RSSI to generate a fingerprint. Other properties could be used in addition to RSSI to improve classification accuracy, such as sensor data, the Bluetooth channel at which a packet was received and the measuredPower factory calibrated beacon constant. The current system is unable to use these properties because it is built around classifying fingerprints - not individual packets.

5.3 Further work

A second iteration of the proposed system could be centered around classifying packets instead of fingerprints, and combining the probabilities of these individual packet classifications to generate an overall location. The advantage of this is mainly that it would allow for more features to be considered for classification, such as the measuredPower constant mentioned earlier as well as the Bluetooth channel. Bluetooth 5.1 (published January 2019) will contain the angle of broadcast (AoD, angle of departure) and the angle when received (AoA, angle of arrival), two features that may improve classifier accuracy even more [25].

(30)

(31)

Bibliography

[1] _{Android Bluetooth Low Energy Overview. url: https://developer.android.com/guide/} topics/connectivity/bluetooth-le.

[2] P. Bahl and V.N. Padmanabhan. “RADAR: an in-building RF-based user location and tracking system”. In: Institute of Electrical and Electronics Engineers (IEEE), Nov. 2002, pp. 775–784. doi: 10.1109/infcom.2000.832252.

[3] Gavin C Cawley and Nicola L C Talbot. On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Tech. rep. 2010, pp. 2079–2107.

[4] Sudarshan S. Chawathe. “Beacon placement for indoor localization using Bluetooth”. In: IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC. 2008, pp. 980– 985. doi: 10.1109/ITSC.2008.4732690.

[5] Liang Chen et al. “Bayesian fusion for indoor positioning using bluetooth fingerprints”. In: Wireless Personal Communications 70.4 (June 2013), pp. 1735–1745. issn: 09296212. doi: 10.1007/s11277-012-0777-1.

[6] Amira Chriki, Haifa Touati, and Hichem Snoussi. “SVM-based indoor localization in Wire-less Sensor Networks”. In: 2017 13th International WireWire-less Communications and Mobile Computing Conference, IWCMC 2017. Institute of Electrical and Electronics Engineers Inc., July 2017, pp. 1144–1149. isbn: 9781509043729. doi: 10.1109/IWCMC.2017.7986446. [7] P. C. Deepesh et al. “Experiences with using iBeacons for Indoor Positioning”. In: Associ-ation for Computing Machinery (ACM), Feb. 2016, pp. 184–189. doi: 10.1145/2856636. 2856654.

[8] Javier J.M. Diaz et al. “Bluepass: An indoor Bluetooth-based localization system for mobile applications”. In: Proceedings - IEEE Symposium on Computers and Communications. 2010, pp. 778–783. isbn: 9781424477555. doi: 10.1109/ISCC.2010.5546506.

[9] _{EddyStone specification. url: https://github.com/google/eddystone.}

[10] _{Estimote Telemetry Specification. url: https : / /developer . estimote . com / sensors/} estimote-telemetry/.

[11] Ramsey Faragher and Robert Harle. “Location fingerprinting with bluetooth low energy beacons”. In: IEEE Journal on Selected Areas in Communications 33.11 (Nov. 2015), pp. 2418–2428. issn: 07338716. doi: 10.1109/JSAC.2015.2430281.

[12] Aaron Fisher, Cynthia Rudin, and Francesca Dominici. “All Models are Wrong but many are Useful: Variable Importance for Black-Box, Proprietary, or Misspecified Prediction Models, using Model Class Reliance”. In: (Jan. 2018). url: http : / / arxiv . org / abs / 1801.01489.

[13] Robin Heydon. Bluetooth Low Energy: The Developer’s Handbook.

[14] Sylvia T Kouyoumdjieva and Gunnar Karlsson. Experimental Evaluation of Precision of a Proximity-based Indoor Positioning System. Tech. rep. url: https://www.arubanetworks. com.

[15] Dimitrios Lymberopoulos et al. “Microsoft Indoor Localization Competition”. In: ACM SIGMOBILE Mobile Computing and Communications Review 18.4 (Jan. 2015), pp. 24–31. issn: 15591662. doi: 10.1145/2721914.2721923.

(32)

[16] _{Rainer Mautz. “Indoor positioning technologies”. In: (). doi: 10.3929/ethz-a-007313554.} url: https://doi.org/10.3929/ethz-a-007313554.

[17] _{Nested versus non-nested cross-validation. url: https://scikit- learn.org/stable/} auto_examples/model_selection/plot_nested_cross_validation_iris.html. [18] _{Phone, Tablet & PC Market. url:}

https://www.bluetooth.com/markets/phone-pc/.

[19] Valentin Radu and Mahesh K. Marina. “HiMLoc: Indoor smartphone localization via activ-ity aware pedestrian dead reckoning with selective crowdsourced WiFi fingerprinting”. In: 2013 International Conference on Indoor Positioning and Indoor Navigation, IPIN 2013. IEEE Computer Society, 2013. isbn: 9781479940431. doi: 10.1109/IPIN.2013.6817916. [20] _{Scikit Learn: Support Vector Machines. url: https : / / scikit - learn . org / stable /}

modules/svm.html.

[21] Fazli Subhan, Halabi Hasbullah, and Khalid Ashraf. “Kalman Filter-Based Hybrid Indoor Position Estimation Technique in Bluetooth Networks”. In: International Journal of Navi-gation and Observation 2013 (Sept. 2013), pp. 1–13. issn: 1687-5990. doi: 10.1155/2013/ 570964.

[22] _{Technical specification of Estimote Beacons and Stickers. url: https://estimote.com/} products/%20https://estimote.com/products/.

[23] Yapeng Wang et al. “Bluetooth positioning using RSSI and triangulation methods”. In: 2013 IEEE 10th Consumer Communications and Networking Conference, CCNC 2013. 2013, pp. 837–842. isbn: 9781467331333. doi: 10.1109/CCNC.2013.6488558.

[24] Andy Ward, Alan Jones, and Andy Hopper. A New Location Technique for the Active Office. Tech. rep.

[25] Martin Woolley. Bluetooth Core Specification v5.1 Feature Overview Bluetooth Core Spec-ification v5.1 contains a series of updates to the Bluetooth core specification. This doc-R

ument summarizes and explains each change. Bluetooth Core Specification v5.1 should be consulted for full details. Tech. rep.

[26] Steven S. Yalowitz and Kerry Bronnenkant. “Timing and tracking: Unlocking visitor be-havior”. In: Visitor Studies 12.1 (Jan. 2009), pp. 47–64. issn: 10645578. doi: 10.1080/ 10645570902769134.

[27] Paul A. Zandbergen. “Accuracy of iPhone locations: A comparison of assisted GPS, WiFi and cellular positioning”. In: Transactions in GIS. Vol. 13. SUPPL. 1. 2009, pp. 5–25. doi: 10.1111/j.1467-9671.2009.01152.x.

[28] Li Zhang et al. “A comprehensive study of bluetooth fingerprinting-based algorithms for localization”. In: Proceedings - 27th International Conference on Advanced Informa-tion Networking and ApplicaInforma-tions Workshops, WAINA 2013. 2013, pp. 300–305. isbn: 9780769549521. doi: 10.1109/WAINA.2013.205.

[29] Wenzhe Zhang et al. “INBS: An Improved Naive Bayes Simple learning approach for accurate indoor localization”. In: 2014 IEEE International Conference on Communica-tions, ICC 2014. IEEE Computer Society, 2014, pp. 148–153. isbn: 9781479920037. doi: 10.1109/ICC.2014.6883310.

[30] Sheng Zhou and John K. Pollard. “Position measurement using Bluetooth”. In: IEEE Transactions on Consumer Electronics 52.2 (May 2006), pp. 555–558. issn: 00983063. doi: 10.1109/TCE.2006.1649679.

Analysing visitor flow using a Bluetooth positioning system

Bachelor Informatica