Mobile Biometric Authentication System

(1)

Mobile Biometric Authentication

System

Master Thesis

Jonatan Felipez Queijo

felipez@live.nl

August 5, 2015, 51 pages

Supervisor: dr. M. Bruntink

Host organisation: bunq B.V.,http://bunq.com

Universiteit van Amsterdam

(2)

Abstract

For years, biometrics have been seen as the promise of easy authentication. In this thesis, a replication report of one such authentication system is presented. Additionally, the replicated system is extended to the mobile domain, that is, the replicated system is extended with the use of an android device as an input channel, thereby replacing the traditional input (a pressure-based drawing tablet). The performance of the system is measured on data gathered from the host organization and its match rate compared to benchmarks of the replicated system. The result is a match rate of the mobile system that is comparable to the replicated system which uses the traditional input.

(5)

Chapter 1

Introduction

1.1 Overview

A well known security problem is that of user authentication: the act of confirming or proving the identity of a user. In contrast to user identification, which refers to the act of stating or indicating the user identity, authentication is the process of confirming such a claim.

The most basic form of user authentication in the context of computer systems is password authen-tication. This method of authentication relies on the ability of the user to remember a username / password combination as well as the ability of the user to keep the password a secret (traditionally, the password is implemented as a string of characters).

While the widespread use (e.g. on the web) of this method somewhat proves its effectiveness in general, it does inhibit some disadvantages. For instance: a user is forced to remember a password, which means that the length and complexity of the password is limited to the ability and willingness of users to do this. Unfortunately, experience has shown that passwords that are easy to remember (”john93”, ”password”, ”admin”, etc.) are often also easy to guess. Additionally, enforcing strict password requirements, often leads to users writing their password down [1], which increases the chance that this private piece of information is shared with others.

In search of an authentication medium that does not have these disadvantages, researchers have turned to the use of biometric technologies [2]:

“Biometric technologies are automated methods of verifying or recognising the identity of a living person based on a physiological or behavioural characteristic.”

Examples of biometric traits, the physiological or behavioural characteristics that are mentioned in the definition above, include fingerprint, iris, face and voice recognition. Although the use of biometric technologies allows us to mitigate some of the disadvantages inherent to the more traditional plaintext password medium, a silver bullet has not yet been found. Moreover, the use of biometrics has its own set of challenges that are either trivial or already solved for the password medium. Maiorana and Ercole [3] present an authentication system architecture that claims to solve several of these challenges. Additionally, two implementations of the architecture are described, both using different biometric traits. This thesis aims to answer the following research questions concerning this article:

Q1 Can the implementation of the architecture that uses dynamic signature be replicated based on the description in the article?

Q2 Do the presented results of the system hold under the use of public datasets?

Q3 Can the system architecture be used in a modern mobile environment?

(6)

1.2 Relevance and Motivation

In any science or engineering discipline, the results of a single experiment can not be viewed as representative for the underlying reality. To increase the credibility of experiments, one can repeat the experiment to double-check the results. This is known as replication.

Inspired by the natural sciences, in the field of Software Engineering many attempts have been made at identical replication, in which all conditions of the original experiment are met, but this has proven difficult. One key reason for this is the sheer complexity of the software development setting [4].

The fact that the article by Maiorana and Ercole is the only available source of information con-cerning the original experiment that is to be replicated in this thesis, means that an exact replication is clearly not reliably feasible [5].

I (and others [4]), however, argue that non-exact replications can still be useful to gain knowledge. By attempting a close replication and explicitly documenting all assumptions and interpretations as much as consciously and reasonably possible, it will be easy to pinpoint probable causes when new results differ from the original results. These probable causes can then be ascertained when more information of the original experiment (e.g. the source code) becomes available. Additionally, making assumptions and interpretation explicit makes it easier for future reproducers to scrutinise.

The mobile version of the system presented in the thesis is in fact a replication of the replicated system under different, yet sufficiently controlled, conditions: a differentiated replication. The goal of such a replication is to ascertain the range of conditions under which the original results will still hold and under which they will not [4].

More practical relevance lies in the possible application of the mobile system. As the use of smart phones becomes more wide-spread, the need for easy mobile authentication also spreads. Bunq, the host organisation, is interested in such mobile authentication and is interested to know whether the use of biometrics can cater to their needs.

1.3 Thesis contribution

This thesis provides four contributions:

1. A publicly available close replication of the system described by Maiorana and Ercole, including documented design interpretation and assumptions that make it easier to replicate as well as attest the closeness of the replication once more information about the original system becomes available.

2. The performance results of the replicated system on publicly available data sources to further increase the ease with which the closeness of the replication can be verified.

3. A publicly available differentiated replication in which the system is modified to work in a mobile environment.

4. The results of testing the mobile application on data gathered from bunq employees. Unfortu-nately, data is not publicly available due to privacy concerns of the subjects.

1.4 Outline

In chapter 2 the background for this thesis, including references to further literature relevant in this field is described. In chapter 3 a summary can be found of the system description in the article by Maiorana and Ercole. Chapter 4 describes the methods used to answer the research questions. Chapter 5 describes the prototype construction of the replicated system as well as the necessary changes to use the system in a mobile setting. Chapter 6 describes the results of testing the performance of the original prototype using public databases as well as the mobile prototype using the gathered data at bunq. Additionally the results are discussed in this chapter. Finally, chapter 7 summarises the results, provides answers to the research questions and describes possible further research.

(7)

Chapter 2

Background

2.1 Biometrics: Dynamic signature

The range of characteristics used in biometric identification and authentication is quite large. Systems have been proposed for face [6], keyboard dynamics [7], retina [8], iris [9], ear [10], gait [11], finger prints [12], voice [13] and palm print [14] recognition. Additionally, multi-modal systems [15] combining several traits have been proposed. In this list, the most relevant characteristic (or trait) for this thesis is the dynamic signature, as it is the trait that is used in the replicated system. One of the challenges of using biometrics in general is to find a way to store the biometric templates such that they are cancellable and secure as discussed in [16].

Typically the performance of authentication systems using Dynamic signature is estimated using the FRR, FARRF FARSF (refer to Figure 2.1). SF refers to the use of “skilled forgeries”, forgeries

that are made by an opponent who has knowledge of the signature (e.g. an image of the signature). RF refers to “random forgeries”, which means the opponent has no knowledge of the signature and has to guess randomly.

Several authentication systems address this issue of user templates protection. For instance, in [17] a fuzzy vault scheme [18] is applied to online signature verification. Although the authors report promising FAR (1.2% for skilled forgeries), the FRR of 57.3% is quite high. Another fuzzy vault scheme is proposed in [19]. The reported system performance is good, but authentication requires both a signature sample and a password. Consequently, it suffers from the weaknesses of traditional password authentication. One property that all the presented systems have in common is a tradeoff: the cost of having template protection is a significant penalty to the authentication performance compared to the unprotected templates.

To benchmark the performance of authentication systems, multiple databases containing dynamic signature data are available freely for research purposes. Examples of these databases are SVC2004 [20], MCYT [21] and SUSIG [22].

The use of biometric authentication on mobile devices has shown to be practically applicable as is apparent of the fingerprint readers appearing on smart phones. Dynamic signature is not an exception to this: multiple systems have been presented that implement dynamic siganture verification [23]. Although the performance of the mentioned systems do not perform on par with traditional (pressure tablet) systems. In [24], however, an implementation is presented for (Android) smartphones that performs on par with traditional systems but does not take into account that user templates have to be protected. No systems have been presented that combine an application to mobile devices and template protection to the author’s knowledge.

(8)

Figure 2.1: A Venn diagram depicting the performance metrics of biometric authentication systems. Typically it is desirable to maximise the intersection between genuine and accepted identity claims: make sure that only genuine identity claims are accepted and no genuine claims are rejected. As a clarification, the proportions of the circles of this diagram do not depict any type of scale.

2.2 Error-correcting codes: BCH and RS

The correction of data errors (in the field of computer science and telecommunication) can be done in multiple ways:

Backward error correction: the error is detected, after which a retransmission of the data is re-quested.

Forward error correction: the data is encoded using an error-correcting code before the transmis-sion and decoded on retrieval. The decoding process detects and corrects a certain amount of errors

Figure 2.2: Forward error-correction process; when the amount of errors is less than the error correc-tion capability (ECC) of the error code, the errors are corrected by the decoder.

(9)

BCH (Bose-Chaudhuri-Hocquenghem) codes as introduced in [25] are forward error-correcting codes that work on a binary alphabet. RS (Reed-Solomon) codes [26] are a generalization of BCH codes: they can be used on non-binary alphabets. The trade-off is that it is less computationally complex to correct BCH codes than RS codes. Additionally, RS codes require more bits per corrected error [27]. Both RS and BCH have found widespread application in data storage (e.g. CD, QR codes). Although a full elaboration on the mathematics and workings of error correction codes is out of the scope of this thesis, it is useful to mention that the encoding and decoding of a codeword requires calculations in finite fields, also known as Galois fields. The workings of these calculations are described in [28].

Although it seems unorthodox, BCH and RS have also found their application in the field of biometric authentication. In [29], to cope with the variability of data aquired by the iris camera, error correction capabilities are added to the user template using RS codes. The fuzzy comitment scheme introduced by Juelz and Wattenberg in [30] uses BCH codes and helper data. Helper data is unencrypted data which is stored as part of the user template to harness the error correction capability of the BCH codes when authenticating. The replicated article uses an authentication scheme that was based on this paper.

2.3 Threshold Cryptography

In 1979, Shamir published a paper [31] in which he discussed the notion of secret sharing: splitting a secret among n persons such that any k (< n) of them are able to reconstruct the secret, yet the collaboration of k − 1 persons does not reveal any information about the secret. In a public key infrastructure, the secret is the private key which is shared among n parties. Desmedt and Frankel [32] were the first to present practically applied secret sharing in threshold cryptosystems. The systems are practical in the sense that there is no need to recreate and distribute the secret key every time it is used, as the key is never revealed to any insider or outsider. Both systems are based on the ElGamal public key cryptosystem [33]. One of the solutions uses Lagrange interpolation. Lastly, the authors report failures to produce such a threshold system using traditional RSA. Shoup [34] however, presents a threshold signature scheme using RSA and Lagrange interpolation. The scheme is generalised in [35], where the authors also claim that the generalization makes it possible to apply the scheme in, among others, RSA shared decryption.

(10)

Chapter 3

Replicated Article

In 2007 an article [3] was published by Emanuele Maiorana and Chiara Ercole in which the authors present a biometric authentication system architecture where they focus on two vulnerabilities. Firstly, they focus on the database where the templates are stored. Secondly, they focus on the communication channel between the matcher and the stored templates. The article provides detailed descriptions of two sample implementations of the architecture: one using dynamic signature and one using the iris as biometric trait. For each of these implementations, the results of performance experiments are provided. As is typical for biometric authentication systems, the system can be divided into the enrolment process and the authentication process. Below, these processes are described for the architecture that does not apply “user-adaptive error correction capability”. After these processes the use of threshold cryptography in the system is described.

3.1 Enrolment Process

Figure 3.1: A flow chart depicting the enrolment flow of a system implementing the architecture.

When the enrolment process starts (at the acquisition sensors), the user provides s samples. A sample is one attempt of the user to draw his signature. From these s signature samples raw features are extracted, resulting in s vectors of length n, i.e. one value for each extracted feature. For each of these n features, the average value vn is computed which yields the average sample sm. Then, the

Bit String quantifier uses the Global Mean (a global average signature, which is estimated by taking the mean signature of the data set) to convert the raw feature values into one bit per feature (the relative feature value). The following formula is used:

(11)

rn=

(

1, if vn ≥ mn

0, otherwise Where:

• vn is the raw nth feature value of the average sample sm

• rn is the relative nth feature value of the sth sample

• mn is the raw value of the nth feature of the Global Mean

this yields the relative signature y. The BCH Encoder receives a random number x of length n, which is then encoded as x0. Here encoding means that the error correcting data is appended to the random number. A hash of x is stored as part of the user template. The other part of the user template, namely the Helper Data is computed and stored using the following formula:

Helperdata = x0⊕ y

As x0 is larger than y due to the added error correcting symbols, y can be zero padded to make up for this. Summarising, only the hash of random number x and the helper data make up the stored user template.

3.2 Authentication Process

Figure 3.2: A flow chart depicting the authentication flow of a system implementing the architecture

The authentication Process is very similar to the enrolment process. One difference is that during the authentication process, the user can only provide one sample. From this sample, n features are extracted. Analogously, the Bit String Quantizer converts the n values into n bits, yielding the Relative Signature y’. The Codeword Generator retrieves the Helper Data for the claimed identity, which is stored in the user template. It performs the following operation:

Codeword = y0⊕ HelperData

The system behaviour then depends on the value of Codeword. The following three scenario’s are possible (y is the Relative Signature obtained during enrolment):

(12)

y’ is exactly equal to y If this condition holds, the generated codeword becomes exactly the encoded random number x obtained at enrolment due to the following equation:

A ⊕ A = 0 (B ⊕ A) ⊕ A = B ⊕ (A ⊕ A) = B ⊕ 0 = B

(3.1)

Decoding and hashing the codeword results in the possibility to compare the result with the hashed random number that was stored in the user profile during enrolment. In this scenario the comparison succeeds and the user is authenticated.

y’ is close enough to y In this scenario, the error correction kicks in during decoding, resulting in the previous scenario. Thus authentication succeeds.

y’ is not close enough to y Here the value of y’ is not close enough to be corrected. Consequently, x can not be reconstructed and thus the authentication fails.

3.3 Feature extraction

The system described in the paper extracts features from the list of signature points produced by the input sources. The authors provide a Table containing the names of the features (Table3.1and Table

3.2), which are divided into the categories Global and Local. The formal definitions of the features can be found in appendixA.

Global Features

The definition of global features is in line with definitions found in other literature [36,37]: for each signature sample, one global feature value is extracted.

Table 3.1: List of names of extracted Global Features given by the article

X Area Y Area M11

Avg X velocity Avg Y Velocity M12

Avg X acceleration Avg Y acceleration M21

Initial X Initial Y M03

Final X Final Y M30

Minimum X Minimum Y Mean pressure Maximum X Maximum Y Minimum pressure Mean X Mean Y Maximum pressure Maximum X Velocity Maximum Y Velocity Number of strokes Abs avg X Velocity Abs avg Y velocity Time Duration Width Height Aspect Ratio(y vs x)

Local Features, Segmental Features

As defined by the reproduced article [3], local features are based on a partition of each signature in five consecutive and disjoint segments with the same temporal extension. More formally:

• Let X be the ordered set of points corresponding to signature sample S, • Let T be the ordered set containing all time values for x ∈ X

• Let tx be the time value of x ∈ X

• Let tspan= tmax− tmin, the difference between the maximum and minimum value in T

(13)

Pn = { x | 0.2ntspan< tx≤ 0.2(n + 1)tspan, x ∈ X}

This definition, however, is not very intuitive as the term local implies that a value is extracted locally, i.e. at each point in the signature sample. Other literature [37] opts for a different, more intuitive name for this kind of feature, namely Segmental features, since the signature is partitioned in a number of segments (in this case five) and from each of these segments a feature value is extracted.

Table 3.2: List of names of extracted Segmental Features given by the article

X Area X Velocity X acceleration Y Area Y Velocity Y acceleration

Although most of the feature definitions can more or less be intuitively extracted from their names, there is still a certain amount of interpretation and assumption involved when implementing them. To remove all doubt for possible replicators of this system, a list of the formal definition of these features has been added to the appendix of this thesis.

3.4 Use of Threshold Cryptography

One feature that is not necessarily typical for biometric systems is the use of a threshold cryptography scheme. The authors apply this technique to avoid corruption, modification and interception of the stored templates. The approach uses RSA and Lagrange Interpolating Polynomial Scheme to distribute the private key on several servers. The explicit use and implementation is, however, not discussed.

3.5 Consequential Claims

The authors claim that the system described above exhibits the following desirable properties:

Cancellable template Since the user template consists of a hashed random number and the helper data and specifically not the raw signature features, it is relatively easy for a user to create a new template which is different from the currently stored one, while not necessarily changing his biometric trait. This property is desirable in the event that a stored template gets stolen by an attacker.

Robustness under noisy channels As discussed priorly, the sampling of biometrics is prone to noise. This is especially the case with behavioural biometrics. The described system compen-sates for this noise by allowing a margin of error.

Security against corruption, modification and interception of stored templates These prop-erties are nice to have as they make the system more robust against attackers.

(14)

Chapter 4

Method

This chapter describes how the research questions will be answered.

4.1 Architecture implementation based on article (Q1)

To answer the first research question, a prototype will be built based on the information in the article describing the system. In chapter 5, the construction process of the prototype is more accurately described, i.e. the gathering of requirements, the architectural constraints, the construction and the testing of the prototype. The paper describes two architecture implementations: one using dynamic signature and one using the iris as biometric feature. The prototype will replicate the system using dynamic signature. Two variants of the system using dynamic signature are described in the original paper: a user-adaptive variant and a non-user adaptive version. User adaptiveness of the error correction capability is a feature that can straightforwardly be added to the simpler non-user adaptive version. As such, only the non-user adaptive version will be built, leaving the user adaptiveness available for possible further research.

4.2 Benchmarking using public datasets (Q2)

The performance of the system will be tested using public databases, due to the original data not being available. The original system is not available either, thus the prototype produced in chapter 5 will be used instead as it is based on all the available information of the original system.

The performance of the system will be tested by estimating the FRR and FAR using random forgeries or skilled forgeries. Analogously to the original article, there will be two variation points: the error correction capacity of the user template and the amount of user templates used for enrolment.

The following three publicly available datasets will be used:

1. SVC2004 second task [20]

This database contains 20 genuine signatures (10 gathered in one session, another 10 gathered in a separate session at least one week after session 1) of 40 users. For each of these users an additional 20 forged signatures are supplied, which were made after some practice by at least four different persons. For the enrolment, the first 10 genuine signatures will be used. To measure the effect of the amount of enrolment samples to the results, all 10 settings will be tested (setting 1 uses one enrolment sample, setting 2 uses two enrolment samples, etc.). The effect of the ECC will also be measured by letting it vary between 0 and 30 for each user, for each enrolment setting. These settings expand upon the settings provided in the original paper. As such, they provide a more complete view of the system performance. The performance will be estimated using the same metrics as the article:

(15)

• The FRR will be tested by trying to authenticate all genuine signatures of the user and calculating at which rate they are rejected.

• The FARSF will be tested by calculating the acceptance rate of the forgeries for each user.

• The FARRF will be tested by calculating, for each user, the acceptance rate of the genuine

signatures of other users.

2. SUSIG Visual Corpus [22]

A database containing the data of 100 users. For each of these users 20 genuine samples have been gathered: 10 in the first session, 10 in the second session. The time in between each session is at least one week. Additionally, there are 5 skilled forgeries per genuine user which where made by randomly selected other data providers and 5 highly skilled forgeries which were made by the authors. The highly skilled forgeries were made by using an extra advantage: the forgeries could be made by retracing a genuine signature on the LCD screen itself. The FRR, FARsfand FARrf will be calculated in the same way as the SVC2004 database mentioned above.

3. MCYT, non-commercial edition [21]

Although MCYT contains a commercially usable signature database of 300 persons, it is not freely available. There is, however, a publicly available research-only database that will be used here. The database contains 25 genuine signatures of 100 individuals gathered in 1 session. Additionally, 25 forgeries are provided for each user, made by the 5 subsequent users that contribute their signature to the database. In contrast to the other databases, the signatures are gathered in a short amount of time as follows: user n (ordinal index) makes 5 genuine signature samples after which he produces 5 forgeries for user n − 1. After this, he provides another 5 genuine signatures and 5 forgeries for user n − 2. This process goes on until the user has provided the desired amount of signatures. The performance will be estimated in a similar way as when using the SVC2004 database:

• The FRR will be tested by trying to authenticate all genuine signatures of the user and calculating at which rate they are rejected.

• The FARSF will be tested by calculating the acceptance rate of the forgeries for each user.

• The FARRF will be tested by calculating, for each user, the acceptance rate of the genuine

signatures of other users.

One problem for answering this research question is the fact that, other than three system perfor-mance results and the corresponding system settings, nothing is known about the data set used in the original paper. As such, it is impossible to perform a statistical analysis to determine whether performance differences are significant or not. In the face of this, an alternative method will be used to consequently answer the question of whether the system performs differently than reported: The reported results may not differ more than 10 percent point for each metric after calibration to be con-sidered similar. If the difference is more than 10 percent point, the result shall be concon-sidered different. Note that 10 percent point is explicitly not based on any scientific results and is an educated guess that will be used as a means to systematically and consequently determine whether values should be considered similar until more information about the original data set becomes available, after which a statistical analysis can be done.

4.3 System architecture in a mobile environment (Q3)

To verify whether the architecture can be used in a mobile environment, the prototype will be extended to the mobile platform, i.e. the system should provide the same functionality without modifying the architecture extensively (changing other parts than the input device). Specifically, a device running the Android operating system (the Nexus 5 phone) will be used to gather the input instead of a pressure tablet as is the case for the three public databases mentioned before.

(16)

4.4 Measuring the performance of the mobile prototype (Q4)

To answer the last research question, similar tests will be done as the ones for measuring the perfor-mance of the first prototype. The FRR, FARSF and FARRF will be measured similarly to the original

prototype. Additionally, the equal error rate (EER) or crossover rate (the point at which FARSFand

FRR are closest to each other) will be used to compare the performance of the mobile system with that of the non-mobile system on the public datasets.

4.4.1 Data gathering process

The signatures of 25 different bunq employees have been gathered to answer this research question. For every subject sample, the subject was seated and the phone was flat on the table. The subjects were told what the purpose of the system was, namely authentication using signatures instead of passwords. The following data was gathered per subject (one sample is equal to one signature entry):

• 10 samples were provided by the subjects in the first session. These are to be used as enrolment samples

• 20 samples were gathered from the subjects at least three days after session 1 in batches of 5. There was a pause of at least two hours in between each batch. The purpose of these samples is to simulate an authentication attempt.

• 20 forgeries gathered in a third session, created by two different forgers. The forgers had access to the static image of a sample that was gathered in session 2 and were instructed to forge the samples as accurately as possible, also taking timing and finger pressure into account. Before delivering the forgeries, a training session of 10 minutes was held in which the forgers could practice and get used to the Nexus 5 phone.

The data gathering process (amounts and sessions) was inspired by the data gathering processes of the public data sets. Additionally, the size was chosen based on the minimal willingness of the subjects to provide data in the period of time available to do the research and write this thesis.

4.5 Dataset validity

One could question how accurate the datasets model the behaviour of the global (or even the local) population. Assumptions need to be made to explore how well the system works under certain conditions, due to the complexity of reality. The authors of all data sets have their own set of assumptions that could affect the validity of their data. For one, the physical data-gathering location might results in biased data as the data sample is not gathered uniformly over the population it tries to model. The SUSIG database was gathered in Turkey, MCYT was gathered in Madrid and SVC2004 was gathered in Hong Kong. The mobile dataset was gathered from bunq employees in the Netherlands. Additionally, the way enrolment, authentication and forgery are modelled might influence the data. For example, the authentication samples from SUSIG and SVC2004 are gathered in a separate session, some time (days) after enrolment. The MCYT set however, does not make such a distinction and recorded every signature in one session. No empirical argumentation is given for either modelling decision, but intuitively the latter solution is a less accurate model of reality as the addition of time between enrolment and authentication, an arguably common and realistic use case, might influence the gathered data. Similar modelling decisions that might affect the data are the following:

• SVC2004 subjects are encouraged to not use their regular signature as part of the data-set. This is not the case for the other datasets

• All databases use a relatively small number of subjects: the mobile database gathered for this thesis uses only 25 subjects and the biggest two databases use 100 subjects.

(17)

• All databases use different input hardware. The calibration and accuracy of the hardware might influence result.

• The forging conditions are different. SVC2004 and MCYT let subjects that have provided data also provide forgeries of others. SUSIG does the same for half the forgeries (the forgers were shown an image of the signature they had to forge). The other half was produced by the authors as “highly skilled forgeries”, having access to the drawing simulations and they were also able to retrace the sample on the lcd input screen.

Due to these different modelling decisions, the results of the experiments may differ from each other and the originally reported results. To partially account for this, the system will be calibrated if the performance of the system is too strict or too yielding.

Two additional points specific to the mobile dataset were revealed during data gathering. Firstly, some subjects mentioned that they found it awkward to use their finger on a small touch screen instead of a pen on paper. This might negatively affect the match rates as the use of dynamic signature as a biometric is based on the assumption that a signature signing process is distinctive enough to distinguish from forgers, yet consistent enough for multiple authentications due to the subject’s habit of signing documents and the lack thereof in other persons. Secondly, some subjects mentioned that they were not used to this specific type of touch screen and requested to do the signatures on their own device. For consistency sake, this request was denied, yet they do make a valid point. Researching the consequences of these two points falls outside of the scope of the thesis. As such, they will be left to further research, even though they might affect the validity of the mobile dataset.

(18)

Chapter 5

The Prototype

5.1 Architecture and Requirements

As for the architecture, the description in the article shall be used as the blueprint for building the prototype, a description of the system in the article is available in chapter 3. The author claims mentioned in that chapter are viewed as architectural constraints and requirements to the prototype (e.g. it has to support cancellable templates).

One additional requirement that is not apparent from the article and consequently not from this description is the use of multiple data sources. Since the experiments will be done using multiple databases, this should be facilitated by the system. To cope with this requirement, the used data source should be encapsulated for it to be easy to change, without changing the rest of the system.

5.1.1 Module View

The logical decomposition of the system into modules can be seen in Figure 5.1. Submodules with related responsibilities have been grouped into separate modules. Although similarly named submod-ules can be found in the Entity Module, their functionality is different. That is, submodsubmod-ules of the Entity Module have in common that they provide some form of persistence service. For example, the ExperimentResult maps to a storage entity (database table). The Data Import Module contains submodules that are concerned with importing data from raw files into the database. Similarly, the Error Correction Module contains classes that are responsible for providing error correction services. The Signature Module contains classes that are responsible for modelling a database independent signature (i.e. the internal representation of the signature). The ‘...’ in the submodules indicate that the submodule contains more classes than are shown in the view. In this case, the list of (segmen-tal) features has been omitted as the list is quite lengthy and would result in a view that would be unnecessarily more complex.

5.1.2 Data-Flow and Processes

The application prototype can be split up into four different processes that are sequentially initialised through the Main method. In the figures describing these processes, a cilinder depicts a (persistent) storage entity and a square is an object. An arrow depicts a data flow from a component to or from a storage entity. The direction of the arrow dictates the data flow and the text closest to an arrow describes what kind of data passes through it. the square brackets in “SignaturePoint[]” indicate a collection of SignaturePoint objects.

Importing Data (Figure5.2)

This process takes care of parsing the raw data from the public datasets and storing the parsed signature points in storage entities. The overlapping objects and entities indicate that every datasource has its own object and/or storage entity. For example, SVC2004 has its own data files (provided by HK university). Consequently it has its own implementation of the data importer

(19)

Figure 5.1: A high level view of the logical grouping of the application. The naming roughly follows the names of the application packages. A submodule represents either a class or a sub package.

and an own storage entity (this decision was made to minimise the chances of using the data from one database in another and to simplify queries to the storage entities. The tradeoff here is extensibility as adding a new data source now requires the addition of each of the named objects and storage entities. The “EntityManager” object is required for persistent storage.

Signature Caching (Figure 5.3)

After the system has imported the data into the storage entities, it will cache the signatures in a persistent storage entity. The reason for this is a gain in performance. During experiments,

(20)

Figure 5.2: A graphical representation of the data flow during the process of importing signature data into the system

many settings have to be tested, which means that the signatures have to be reconstructed many times. This runtime penalty can be avoided by caching the signatures in the database. The tradeoff in memory is neglectable for the size of the datasets that will be used. The “DataIm-porters” have inherited the signature caching responsibility for their own database as they have the knowledge of what storage entity to query and in what storage entity to store the cached signatures. The SignaturePoints are queried by UserId, after which they are stored in a “Sig-natureUserPoints” object. This object models a collection of signaturePoints from one user. It sorts the signature points by sample id and stores these sub-collections into a “SignatureSam-plePoints” object, which models a collection of SignaturePoints from the same sample. When constructing a “SignatureSample” from these points, there is some inversion of control: a Sig-natureSample object is created and a reference of this object is passed to each feature with the request to update the property of the object that will be stored. A positive consequence of this is that it is easy to add new features almost modularly. There is also a negative consequence: when a property is changed unexpectedly or the wrong property is changed, it is relatively dif-ficult to pinpoint the feature that is causing this.

Average Signature Calculation (Figure 5.4)

The next process is that of calculating and caching the average signature of each data source. The average signature is needed to enrol or authenticate a user. Caching it makes sure it does not have to be reconstructed every time either of those activities are performed. As every data source has only one average signature, the decision was made to store the average signatures in one storage entity. As a consequence, it is not necessary to create a new storage entity for the average signature when adding a new data source.

Running Experiments (Figure 5.5)

Upon ending the average signature calculation, the main component will start the experiments. The main component passes the EntityManager to a “SignatureProvider” which takes care of providing enrolment and authentication samples when needed. After this, the main component passes a reference of this provider, the settings (the ECC to be used and the amount of enrolment samples to be used) and the EntityManager to the Experiment object. The Experiment object has no knowledge of the type of SignatureProvider, but only of its interface. This decision makes it easy to change the implementation or addition of a new SignatureProvider without changing the Experiment object as well. The Experiment object passes the ECC and enrolment samples (SignatureSample[ ]) to the enrolmentManager, which converts them to user templates and stores them in a storage entity. The error correction capacity of the user templates is encoded into them by the Reed-Solomon provider at the request of the EnrolmentManager. At the request of the AuthenticationManager, the Reed-Solomon provider tries to correct mistakes in the provided authentication attempt data. Afterwards, a reference to an “AuthenticationManager” is passed to the calculators. This object is connected to the Enrolled Users storage entity and can thus access the templates of the enrolled users for use in authentication requests. The access to these storage entities allows the calculators to perform authentication attempt simulations. The

(21)

Figure 5.3: A model of the data flow for the process of caching signatures into a storage entity after importing their points into the system.

Figure 5.4: The data flow during the process of calculating the average signatures of the different data sources

results of calculating the specific metrics is passed back to Experiment object, which stores the results into the Experimental Results storage entity.

5.2 Construction

This section describes the implementation details of the prototype that has been built based on the description in the article and the requirements and constraints in the previous section.

(22)

Figure 5.5: A model of the data flow during the process of running the experiments to calculate the match rate of the system in a certain configuration

5.2.1 Technological Choices

The application has been built in Java. The motivation for this choice is the abundance of libraries and community support available for this programming language as well as a possibly easier construction of the mobile prototype using Android SDK, which uses Java. Additionally, the platform independence of the JVM weighed in this decision due to the technologies used at bunq. The RDBMS used for providing persistence services throughout the application (e.g. the caching of signatures or stroring the user templates) is MySQL, which is available free of charge and is open-source. For mapping objects to relational entities, the Hibernate framework is used, as it provides many features out of the box and has a minimal configuration, which makes it easy to get it up and running on a short notice. The application dependencies are managed using Maven as this simplifies building the application: missing dependency JAR’s are downloaded automatically from the Maven repository and packaged once the system is published.

5.2.2 Implementation Change: Error Correction Codes

The BCH error codes have been replaced by Reed-Solomon codes. Specifically the reason is the following: a Python RS library, that is better documented and easier to port to Java than the available BCH library (written in C). To replicate the behaviour of the original system, that uses BCH codes, as accurately as possible, an implementation must be chosen that replicates this behaviour.

As Reed-Solomon codes work on non-binary alphabets (bytes, in this case) unlike BCH codes (which work on bits), there are two straightforward adaptations of the system to incorporate these RS codes:

1. Given the 63 feature bits of the system, they can be grouped to form bytes (symbols). As such, RS will be used optimally in terms of memory as the 63 bits are encoded in a codeword that is as small as possible (see Figure5.6).

(23)

Figure 5.6: Using RS to encode multiple bits in one byte codeword

2. Given the 63 feature bits of the system, each of these bits can be appended to 7 random bits to form 63 bytes. These can then be encoded as separate keywords. Consequently, this results in higher memory usage than the former method (see Figure5.7).

Figure 5.7: Using RS to encode one feature bit in one byte codeword

Although the first method is more efficient in terms of memory, there is a problem to control the error correction capability at the same level as the BCH codes. As RS works on non-binary symbols (in this case 8 bits per symbol), an error correction capability of 1 can result in up to 8 bits being corrected, depending on whether the bits are grouped into the same code symbol or distributed across the codeword uniformly. Although this is normally a good property when there is a high likeliness of error in close succession, it is clearly an undesirable property if the goal is to mimic the behaviour of BCH codes, which correct up to one bit given an ECC of 1. Given that our goal is indeed to mimic the behaviour of the original system that uses BCH codes, the second method is implemented.

5.2.3 Implementation Change: Threshold Cryptography

Due to the non-explicit description of the use of threshold cryptography, combined with non-explicit claimed advantages of its use, no natural application of the technique within the architecture came to mind. The following scenario’s were considered, bearing the author’s claims (security against corruption, modification and interception of user templates):

Scenario 1 (Figure5.8)

Enrolment When a client wants to enrol, it encrypts the enrolment data before sending it to the Matcher, using the Matcher public key. The matcher decrypts the data, uses it to create the user template and stores and encrypted version using the shared public key of the databases.

Authentication When a client wants to authenticate, it encrypts the authentication data before sending it to the Matcher, using the Matcher public key. The matcher decrypts the data and requests the partial encryptions of the user profile from the databases. The matcher combines the partial decryptions, which results in the user profile. This way, only the matcher can access the user profile, an adversary would have to corrupt the decryptions of at least n out of k databases to make authentication impossible.

(24)

Problems The problem with this approach is that when the databases output their partial decryptions, an adversary listening in to this communication can also collect the partial decryptions, hereby gaining information about the user profile.

Figure 5.8: Threshold cryptography Scenario 1 and 2

A second scenario is similar to scenario 1, but instead of encrypting the message and using shared decryption, the scheme makes use of shared signing by the databases to authenticate the data they send. The idea is that the signing mitigates corruption and modification. The problem with this scheme, however is that again an adversary can listen in on the connection between the databases, thereby making the system vulnerable to the interception of the user template.

Yet another scenario distributes the information of users among different databases. The idea behind the scheme is that if one database is compromised, only part of the information will be available to the attacker. The problem with this scheme is, however, that this is not true threshold cryptography: the failure of one database would make the system unusable as all pieces of information are needed for authentication to succeed.

In conclusion, as no scheme was found using threshold cryptography that upholds all of the claimed security properties described in the method, no such scheme was implemented.

This should not influence the performance metrics (FRR, FARrf and FARsf), since the encryption

and decryption techniques mentioned in the paper do not influence the information a message contains (a decrypted message is the same as the original, if it has been correctly decrypted). As such, the only penalty for using these cryptographic techniques should be a runtime penalty: the signing, encryption and decryption of data takes time.

Contextually, a simplified security scheme can be implemented without using threshold cryptog-raphy and distributed databases. The client could encrypt the data that is sent to the server using public key cryptography, thereby protecting against eavesdropping opponents during the enrolment and authentication phase. Specifically, this protection applies to the communication line between

(25)

Figure 5.9: Threshold Cryptography Scenario 3

client and server. Note that this scheme does not protect against template corruption as the op-ponent could intercept the cipher texts between server and client and corrupt them, thus possibly making the system unusable. Additionally, this scheme assumes the server is trustworthy and that the connection between server and database can not be eavesdropped.

5.2.4 Application Modules

This section describes how the responsibilities of the system were divided among objects per module and why these design decisions were made. The figures that depict the submodules of the entity module do not include the objects that are mapped to database entities by the Hibernate framework. These objects are omitted as they offer no added value to this documentation: they contain only the elements required by the hibernate framework (an empty constructor, getters, setters and annotations)

Signature (figure 5.10)

This group of classes models a signature, independent of any database. The signature has a SignatureFeatures object that encapsulates a collection of IFeatures, the interface that every signature feature must implement. This makes it easier to add more features and to delegate feature operations to the features themselves. As such the implementation is hidden from the signature object, thereby increasing the maintainability. SegmentalFeature models features that work on segments of the signature. It is an abstract class as all the segmental features share segment calculation as common functionality. As such, the amount of code duplication is lessened and it becomes easier to add a new segmental feature without thinking of how the signature segments are calculated.

Experiment (figure 5.11)

The experiments are encapsulated in this module. Experiment objects are independent of sig-nature source, they expect an ISigsig-natureDataProvider (figure5.15) to provide them a means to get the signatures used in the experiments. As such experiments are reusable for any signature database. As the metrics are relatively complicated, their calculation is encapsulated in one class for each metric, resulting in a cleaner and shorter experiment class.

(26)

Figure 5.10: Module: Signature

Figure 5.11: Module: Experiment

Error Correction (figure 5.12)

The module responsible for error correction has a fairly simple interface, it encodes messages and tries to correct them. The finite field arithmetics needed for these operations are encapsulated in the GaloisFieldArithmetics class. Although the structure of this module is fairly simple, separating it from the other modules allows for easy replacement. This is desirable in the scenario that other forms of error correction have to be implemented. Moreover, the module can, in this state, be effortlessly imported into other projects that require error correction.

Entity - SignaturePoint (figure5.13)

These classes are responsible for providing communication means with the database (to retrieve a certain part of the signature points). As the signature points are processed in a similar way (the same signature features are extracted from them, regardless of which public database they come from) they are bound to the same interface (IPointData). The interface is generic, since the classes do not return the same type, but a similar one: they return the hibernate entity objects that are roughly the same, but are mapped to different database tables and thus must be handled separately. Making this interface generic makes it easier to add a new kind of PointData

(27)

Figure 5.12: Module: Error Correction

(for example, from a different provider) as it is explicit what interface a new provider should accommodate.

Figure 5.13: Module: Entity - Signature Point

Entity - Experiment and User Template (figure 5.14)

The experiment and User template entities do not require much explanation due to their reduced complexity. This is the reason they were put together in this description. The UserTemplate-Manager provides functionality to insert, delete and retrieve user templates. As such, it is the only class that offers user template persistence. Similarly, the ExperimentDataManager provides persistent storage for experimental results.

Figure 5.14: Module: Entity - Experiment and UserTemplate

Entity - Signature (figure 5.15)

The signature data provides access to signatures stored in the database. All signature databases implement the same interface so an experiment can then be implemented against an interface instead of an implementation. This makes for a more extensible application.

Entity - Queries (figure 5.16)

(28)

Figure 5.15: Module: Entity - Signature

that is used to identify said query. In order to prevent typing the string multiple times and to enforce uniqueness among queries, (static final) query names are built up of a prefix that is unique for each entity and represents the entity that the query interacts with. To this prefix a period ( ‘.’ ) is added followed by a descriptive name of what the query does.

Figure 5.16: Module: Entity - Queries

Data Import (figure5.17)

This module is responsible for parsing data from the public databases into usable data in the form of signature points. Parsing the points is fairly straightforward with the exception of the MCYT database in which the data was provided as little endian binary files (java uses big endian) and a Matlab file that parsed the data using filepointer operations. The relevant part of the parsing program has been ported into a separate class, namely FPGFileParser, FPG being

(29)

the filetype of the binary files. Although significant similarity can be seen between the classes, the decision was made not to use a generalised interface (example: IUserPoints). The reason being that the classes have different inputs and outputs and that java does not support covariant generics: a list of cats is not a subtype of a list of animals. As such, cats can not be added to a list of animals. Of course, an array could have been used to work around this problem, but compile time checks would have to be sacrificed, making the runtime reading unsafe. It was decided that the sacrifice of a small piece of code reuse would be the lesser of these two evils.

Figure 5.17: Module: Data Import

Authentication System (figure5.18)

Enrolment and authentication happens in the Authentication System Module. ByteUtils is a library class that provides conversion functionality from and to unsigned bytes. This is neces-sary as the Reed-Solomon error correction operates on finite fields of 0-255, but the Java byte datatype has a range of -128 to 127. The HelperData class encapsulates the creation of helper data which is used to determine whether a signature is close enough to the signature with which the user was enrolled without a direct comparison. The enrol function takes an integer as the user Id and an array of Signature of which it takes the average to enrol a user.

(30)

5.3 Testing

As for testing, a testing project was made mirroring the packaging of the system. The testing frame-work for writing and running the unit tests is JUnit: the Java instance of the xUnit architecture for unit testing. In particular the testing effort was focussed on the feature extraction in order to gain confidence that the extracted features were correct implementations of the formal feature definitions (these can be found in the appendix). Additionally, the error correction capability was tested in order to check for correct behaviour of the Error Correction Module.

5.4 Extension to Mobile Devices

As for the mobile extension, an implementation of the client-server model was chosen. Here the client is a mobile device and the server is a device with access to the database. The only change to the original architecture is the replacement of the input component. Instead of a wired machine (in the prototype this was replaced by the data import component), the idea of the mobile version is that it features an API that provides enrolment or authentication services to mobile devices that connect to it. As it is optimised for experiments, the implemented mobile version does not provide an authentication service. Instead, the authentication part of the API saves the signature data as authentication samples to be reused in the experiments. This was done to facilitate the reuse of the input data for testing the performance of the system under different settings.

5.4.1 API

A REST service was chosen over a SOAP service as the former is based on a more loosely coupled design [38]. The “RESTful web service” framework of Spring was used as it has good support and is relatively easy to set up, given an already existing project. One consequence of using the Spring framework was that the use of Spring Data would simplify the configuration and make the application code cleaner: it features an extra abstraction over databases, namely repositories which are interfaces that are automatically implemented based on the entity classes they use and the method names and return types within these “repository” interfaces. As mentioned before, the current implementation is optimised for reusing the authentication data for experiments. The adaptation needed for the API to provide the authentication service mentioned before is to change the the authentication controller to propagate the request to the authentication module and propagate the result, which is a boolean representing whether the authentication attempt was successful, back to the client.

5.4.2 Android App

The Android app provides the enrolment functionality. The user is presented with a canvas onto which he draws his signature 10 times (one session). After this the enrolment request is sent to the server along with the username. Additionally, there is a forgery mode, where a forger can pick the name of the user whose signature he is trying to forge. After a number of attempts, the forgeries are sent to the server to be used as forgeries for that signature in the experiments.

One problem in the first version of the app is that that the touch events that resulted from the signatures were not smooth. Although hardware limitations were the first suspect, the Android source states that android batches touch events. The touch event getters only returns the last measured touch event. Android does, however provide a way to get all the touch events batched in the one that is fired (getHistoricalX(int)). Upon including these batched events, the signatures became much smoother and the sampling rate became acceptable.

Another problem is that in general smartphones that measure touch through capacitive touch screen do not measure pressure (the displays rely on the electrical properties of the human body to detect when and where on a display the user touches)). As such, many phones can only give an estimation of the pressure. The Nexus 5, however has a built-in pressure meter and does provide this functionality. To send messages to the REST API, the Robospice Android library has been used. The library provides serialisation and deserialisation functionalities for java objects. Additionally, the library handles asynchronous tasks on android gracefully and has out of the box compatibility with Spring

(31)

Rest. In short, the use of this library makes communication with certain REST services (of which Spring Rest is one) relatively painless as it provides a means to abstract robustly from asynchronous messaging on Android.

(32)

Chapter 6

Performance Results and

Discussion

In this chapter the most relevant performance results of the experiments are presented. The rest of the experimental results have been added to the appendix. Additionally the results will be discussed. The results reported in the original paper can be found in Table6.1. Here, the ECC stands for the error correction capacity in number of symbols (1 symbol = 1 byte = 1 encoded bit), the definition of the abbreviations in the Table are discussed in section2.1.

Table 6.1: The reported results of Maiorana and Ercole (left: 5 enrolment samples, right: 10 enrolment samples)

ECC FRR FARRF FARSF

10 14.33% 9.55% 24.44% 11 10.86% 10.96% 27.33% 13 4.20% 14.36% 36.66%

ECC FRR FARRF FARSF

10 13.80% 10.48% 32.33% 11 11.20% 11.80% 34.66% 13 6.13% 15.15% 40.33%

6.1 SVC2004

Firstly, the results of SVC2004 are presented. Table6.2 shows the match rates of the system using the system settings that were used in the article by Maiorana and Ercole. Additionally, Figure6.1

shows the effect on varying the ECC on the match rates.

Table 6.2: Replicated match rate results on SVC2004 (left: 5 enrolment samples, right: 10 enrolment samples)

ECC FRR FARRF FARSF

10 38.75% 0.07% 16.12% 11 32.75% 0.14% 20.5% 13 22.25% 0.44% 32.75%

ECC FRR FARRF FARSF

10 33.38% 0.08% 14.00% 11 25.12% 0.13% 18.50% 13 17.12% 0.47% 28.25%

6.1.1 Discussion

When compared to the performance reported by Maiorana and Ercole, there are a few differences. The false rejection rates are higher and the false acceptance rates under skilled forgeries are lower. This suggests that the system is behaving more strictly than was the case for the original system. It might thus be the case that the system needs to be calibrated to achieve the same performance. Figure 6.1 shows how the system behaves when varying the ECC setting. To find out whether the

(33)

Figure 6.1: the performance results of the system on the SVC2004 database. 0 10 20 30 40 50 60 70 80 90 100 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930 Error correction capacity

Enrolment Samples: 5 FRR % FARsf % FARrf %

0 10 20 30 40 50 60 70 80 90 100 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930 Error correction capacity

required calibration exists, the experimental results where filtered for the setting that featured an FRR equal to or lower than the reported FRR. The results of this search can be found in Table6.3. Given these results, it can be concluded that the system performance is lower than the reported result with regard to the FARSF in all cases that use 5 enrolment samples. As for the 10 enrolment sample

case only when calibrated for an FRR of 13.80% does the system perform similarly to the report. Interestingly, the system performs better in most cases with regard to FARRF. The only time the

performance is not better than reported for this metric, is when it is within the similarity threshold of 10 percent point (as defined earlier in the Methods chapter).

Table 6.3: SVC2004 performance with FRR calibrated on reported result by Maiorana and Ercole (5 enrolment samples left, 10 enrolment samples right)

Calibrated FRR ECC FARRF FARSF

≤ 14.33% 15 1.15% 42.75% ≤ 10.86% 16 1.84% 47.75% ≤ 4.20% 19 5.75% 66.25%

≤ 13.80% 14 0.79% 34.38% ≤ 11.20% 16 1.93% 45.75% ≤ 6.13% 19 2.80% 52.25%

6.2 SUSIG

Similarly to SVC2004, the results of SUSIG are presented. Table 6.4 shows the match rates of the system using the system settings that were used in the article by Maiorana and Ercole. Additionally, Figure6.2shows the effect on varying the ECC on the match rates.

Table 6.4: Replicated match rate results on SUSIG (left: 5 enrolment samples, right: 10 enrolment samples)

ECC FRR FARRF FARSF

10 66.60% 0.02% 4.95% 11 58.61% 0.06% 6.52% 13 42.23% 0.28% 12.31%

ECC FRR FARRF FARSF

10 57.34% 0.02% 2.63% 11 49.36% 0.6% 4.10% 13 33.93% 0.24% 9.80%

6.2.1 Discussion

Like the SVC2004 database, the match rates from the SUSIG database differ from the reported results on the given settings. Figure6.2shows that this might be a calibration issue. Thus, in a similar way

(34)

Figure 6.2: the performance results of the system on the SUSIG database. 0 10 20 30 40 50 60 70 80 90 100 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930 Error correction capacity

Table 6.5: SUSIG performance with FRR calibrated on reported result by Maiorana and Ercole (5 enrolment samples left, 10 enrolment samples right)

≤ 14.33% 18 3.18% 34.84% ≤ 10.86% 19 4.60% 40.53% ≤ 4.20% 23 15.65% 68.32 %

≤ 13.80% 17 2.07% 28.74% ≤ 11.20% 18 3.09% 34.74% ≤ 6.13% 20 6.35% 47.68%

to the SVC2004 database, the results were searched for a setting that features the reported FRR or lower, resulting in Table6.5. Again, the results show that the system performs better than reported in all cases when looking at the resulting FARRF. Looking at the FARSF however, the results suggest

that the system shows lower performance than reported for 5 enrolment samples. This is not the case when 10 enrolment samples are used: when calibrating for an FRR ≤ 14.33%, the corresponding false acceptance rate is 28.74% (an 11% decrease of the reported value). This value is within the threshold of 10 percent point and will thus be considered the similar to the reported value. The same similarity can be concluded when the system is calibrated for an FRR of 11.20% (a difference of 0.08 percent point). At an even lower FRR (6.13%), the system performs lower than the reported results, yet is within the threshold. Thus, the performance shall be considered similar.

6.3 MCYT

As for the MCYT database, the performance results will be presented in a similar way to the previous databases, namely in Table 6.6. Additionally, the effect of varying the ECC on the results can be observed in Figure6.3.

Table 6.6: Replicated match rate results on MCYT (left: 5 enrolment samples, right: 10 enrolment samples)

ECC FRR FARRF FARSF

10 77.52% 0.07% 2.48% 11 73.52% 0.14% 3.72% 13 60.96% 0.49% 7.04%

ECC FRR FARRF FARSF

10 75.28% 0.05% 1.96% 11 69.04% 0.11% 2.92% 13 54.27% 0.38% 6.60%

(35)

Figure 6.3: The performance results of the system on the MCYT database. 0 10 20 30 40 50 60 70 80 90 100 1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930 Error correction capacity

6.3.1 Discussion

Similar to the results of the other datasets, the system seems to be too strict. The same FRR calibration is used as in the analysis of the results of the other datasets. The results of this calibration are shown in Table6.7.

Upon examining the results of using 5 enrolment samples the following conclusions can be drawn:

• In all cases, the FARRF is within the similarity threshold

• In all cases, the FARSF is outside of the similarity threshold, in particular, the ratings are much

higher than reported, i.e. the system performance is worse than reported.

This is not the case when using 10 enrolment samples. Other than when the system is calibrated for an FRR of ≤ 6.13% (which results in an FARSF that is 0.26 percent point above the similarity

threshold), the system match rates are within the similarity threshold and can thus be considered similar to the reported performance.

Table 6.7: MCYT performance with FRR calibrated on reported result by Maiorana and Ercole (5 enrolment samples left, 10 enrolment samples right)

≤ 14.33% 21 11.43% 43.28% ≤ 10.86% 22 14.74% 49.24% ≤ 4.20% 24 23.18% 61.76%

≤ 13.80% 20 7.94% 37.88% ≤ 11.20% 20 7.94% 37.88% ≤ 6.13% 22 13.92% 50.56%

6.4 Mobile

This section presents the experimental results of the Mobile prototype performance. Table6.8shows the match rates of the system using the same settings as Maiorana and Ercole in the replicated article.

Table 6.8: Match rates of the mobile database with the same settings as the replicated system (left: 5 enrolment samples, right: 10 enrolment samples)

ECC FRR FARRF FARSF

10 57.20% 0.16% 3.2% 11 48.80% 0.37% 4.2% 13 36.8% 1.01% 13.8%

ECC FRR FARRF FARSF

10 57.00% 0.21% 2.00% 11 49.2% 0.38% 3.60% 13 36.60% 0.99% 10.60%

Mobile Biometric Authentication System