Performance of Face Recognition Algorithms on Mobile Devices

(1)

Performance of Face Recognition

Algorithms on Mobile Devices

Karl-John Chow

karl.chow92@gmail.com

August 17, 2015, 45 pages

Supervisor: Tijs van der Storm

Host organisation: bunq b.v.,http://www.bunq.com

Universiteit van Amsterdam

Faculteit der Natuurwetenschappen, Wiskunde en Informatica Master Software Engineering

(2)

2.3.3 Algorithm . . . 14 2.4 Match rates . . . 15 3 Experimental Setup 16 3.1 Databases . . . 16 3.1.1 Preprocessing . . . 18 3.2 Experiment One . . . 18 3.2.1 Database . . . 19 3.2.2 Devices . . . 19 3.2.3 Setup . . . 19 3.2.4 Extraction. . . 20 3.3 Experiment Two . . . 21 3.3.1 Database . . . 22 3.3.2 Context . . . 22 3.3.3 Setup . . . 22 3.3.4 Expectations . . . 22 3.3.5 Extraction. . . 23 4 Prototype 24 4.1 Requirements . . . 24 4.2 OpenCV . . . 24 4.3 Face Recognizer. . . 25 4.4 Core structure . . . 25

(3)

4.4.1 Algorithm Interface . . . 26

4.4.2 Subject Label Manager . . . 26

4.4.3 Interface Subject Database Model Factory. . . 26

4.4.4 Subject Database Model. . . 27

4.4.5 Arguments Object: Database . . . 27

4.4.6 Visitor and Factory Pattern . . . 27

4.5 Prototypes . . . 28

4.5.1 Java common project . . . 28

4.5.2 Android Prototype . . . 28

4.5.3 iOS Prototype . . . 28

4.5.4 Computer Prototype . . . 29

4.6 Testing . . . 29

5 Results and Evaluation 30 5.1 Experiment One . . . 30 5.1.1 Eigenface algorithm . . . 30 5.1.2 Fisherface algorithm . . . 32 5.2 Experiment Two . . . 33 5.2.1 Eigenface algorithm . . . 33 5.2.2 Fisherface algorithm . . . 35 5.3 Evaluation. . . 37 5.3.1 Eigenface algorithm . . . 37 5.3.2 Fisherface algorithm . . . 38 5.4 Discussion . . . 38

5.4.1 Context and Usability . . . 38

5.4.2 Trusting the device. . . 39

5.4.3 General . . . 39

6 Conclusion 40

7 Future work 41

Bibliography 43

(4)

Abstract

If a biometric authentication method is used to authenticate a person it becomes less likely to forget the password of the user since it is attached to the body of the user or it is the behavior of the user, but can these authentication methods work on mobile devices? In this thesis an attempt has been made to research the performance of the eigenface and fisherface algorithm on several mobile devices. Former researches states that the implementation usually only covers the prediction of the algorithm. This research covers the enrollment of the algorithm.

The results of this research show that the eigenface algorithm can run the enrollment phase on the selected mobile devices. The results also show that the fisherface algorithm needs more investigation. It also does not perform as well as shown by other research on the data of the experiments of this thesis.

Keywords Biometric authentication, face recognition, mobile, software engineering, eigenface, fish-erface

(5)

Chapter 1

Introduction

1.1 Overview

“Biometrics is a method of recognizing a person based on his/her unique identification”[18]. Biometrics is an identification method which can be used as a security method. This security method differs from other security methods that can be described as something that we have (a card key) or something that we know (a password), biometrics can be described as something that we are[11,12]. There are several biometric authentication methods that have been researched, it became a well-studied area of research[6, 21, 19]. It is divided into two categories, physiological biometrics and behavioral biometrics[1,11]. Physiological biometrics are authentication methods which capture some-thing that what we are, for example: facial recognition, voice recognition, finger scan and iris scan. Behavioral biometrics are methods which capture our behavior by doing an instruction, for example: signature scan, keystroke scan and gait recognition.

A mobile device is nowadays a commonly used medium to communicate with each other, it is also very common to log into a chat application or to make a financial transaction, for the login process and the transaction it is necessary to identify the right person. The traditional way to authenticate, on a mobile device, is to fill in a password[19]. The drawback of a password is that it may be forgotten. If the password is forgotten the user usually needs to follow a password reset procedure which can be inconvenient.

If the biometric authentication method is used to authenticate a person it becomes less likely to forget the password since it is attached to the body of the user or it is the behavior of the user, but can these authentication methods work on mobile devices?

1.2 Problem statement and motivation

A common biometric authentication method usually consists of two phases. The first phase is the enrollment phase where biometric samples are collected. For example, collecting samples for face recognition could be done by taking photos of the faces of subjects. When the samples are collected the algorithm creates a template based on the samples which represents an average for every person. In the second phase, which is called the identification phase, the method collects one sample which gets saved as a template where this template get compared to the average templates created in the first phase. If the comparison of the template of the second phase is close enough to one of the templates from the first phase the method says that it is the person which has been enrolled[5,11].

The first phase of the algorithm usually takes a longer time to be computed because more templates need to be computed into one template. In [5, 3, 21] Darwaish et al., Chen et al. and Vazquez-Fernandez et al. presents their implementation of offline face recognition with the use of local binary patterns on various Android and Nokia devices. These papers describe that the whole process is done on the device, which means enrollment and identification. These papers show that it is possible to do face recognition on mobile devices.

(6)

In [6] Guillaume et al. present their implementation of the eigenface and fisherface algorithm. The algorithms were first profiled in MATLAB and then implemented on various Android devices. The results of the research show that a recognition rate of 84.3% on the eigenface algorithm and 94.0% of the fisherface algorithm can be achieved. The conclusion also stated the Equal Error Rate (EER) of the eigenface algorithm which was 25.0% and the fisherface algorithm which was 35.0%. Both EER indicates that the fisherface algorithm had a better performance in recognition.

However, Guillaume et al. trained the algorithms in MATLAB and then stored the templates on the devices to do face recognition. Since the templates are stored and not trained on the device itself curiosity grew towards the achievability of the enrollment phase with the eigenface and fisherface algorithm on mobile devices.

In this thesis the achievability of offline face recognition on mobile devices with eigenface and fisherface algorithm is researched. Achievability is described in this thesis as the performance of the algorithm and the execution time and memory usage of the enrollment phase on a mobile device. The performance is measured in a percentage which represents the total success rate of an algorithm, the execution time is measured in milliseconds and the memory usage is measured in megabytes.

The problem, however, is that it usually takes a longer computation time to execute the enrollment phase, but how long is an user willing to wait till a process is finished on a mobile device? The sub-section Acceptable timespan in the related work sub-section in this chapter provides a maximum execution time of 9 seconds. The amount of time taken for the enrollment phase is also set for 9 seconds in this thesis.

To verify if the algorithms are implemented correctly, a verification threshold is needed. The subsection Correct implementation in the related work section shows why this threshold is defined as 90% for both algorithms.

1.3 Research Questions

These problems propagate to the following research question:

“Is it possible to get a total success rate of 90% for a given training set where the execution time of the enrollment phase is 9 seconds or less on a mobile device with the eigenface and fisherface algorithm?”

To help to find answers several subquestions are defined which are listed below. • How many images could be enrolled on mobile devices?

• How much training is required per algorithm to have a total match rate of 90% for a given training set?

1.4 Bunq

This research has been hosted by bunq1_{. Bunq is a startup company with a banking license located in}

Amsterdam. This research has been done because bunq was interested in face recognition on mobile devices. They wanted to know if Android and iOS devices can perform offline face recognition.

1.5 Contribution

There are 3 contributions presented in this thesis:

iOS Prototype In [5] and [6] the prototypes are only implemented on Android devices, in this research iOS devices will also be included.

(7)

Limit This thesis will describe if the eigenface and fisherface algorithm can run natively on mobile devices. Therefore a limit is described in the number of images that can be trained in a timespan of 9 seconds.

Test Approach A different perspective of testing the performance of the eigenface and fisherface algorithm is introduced in this thesis. In [2, 5, 6] all the experiments consist of one test set where the variety lies in the variables of the algorithm. In this research instead of changing the variables, the datasets will be different to investigate other results.

1.6 Research Method

To investigate if the eigenface and fisherface algorithms can do offline recognition, the empirical research method is used. To achieve this, 2 experiments are set up. The first experiment measures the execution time of the enrollment phase. The second experiment measures the total success rate of the algorithms. Therefor prototypes are setup that meets a list of requirements described in chapter

4.1 to run the experiments. The Dalvik Debug Monitor Server and Instruments are used to gather the data and Numbers from Apple is used to represent the data in graphs. These tools are extensively discussed in section3.2.4.

1.7 Related work

In this section, different approaches and other existing technologies affected the work of this thesis are described.

Preprocessing

Principal component analysis is usually used in face recognition to extract the features but this is easily affected by lighting and facial expression. This affects the recognition rate in a bad sense. If preprocessing is applied the recognition rate will improve[9]. In this thesis several preprocessing methods are used based on what is commonly used according to [9] and what was used to analyze the impact of preprocessing techniques in [7]. [7] showed that the recognition rate increases by 29.85% till 56%.

Acceptable timespan

In [10] Hoxmeier et al. presented a research on the response time of a system and user satisfaction where their prototype is a browser-based application. They conducted an experiment where they monitored the behavior of 5 groups of subjects. The first group experience no response time of the application, the second group experience 3 seconds, the third 6 seconds, the fourth 9 seconds and the last group 12 seconds. The subjects were asked to use the prototype where afterwards a survey gets filled in. Based on that survey several variables were measured. One of the measured variable in the research was the satisfaction of System speed, the satisfaction is measured with a number which scales from 1 to 10 where 10 is completely satisfied to 1 totally not satisfied. The results of this measured variable showed that the subjects satisfaction of were not satisfied when the system took 9 seconds or longer. Based on this research the maximum timespan is defined for this thesis which means that the enrollment phase cannot take longer than 9 seconds.

Fisherfaces

In [2] Belheumer et al. presented a new linear discriminant based face recognition algorithm: the fisherface algorithm. With the fisherface algorithm an attempt has been made to improve face recog-nition in different lighting condition. The results shows a recogrecog-nition rate of 90% or higher for a specific training set. This thesis tries to get the same recognition rate in another context. Different sets of faces are defined to train the algorithm to see if the results are the same.

(8)

Offline Face Recognition with Local Binary Patterns

In [5, 3,21] Darwaish et al., Chen et al. and Vazquez-Fernandez et al. presented an implementation of a face recognition algorithm using local binary patterns on Android and Nokia devices. All papers show that the face recognition works offline with local binary patterns. These implementations were the motivation to make an attempt to show that other algorithms work too.

Offline Face Recognition with the Eigenface and Fisherface algorithm

In [6] Guillaume et al. implemented the eigenface and fisherface algorithm on Android devices where the enrollment phase was done on a computer where the classifier was later stored on the devices to classify. This paper showed that prediction was possible for both algorithms. This thesis shows an attempt to execute the enrollment phase possible on mobile devices.

Offline Face Recognition based on Sparse Representation

In [4] Chou et al. presented face recognition based on sparse representation on mobile devices. Sparse representation is based on eigenface which gives a clue that offline recognition with eigenface also might work. The results show that the recognition rate for this algorithm is above the 99%.

Offline Face Recognition with Fisherfaces using the internal GPUs

In [8] El-Mahdy et al. implemented an offline face recognition algorithm using fisherfaces on the graphical processing unit of a device using OpenGL. They made a comparison on 2 devices between the face recognition implementation on the graphical processing unit and the central processing unit by measuring the execution time and memory. These variables are also used in this thesis to measure the performance of a device. The results of the paper show that the GPU performs 10 times faster than the CPU and consumes 7.2 times less energy.

Correct implementation

A correct implementation of the algorithm is crucial. The verify if the implementation is correct or not a threshold has been set based on the total success rate. Investigation has been done on [2,6,20,15,23] where these researches at least achieved a success rate of 90% with both algorithms. This percentage is set as a threshold for this research to verify if the implementation is correct or not. Experiments

The experiments in this thesis are based on the experiments in [5], they defined a database to do the experiment but in this thesis multiple databases are used where some databases have subsets. They measured the algorithms False Acceptance Fates (FAR), FAR further described in section2.4. These metrics are also used in this thesis to see if the algorithm performs or not. In [21] a performance analysis is done on algorithms by measuring the execution time of individual components of the algorithm. This is also done in this thesis while also monitoring the memory usage.

1.8 Outline

This thesis is organized as follows. In section 2, the general face recognition process, eigenface and fisherface algorithms and the way of measuring the performance of these are extensively explained. In section 3, the experimental setup is described with the theory from section 2. This section covers the setup of the experiment, the used databases, the preprocessing of the images, the chosen devices, the data extraction and the tools to extract the data. In section 4, the implementations of the 3 prototype are described with requirements testing and diagrams that helps the experiments to get results. In section 5, the results are presented, evaluated and discussed. In section 6 a conclusion is drawn from the results and evaluation and lastly the future work is presented.

(9)

Chapter 2

Background

In this chapter the face recognition phases are first explained. Secondly the eigenface and fisher-face algorithms are described with the associated math. At last the calculations of match rates are explained.

2.1 Face Recognition

As mentioned in section1.2, a biometrics process consists of 2 phases, the enrollment phase and the identification phase. The enrollment phase is also known as the training of an algorithm and the identification phase is also known as prediction of an algorithm[11, 18].

A face recognition algorithm consists of 3 steps in the enrollment phase[13,18]: 1. Training images

The first step in the enrollment phase is to acquire images of the faces. This can be done by making photos of the subjects for example.

2. Feature extraction and template creation

In the second step the features gets extracted and the templates are created for each image. 3. Distribute to the correspondents

The templates are calculated they are sorted by subject. After that, a mean template is created from all the templates per subject which gets compared with the input of the identification phase.

The global flow of the enrollment phase is shown in Figure2.1.

(10)

In the identification phase of a face recognition algorithm a template will be created from the input image where this template gets compared to the templates that are created in the enrollment phase to determine who the identifying subject is.

The identification phase consists of 3 steps[13]: 1. Face detection

In the first step the algorithm tries to determine if the input is a face or not. 2. Feature extraction and template creation

If the input is a face, the algorithm also extracts the feature and creates the template. Then it calculates a set of weights of the template of the input image and the templates from the enrollment phase templates. These weights represent the parts of the template of the input image and the templates from the enrollment phase. If the input is not a face the prediction will not be made.

3. Prediction

The last step is the prediction where the person is identified based on the weights of the tem-plates.

The global flow of the identification phase is shown in Figure2.2.

Figure 2.2: Flow of the Identification Phase

2.2 Eigenface Algorithm

The eigenface algorithm uses eigenfaces to form a template to do face recognition. Eigenfaces represent the principal components of a set of eigenvectors.

2.2.1 Principal Components Analysis

Principal component analysis (PCA) is a linear transformation technique to reduce dimensionality[13]. This technique is to get the principal components out of set of data. Principal components are presented as vectors through scattered data points in a plot. These vectors are called the eigenvectors, the number of scaling of a vector is called the scalar, also known as eigenvalue that belongs to the eigenvector. The number of variables presents the number of eigenvalues and eigenvectors during a measurement. Each eigenvector represents an components axes which maximizes the variance of the data. See Figure2.3for an example where the triangles and circles represent values in datasets and the arrows indicate the vectors.

This example has 2 datasets where both X and Y are measurements. For each dataset, N number of principal components can be drawn where N is the number of variables measured. This means that there are 2 eigenvectors with 2 eigenvalues for this example.

(11)

Figure 2.3: Example of the Output of the PCA

An eigenvector is a point in a covariance matrix which represents an important feature for the template. Eigenfaces are used to[13, 17]:

• Extract relevant facial information. By having this information an analysis can be made based on the human intuition of face features such as eyes nose and lips or an statistical variation between faces images can be made to compare them.

• Represent face images efficiently, it reduces space complexity and computational power, each face can be represented using a small number of dimensions.

2.2.2 Eigenvalues, Eigenvectors and Eigenfaces

Vectors are first described to make eigenvectors and eigenvalues easier to understand. A vector is a representation of an linear transformation which can be represented as:

~ v = 1

−2

(2.1) Where the first number is represented as the transition along the x-axis and the second number is represented as the transition along the y-axis[17].

An eigenvector is a scaled vector. The eigenvalue is the value which is the scalar of the vector represented by λ. To determine λ, consider an n × n matrix A where a vector ~v acts on. This vector is called the eigenvector of A if A~v is a scalar of multiple of ~v, see equation2.2[17]. In this equation the assumption is made that ~v is not ~0, since a vector with ~0 does not move[2] .

A~v = λ~v (2.2)

The easiest way to find λ is to find the zeros out of equation 2.2. This is done by bringing the formula on the left hand side to the right hand side of the equation where the equation becomes equation2.3

~0 = A~v − λ~v (2.3)

After that the identity matrix, In, can be extracted from λ~v where λ~v becomes λIn~v. By having

the identity matrix extracted the ~v can be extracted out where the equation becomes equation2.4[13].

~0 = (A − λIn)~v (2.4)

The next step is to determine the determinants, see equation2.5where the outcome of the det() is a characteristic polynomial of degree n. This outcome is also known as the characteristic equation of A[13].

~0 = det(λIn− A) (2.5)

If A is n × n then there are n solutions of the characteristic polynomial which means that there are n eigenvalues[13].

(12)

If the eigenvalues and eigenvectors can be computed, the principal components can be computed. A principal components in the eigenface algorithm are the eigenfaces. The eigenfaces are computed by computing the eigenvalues and eigenvectors on a covariance matrix. To do that the mean of a face must be calculated first with equation2.6, where Ψ is the mean face, m is the number of images and Γ is the image[13]. Ψ = 1 m m X i=1 Γi (2.6)

This generates a mean face which represents all training images per subject. To make the covariance matrix of the eigenfaces the mean subtracted face must be computed by equation2.7, where Φ is the subtracted mean for each face.

Φ = Γi− Φ, i = 1, 2, . . . , m (2.7)

If all the mean-subtracted faces are computed per image, collection A = [Φ1, Φ2, Φ3, . . . , Φm] is made, where A is the mean-subtracted matrix.

If the mean-subtracted matrix is computed the covariance matrix, C can be computed by taking the multiplication of A and the transposed version of AT_{, see equation} _2.8_.

C = A × AT (2.8)

The covariance matrix describes all the differences in the images, by having this the eigenvalues and eigenvectors can be computed which is described in chapter2.2.2, by having the eigenvalues and eigenvectors the eigenfaces can be computed. First the vectors must be applied on the covariance matrix, C, where this matrix becomes Φm. These vectors determine the linear combinations of the training set images to get the eigenfaces, Uk with equation2.9.

Uk = m X n=1 ΦnVkn, k = 1, 2, . . . , m (2.9)

2.2.3 Algorithm

Enrollment phase

Now that the eigenvectors, eigenvalues and eigenfaces are explained, the algorithm itself can be explained with these formulas.

Firstly a mean face is calculated by taking all the training images with equation2.6. An example of this process is shown in Figure2.4[13].

Figure 2.4: Mean Face Calculation Example[13]

If the mean face is acquired the differences are calculated from the mean with 2.7 where the differences are plotted into a covariance matrix with equation2.8. See Figure 2.5for the process[13].

(13)

Figure 2.5: Difference Calculations between Mean Face and Training Images and Covariance Matrix Calculation[13]

With the covariance matrix the eigenfaces can be computed with equation2.9. If they are computed, they get sorted by the eigenvalues. See Figure2.6for an example of the sorted eigenfaces[13].

Figure 2.6: Example of Sorted eigenfaces[13]

When the eigenfaces are sorted by the eigenvalues the algorithm is ready to predict. Prediction phase

In the prediction phase a person is identified by an input image. From the input image the features are getting extracted and then the eigenface is made. The eigenface is compared to the eigenfaces which were calculated in the enrollment phase to identify the subject.

2.3 Fisherface Algorithm

The eigenface algorithm uses principal component analysis to gather the data and create eigenfaces. The problem with the eigenfaces is that whenever 2 classes are similar the components might collide where the algorithm cannot distinguish the differences. The fisherface algorithm uses linear discrimi-nant analysis and principal components analysis to minimize this problem.

2.3.1 Linear Discriminant Analysis

Linear discriminant analysis (LDA) is a linear transformation technique, it is described as a super-vised algorithm[22] while PCA is described as unsupervised [23, 20, 22]. Supervised algorithms are algorithms that keep track of class labels while unsupervised do not. Linear discriminant analysis tries to maximize the separation between multiple classes between scatter-matrices while PCA tries to maximize the variations.

(14)

2.3.2 Fisher’s Discriminant Analysis

Now the class labels are kept, a different approach can be taken to identify a person. Fisher’s linear discriminant (FLD) is a class specific method which uses LDA to shape the scatter to make it more reliable for classification[2].

To compute the projection of the classes, the ratio of the between class scatter and the within-class scatter is maximized. The between-class scatter matrix is defined as Sb in equation2.10where µi is

the mean face of Ni and T is the transformation[2].

Sb= c

X

i=1

Ni(µi− µ)(µi− µ)T (2.10)

The within-class scatter matrix is defined as Swin equation2.11where Xkis the sample image and

µi are the mean faces from Xi.

Sw= c X i=1 X xk∈Xi (xk− µi)(xk− µi)T (2.11)

If matrix SW is nonsingular, the optimal projection Wopt is chosen as a matrix with orthonormal

columns with equation2.12. This matrix maximizes the ratio of the determinant of the between-class scatter matrix of the projected samples to the determinant of the within-class scatter matrix of the projected samples.

Wopt= arg maxw

WT_S bW WT_S wW (2.12) = [w1, w2, . . . wm]

Where wi in equation 2.12is {wi|i = 1, 2, . . . , wm}, a set of generalize eigenvectors of Sb and Sw

corresponding to the m largest generalized eigenvalues {λi|i = 1, 2, . . . , m}[2], see equation 2.13.

Sbwi= λiSwWi, i = 1, 2, . . . , m (2.13)

In Figure2.7an example is given on the differences of the PCA approach and the FLD approach. The figure shows that the classes are put together in PCA when it is displayed in one dimension. When LDA is projected in one dimension, the classes are separated by choosing the matrix with the right orthonormal column. This makes it easier to distinguish and do classification because the data is not scatted within or in between.

(15)

Figure 2.7: PCA and LDA Classification Differences[2]

One problem which challenges face recognition algorithms is that the within-class scatter matrix is always singular. This means that it is possible to make a W matrix where the within-class scatter matrix of the projected samples is zero. In order to solve the singular within-class scatter matrix problem an alternative method is proposed called fisherface. This method avoids the problem by projecting the image set to a lower dimensional space so that the resulting within-class scatter matrix is not singular[2].

The fisherface method uses PCA to reduce the dimension of the feature space and then applies the standard FLD to reduce the dimension. See equation2.14[2].

W_optT = W_{f ld}T W_pcaT (2.14) Where WT

pcaand Wf ldT is described in equation2.15and equation2.16.

Wpca= arg maxw=

WTSTW (2.15) Wf ld= arg maxw= WT_WT pcaSBWpcaW WT_WT pcaSwWpcaW (2.16)

2.3.3 Algorithm

The only difference between the eigenface and fisherface algorithms in the enrollment phase is that the fisherface algorithm uses LDA in addition to PCA to do the feature extraction. This means that the results are also different if LDA is applied on PCA. After getting the eigenfaces by executing equation 2.9 the scatter-between and scatter-within matrices are computed with equation 2.11and

2.10. If these are calculated the optical projection matrix is calculated with equation2.12and saved as fisherfaces to do identification[2]. In Figure 2.8 several examples of fisherfaces are shown from OpenCV.

(16)

Figure 2.8: Example of the Fisherfaces from OpenCV1

2.4 Match rates

To measure how biometrics performs in general, various parameters are used to measure it. Accuracy is the most important result of a biometric systems. The accuracy of any system is divided into 3 parts, the False Reject Rate (FRR), the False Acceptance Rate (FAR) and the Total Success Rate (TSR) [18]. The FRR is based on how many miss matches were measured. The formula for the FRR is shown in equation (2.17).

F RR = (R

N) × 100 (2.17)

Where R is the number of false rejections, N is the number of verification and FRR is the false rejection rate in percentage.

The FAR is based on how many matches are matched with the wrong subject. The formula of the FAR is shown in equation (2.18).

F AR = (A

N) × 100 (2.18)

Where A is the number of false acceptance, N is the number of verification and FAR is the false acceptance rate in percentage.

The TSR is the rate which describes the success rate of the prediction. The formula for the TSR is shown in (2.19).

T SR = (1 − F RR + F AR

N ) × 100 (2.19)

To get a quick comparison of two biometric system the EER is often used to determine which one performs better. The EER is the rate where both the accept and reject errors are equal. The lower the EER, the more accurate the system is considered to be[18].

(17)

Chapter 3

Experimental Setup

In this chapter the databases which are used by the experiments, the experiments itself and the extraction of the data is described. The first experiment tests the devices on how many images can be trained and what the execution time is to do that. The first experiment sets a benchmark on what can be achieved in terms of training, there will be no prediction involved in this experiment. The second experiment tests what TSR can be achieved with a certain number of training images per subject. If both experiments are run, 2 sets of results are yielded. The first set from the first experiments contains the thresholds of how many images there can be trained where the second experiment says what the TSR are for N number of subjects and N number of training images. With both combined a relation can be made which states:

Algorithm X got a TSR of N %, this took M milliseconds for T number of training images per S subjects on the Y .

This relation is set up to answer the research question where the achievability of the devices and algorithms were questioned.

3.1 Databases

All of the chosen databases have standard frontal face images. Some of the databases provide images from the subject with different lighting, shadows, angles and facial expressions. These databases are used as test databases and to run the algorithms to test the TSR in different conditions.

The following databases were used.

1. The database of Faces (from AT&T).1

The AT&T database consists of images where the face was captured with a little difference in angles and facial expressions, besides that there are no lighting differences or shadow. It has 40 subjects where each subject has 10 images.

This database was chosen because of the size of the images per subject and the preprocessing of the images, it does not contain images with difference in lighting and shadows. This database is used to test if the algorithms work on images without any variation in lighting shadows angels and facial expressions and to see how the algorithm performs compared to other databases with lighting differences. See Figure3.1for some example images of the database.

(18)

Figure 3.1: ATT Images Example

2. The Yale A Face Database 2

The Yale A database consists of images with different facial expressions, lighting and shadows. It has fifteen subjects where each subject has 11 images.

The Yale A database was chosen because of the difference in lighting, facial expression and shadows. This database is a more extensive database than the AT&T database. It has the same purpose but more conclusions can be drawn since it has lighting, facial expression and shadow differences. Examples images of the Yale A database are shown in Figure3.2.

The Yale A face database also known as the Yale A database in this thesis.

Figure 3.2: Yale A Images Example

However, this database came with images that do not only contain the face but also a part of the upper body. To extract the faces face detection has been used, after detecting the faces the rest of the image got cut off. The remaining face also got resized so they are all the same size. See Figure3.3for the original image and the image with only the face.

(19)

Figure 3.3: Detection and Cutting the Image. Left: Original Image, Right: Resulted Image

3. The Extended Yale Face Database B 3

The Yale B database consists of images with different lighting and shadows. It has 39 subjects where each subject has about 63 images[14]. See Figure3.4for a couple examples of the images of the Yale B database.

The Yale B database was chosen because of the extensive lighting and shadow images. The extended Yale face database B also known as the Yale B database in this thesis.

Figure 3.4: Yale B Images Example

3.1.1 Preprocessing

Since multiple databases are used, the images will be preprocessed to make the general results more accurate. All the images in every database have been converted to an equal image size and converted to a gray scaled image. Converting the images to an equal image size is done by writing a small script in Java and converting it to gray scale image was done by writing a small Bash script with ImageMagick4. The images are also preprocessed with the histogram equalization method and the discrete cosine transformation method. Histogram equalization is used to get an enhanced image contrast and discrete cosine transformation has been used to decorrelate the image data which yields better results for the images with the difference in lighting[1,9,7].

3.2 Experiment One

The first experiment is set up answer the first sub question which was: How many images could be enrolled on mobile devices?. This experiment is not meant to measure any TSR of the algorithms, it only tries to runs the enrollment phase with a number of images which grows after each run. This is a purely computational experiment for the devices.

3_{http://vision.ucsd.edu/content/extended-yale-face-database-b-b}_{last visited: 9 June 2015} 4_{http://www.imagemagick.org/script/index.php}_{last visited: 29 June 2015}

(20)

3.2.1 Database

To observe how many images the mobile devices can train, the Yale B database is used. This database is chosen because it has a larger number of images per subject. By having a larger number, this experiment can have more variety in the tests. Other databases have more subjects but less training images where the number of training images is mandatory because the limits of the devices are tested. The number of subjects may increase but the limits cannot be tested if there are not enough images per subject.

3.2.2 Devices

The devices were chosen by filtering out the kind of processor, the processor speed, release date and which are available to the author. This research will exclude tablets due to the scope of this thesis. However, the inclusion of tablets as an extension of this research can be interesting for future research.

The following devices are chosen for iOS: • iPhone 6

• iPhone 5S • iPhone 5C

The iPhone 6 is the newest iOS device of Apple which contains an Apple A8 chip, a 64-bit processor which runs at 1.4Ghz. The iPhone 5S was released in September 2014 and contains an Apple A7 chip, this iPhone also has a 64-bit processor that runs at 1.3Ghz. The iPhone 5C, released in September 2013 has an Apple A6 chip, it also runs at 1.3Ghz but instead of having a 64-bit processor it has a 32-bit processor. All the iOS devices contain ARM-based dual-core processors.

By choosing these devices the differences can be observed on the different kind of processors of the iPhone 5S and iPhone 5C and the difference in processor speed on the iPhone 6 and iPhone 5S.

The following devices are chosen for Android: • Samsung Galaxy S6

• Motorola Nexus 6 • LG Nexus 5

The Samsung Galaxy S6, released in 2015, is a device with 2 quad-core processors where the first processor runs at 1.5Ghz and the second 2.1Ghz on a 64-bit processor. The Motorola Nexus 6, released a year earlier than the Galaxy S6, has a 64-bit quad-core processor that runs at 2.7Ghz. The last device that will be used is the LG Nexus 5, a device which runs at 2.3Ghz on a 32-bit quad-core processor which was released in 2013.

3.2.3 Setup

The first experiment tests how many images can be trained on a device. To do this extensively, variations of the number of subjects and the number of images are made. The total number of subjects is 39, the variations defined for this experiment are 5, 10, 20, 30 and 39. The total number of images per subject is 60 where the variations are 5, 10, 20,. . . 60. For each combination of the number of subject and number of images per subjects the experiment is run.

During the experiment the amount of memory, in megabytes, and the time taken, in milliseconds, will be measured. The time measurement will be done for the following processes: the preprocessing of the image and feature extraction of the image. The memory measurement will be done for all the processes together. In Table3.1an example is given for an experiment with 5 subjects.

This experiment is run with all the combinations on each device and algorithm. By taking this approach other variables could be monitored which might have an impact on the results like the number of subjects.

(21)

Images Preprocessing Features Memory Run 1 5 Run 2 10 Run 3 20 Run 4 30 Run 5 40 Run 6 50 Run 7 60

Table 3.1: Example Table of a Set of Runs for N Number of Images for 5 Subjects.

3.2.4 Extraction

To measure the data in iOS, Instruments5 will be used. Instruments is a performance-analysis and testing tool for dynamically tracing and profiling OS X and iOS code. Instruments contains a tool to measure time and memory which is called Time Profiler and Allocations. By choosing both tools and run Instruments, Instruments will record the actions taken with the tools on a device. The experiments will run on the devices while recording, all the data will be recorded to multiple trace files and extracted. The trace files look like Figure3.5.

Figure 3.5: Example Instruments

Analyzing memory will be done by selecting a period in Instruments where the begin of the period was the lowest memory usage of the run and the end the highest memory usage of the run. The call tree will show the memory usage per method of the selected period. To measure the time the same thing was done but instead of selecting the lowest memory usage and highest memory usage the whole run will be selected.

Impact

Instruments is designed to minimize its impact on data collection6_. _{To see what the impact of}

instruments is a small experiment has been conducted where a run of the first experiment is done

5_{https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/}

InstrumentsUserGuide/Introduction/Introduction.htmllast visited: 30 June 2015

6_{https://developer.apple.com/library/prerelease/watchos/documentation/DeveloperTools/Conceptual/}

(22)

without turning on instruments and with turning on instruments. Measuring time was normally done with Instruments but since it is turned off for a part of this experiment, the NSTimer7 _{API was used}

to measure the time for both cases. The results of the small experiment showed that the impact of instruments on the execution time is almost nothing. It varies from 11 to 40 milliseconds depending on how much images there will be trained. This impact is not taken into account in the results since it is insignificant. The impact of the amount of memory taken from Instruments could not be measured without a tool. In this thesis the assumption is made that instrument does not take memory to measure the algorithms.

Dalvik Debug Monitor Server

The Dalvik Debug Monitor Server8_{(DDMS) will be used to measure the data in Android. To measure}

the execution time the tool: traceview9 _{will be used. By placing 2 methods call in between a piece}

of code, the DDMS will measure the execution time of that piece of code and save it to a trace files. These files shows how long it took to execute that piece of code in a graph and call tree table, see Figure 3.6 for an example of a trace file. The execution time will be extracted from the table and plotted in a graph later.

Figure 3.6: Example Android Traceview

Measuring the memory will also be done through the DDMS, the heap tool is used in this case where a heap dump file is created on a chosen moment which means that 2 heap dump files must be created on the moment of lowest memory usage and highest memory usage.

The heap dump files can be viewed in the Memory Allocation Tool10 _{(MAT) from Eclipse where}

the data could be extracted. Impact

The impact on Android devices while using DDMS is also measured. The same method as in chapter

3.2.4is used to do that but instead of using NSTimer System.currentmillis(); is used. The results of this experiment show that the impact on Android devices is also small. The execution time of the tracing method varies from 8 to 238 milliseconds. Since the difference is also very small, the execution time is not taken into account while analyzing the results. The amount of memory taken from the trace method could not be measured because the DDMS is needed to do that. In this thesis the assumption is made where the memory cost of this method takes no impact on the memory used.

3.3 Experiment Two

The second experiment is set up to answer the second sub question which is: How much training is required per algorithm to have a total match rate of 90% for a given training set? This experiment

7_{https://developer.apple.com/library/mac/documentation/Cocoa/Reference/Foundation/Classes/NSTimer_}

Class/last visited: 2 June 2015

8_{http://developer.android.com/tools/debugging/ddms.html}_{last visited: 30 June 2015}

9_{http://developer.android.com/tools/debugging/debugging-tracing.html}_{last visited: 30 June 2015} 10_{https://eclipse.org/mat/}_{last visited: 30 June 2015}

(23)

also verifies the implementation and shows the performance of the algorithms.

3.3.1 Database

All of the chosen databases are used for this experiment. The AT&T database is taken to test if the recognition works on basic images without difference in lighting and shadow. The Yale A database is taken to test whether the recognition rate is high for images with difference in lighting and shadow. The Yale B database is taken to test whether the algorithm works in extreme different lighting and shadow conditions for a subject. Subsets were made from the databases in this experiments to acquire different perspectives in the number of subjects. The AT&T database has 40 subjects and Yale B database has 39 subjects. Two subsets were created for both databases where the first subset has 20 subjects and the second subset 10 subjects.

3.3.2 Context

This experiment is done on a computer which has a 2.6 Ghz Intel i5 Core with 8GB ram. This experiment is only to see what the recognition rates are which can be done on the computer instead of a mobile device since it is the same algorithm and the same abstract implementation.

3.3.3 Setup

This experiment has two tests. The first test is to examine the TSR of the eigenfaces algorithm. The second test does the same but for the fisherface algorithm. To get these all of the images per subject are split up into two subsets. The first set is the training set and the second set is the test set. The test set is the set that contains the leftover images which are not in the training set. E.g. the whole set got 10 images, 2 images are used for training which means that there are 8 images left for testing. It is required for this experiment to have at least five 1 image per subject to verify the TSR for each subject.

Each test contains four runs, one run for each database. In every run the algorithm gets trained by N where N ranges from [1...M − 1], where M is the number of images per subject. If the algorithm is trained in, e.g. the first run and the number of images per subject is 10, one image per all subject would be trained and nine would be predicted.

The results of this experiment will be, for each algorithm and database, a list of total success rate for [1...M − 1] images per subject. In Table3.2an example is shown of how the results will be saved. The number of total images trained for these total success rates are examined in combination with the first experiment where the processing time and the memory is measured. By having these two variables an examination has been done on how these perform against each other.

Training images per subject Test images per subject TSR Run 1 1 19 Run 2 2 18 Run 3 3 17 Run 4 4 16 Run 5 5 15 Run .. .. .. Run 19 19 1

Table 3.2: Example of a Results Table where the Total Images per Subject is 20.

3.3.4 Expectations

Whenever an face recognition algorithm enrolls more images per subject, the assumption is made that the algorithm can predict with a higher TSR rate. For this assumption the following expectation is

(24)

made: whenever the number of training images increases the TSR will increase. This will show a linear relation over every increased image.

3.3.5 Extraction

This experiment will run on the computer. After each run in the second experiment the match rate will be saved to a text file. The match rates were later extracted and plotted into a graph.

(25)

Chapter 4

Prototype

In this chapter the requirements and core implementation decisions of the prototypes are described. The requirements of the prototypes are described first. Secondly the main library, OpenCV, is de-scribed. Then the core structure of all prototypes is described and at last all the prototype specific decisions are described.

Three prototypes are made to run the experiments. The first prototype is written in Java for the Android devices, the second prototype is written in Objective-C for the iOS devices and the last prototype is also written in Java for the computer.

4.1 Requirements

To execute the experiments the desired prototype must be as flexible as possible. Therefor a list of requirements is made.

1. Automated experiments

The prototype must be able to execute the experiments automatically. This requirement is set to save time on experimenting.

2. Ease of adding databases

Adding a database should not cost more work than adding a class representing the database or a reference (e.g. an URI) to point to the location of the database. By having classes or references that represents the databases, the modification of the images or the maximum number of subjects and number of images per subject can be chosen and modified before loading into the prototype. 3. Dynamic adjustments

The prototype must be able to set the number of training images, test images and subjects on the database after loading into the prototype. This requirement ensures that a database can be modified on the number of training images, test images subjects whenever it is desired during the automated experiment.

4. Different platforms

There must be an abstract prototype that can be implemented on different platforms. Since the second experiment is run on a laptop and the first are ran on a device it must be able to implement it on different platforms.

These requirements lead to several implementation decisions which are described next.

4.2 OpenCV

The library OpenCV is used for this research. OpenCV stands for Open Computer Vision which is a computer vision and machine learning library. OpenCV is the facto standard for computing computer

(26)

vision and machine learning algorithms. The eigenfaces and fisherface algorithm is also included in this library. These algorithms were not created and contributed by the author itself. They were just used to help to build the prototype.x

This library has been chosen because of three reasons. The first reason is because of [18] and [5], in both papers the prototypes are also implemented with OpenCV, besides that, methods for preprocessing is also included. By using the same library the comparison of the results of this research and their research is more accurate because the same library is used. The second reason why this library is chosen is because it supports both Objective-C and Java which makes it feasible for this research. The last reason is because of the core of the library. The core of the library is written in C++ so the assumption is made that the core of the algorithms behaves the same.

4.3 Face Recognizer

The most important module is the face recognizer of OpenCV. The interface of the FaceRecognizer is shown in Listing 4.1. To get an instance of the FaceRecognizer for a specific algorithm a method must be called to return an implementation of the FaceRecognizer.

This interface contains a function to train(. . . ); the chosen algorithm. This method creates the classifier for all faces. After that the predict(. . . ); can be called which returns the predicted subject based on the input image.

This interface also contains functions to save(. . . ); the classifier so that the classifier could be loaded next time by calling load(. . . );. If the classifier is saved and loaded next time the algorithm does not have to train anymore. It uses the loaded classifier.

This interface can also set a label with setLabelsInfo(. . . ); and get a label with getLabelInfo(. . . );. A label is an integer representation of an image which belongs to a subject. When the algorithm classifies to determine which image is the closest to the one that must be predicted a label is returned.

1 class FaceRecognizer : public Algorithm 2 {

3 public:

4 virtual FaceRecognizer() {} 5

6 virtual void train(InputArray src, InputArray labels); 7

8 virtual int predict(InputArray src); 9

10 virtual void save(const string& filename); 11

12 virtual void load(const string& filename); 13

14 void setLabelsInfo(const std::map<int, string> &labelsInfo); 15

16 string getLabelInfo(const int &label); 17

18 vector<int> getLabelsByString(const string &str); 19 };

Listing 4.1: OpenCV FaceRecognizer Interface

4.4 Core structure

The core structure of the prototypes is described in this section to help the FaceRecognizer work. The structure of the prototypes contain 6 main interfaces: the algorithm interface, the subject label manager, the subject database model visitor, the subject database model factory, the subject database

(27)

model and the arguments object: database. See Figure4.1for the abstract architecture diagram. See FigureA.1in AppendixAfor the complete architecture.

Figure 4.1: Abstract Core Architecture

4.4.1 Algorithm Interface

An algorithm interface is made in both programming languages where the basic functions train(. . . ) and predict(. . . ) of a face recognition algorithm is defined. The abstract class algorithm is made which implements the interface. This abstract class is used as a wrapper to hide the implementation details of the FaceRecognizer of OpenCV. For each prototype a concrete class is made which has the implementation of the eigenfaces and fisherface algorithm. The abstract algorithm class acts like the facade design pattern which collects all the implementation of the classes needed and uses them together. See Figure4.2 for the class diagram of the algorithm.

The classifier of the algorithm is also created by the FaceRecognizer. It contains the information for each label coupled to each name of the subject in Java but not for Objective-C, a function to save this in the classifier was not found during the implementation. To make this work a manager is made which saves the labels externally with the subject label manager.

Figure 4.2: Algorithm Implementation

4.4.2 Subject Label Manager

The subject label manager is made to provide a way to save the labels externally for each prototype which means that every prototype has its own implementation. A class diagram is shown in Figure

4.3.

4.4.3 Interface Subject Database Model Factory

To make the input source of the databases independent, an interface is made to load database where each prototype implements its own way to load the images.

(28)

Figure 4.3: Subject Label Manager Implementation

4.4.4 Subject Database Model

The subject database model is made to change the number of test images, training images and the number of subjects to the desired number dynamically as desired in requirement 3. The algorithm can thereafter train the list of training subjects and predict with the list of test subjects.

4.4.5 Arguments Object: Database

Since the subject database model is created from an external database that contains images, the training images per subject and the number of subjects can be set by an abstract Arguments Object [16]. Each database has its own implementation. By passing an arguments object to the subject database model the model directly knows how many subjects and images per subject must be set.

This model also serves for the creation of a classification model of the algorithms. By doing this the next time whenever the algorithm is created with the arguments object the algorithm searches for the classifier with the generated name based on the arguments object so the classifier model does not need to be created again.

4.4.6 Visitor and Factory Pattern

In the early implementation of the prototype the arguments object was a plain object without any abstraction which takes the same arguments as the current arguments object and besides that an enum which defines the database. To choose which database should be loaded a static method with a switch statement was introduced to solve the problem but this forces other pieces of code to give that method an enum in its parameters. This seemed to be a bad idea, it broke the abstraction of the whole prototype because everything was statically defined.

To solve this problem a visitor pattern has been implemented since the old plain object was re-placed by an abstract arguments object. Any subclass of the arguments object could visit the visitor to execute its own specific implementation for that specific subclass. This combination solved the problem of choosing, that is done by the visitor, which database should be loaded and how, that is done by the factory, it should be loaded or created.

In Figure 4.4 an class diagram is shown of the subject database model factory, subject database model visitor, subject database model and the arguments object working together.

(29)

Figure 4.4: Subject Database Model Factory, Subject Database Model and Arguments Object Working Together

4.5 Prototypes

4.5.1 Java common project

Since there is an Android prototype and computer prototype a common project has been made which defines all the interfaces such as: IDatabase and the needed subclasses, ISubjectDatabaseModelFactory for the subject database model factory, ISubjectLabelManager for the subject label manager and the IDatabaseVisitor and the needed implementations. Each prototype implements their specific implementation for each interface if it is needed to make it platform independent.

This common project also defines the abstract implementation of the algorithm of OpenCV but the provided Java libraries by OpenCV do not contain the eigenfaces and fisherface algorithm. A FaceRecognizer instance could not be made with these libraries, to access these algorithms JavaCV1 has to be used. JavaCV is an extended library of OpenCV which contains extra functions such as: hardware accelerated full-screen image display, object detection, blob analysis and more algorithms. Maven is used as a dependency manager tool to get these libraries.

4.5.2 Android Prototype

The Android prototype is made with the Android Software Development Kit which allows developers to write an Android application in Java but since the core of OpenCV is C++ the Android Native Development Kit must be used. The Android Native Development Kit is a kit that allows developers to compile C and C++ code and use it in the Android SDK.

The databases were saved on the memory storage of each Android device manually, the Subject-DatabaseModelFactory implementation is implemented in a way where the URLs from the images are returned. The usual option to do this is to save the databases in the application but, since OpenCV is written in C++ an absolute path must be used to load the images which were not available when the images are already compiled in the application package.

The implementation of the ISubjectLabelManager is done by using SharedPreferences, an easy way to save and load persistent key-value pairs of primitive data-types to the memory of the application.

Maven is also used as a dependency manager to get OpenCV and JavaCV.

4.5.3 iOS Prototype

Since the default compiler of Xcode, Apple LLVM 6.1, supports C++ compilation OpenCV can be directly accessed by writing C++ code through Objective-C code. To make the iOS application

(30)

Cocoa Touch is used, the default iOS framework. CocoaPods2 _{is used as a dependency manager to}

get OpenCV library.

The databases are saved in the iOS application as a bundle file. A bundle file is uploaded in the application package in iOS with a direct path and without being modified. The database does not have to be saved externally on the memory. The SubjectDatabaseModel can be propagated easily in this way because there is a direct URL.

The iOS application does not have an implementation of the ISubjectLabelManager. it saves the data on the applications memory as an archived NSDictionary which is also known as a Hashmap in Java.

4.5.4 Computer Prototype

The computer prototype is written in plain Java with JavaCV. The Java prototype was used to test if the algorithm works on the machine before it was implemented on the Android devices and iPhone and to execute the second experiment.

This prototype uses MySQL as the data source for saving and loading labels. To use the database with more ease Hibernate3is used which is also implementation in the implementation of ISubjectLa-belManager. Hibernate is an Object-Relational Mapping library which eases the use of databases.

4.6 Testing

To test if the implementation works several test cases were written for the core interfaces. To test against an interface the behavior of the implementation can be monitored and see if the implemen-tation applies to the test conditions for each platform. All test cases covered the functionality of the interfaces and the use cases which are applicable for the experiments to make sure it works.

2_{https://cocoapods.org/}_{last visited: 12 June 2015} 3_{http://hibernate.org/orm/}_{last visited: 14 July 2015}

(31)

Chapter 5

Results and Evaluation

In this chapter the results of the experiments are presented.

5.1 Experiment One

The goal of the first experiment was to try to observe what the limits are of these devices while trying to answer the question: How many images could be enrolled on mobile devices?. The results are evaluated and discussed in the sections below.

5.1.1 Eigenface algorithm

The first thing noticed in the result of the first experiment with the eigenface algorithm is that the algorithm behaves exponential. Further investigation showed that the PCA is causing this behavior. Figure5.1shows a graph with the execution time which shows this behavior clearly.

Figure 5.1: The execution time of PCA on the iPhone 6 for the eigenface algorithm with training 10 subjects

The experiment was run according to the experimental setup but during the experiment all iOS devices stopped whenever the algorithm uses more than 600 megabytes, with 600 megabytes 600 images could be trained. The average time to train 600 images for 10, 20 and 30 subjects with the eigenface algorithm is shown in Figure5.2. An investigation has been done to determine if the number of subjects affects the execution time where the results showed that it affects too little to take it into account (lower is better).

(32)

Figure 5.2: Average execution time in seconds for 600 images for 10, 20 and 30 subjects on iOS and Android devices for the eigenfaces algorithm (lower is better)

In the figure above this difference in performance between the iPhone 6, iPhone 5S and iPhone 5C is clearly visible. This difference is probably caused by the difference in the bit architectures. The results of the iPhone 6 and 5S clearly show small differences. They both have a 64-bit processor while the iPhone 5C, that has a 32-bit processor, shows a huge difference compared to the other Apple devices.

The experiments on the Android devices yielded insufficient results because of the lack of informa-tion on the trace files generated by the traceview. The traceview gave a line of informainforma-tion which shows the execution time of train(InputArray src, InputArray labels) method, expanding the call tree did not give any further information which means that the feature extraction and preprocessing could not be distinguished. The traceview would normally give the call stack but since the algorithm is compiled in C++ with the NDK the data was not available to extract.

It was also not possible to measure the memory precisely on Android. Measuring the memory usage was also done using the DDMS and MAT but, the heap dump files created by the DMMS were either corrupted or imprecise. Creating a heap dump file can be done by using the graphical user interface of the DDMS but, creating the dump file was hardly possible because the tests are automated, a guess had to be made on whenever a test starts and a test finishes to capture data.

Even with less information some observations can still be made. The first observation is that the total execution time on training the algorithm is exponential which probably is caused by the PCA. The second observation is that Android devices can train more than 600 images but, the execution time of the training increases drastically. The Samsung Galaxy S6 and the Nexus 5 managed to train up to 1200 images which took over 20 minutes. The time taken to train 600 images on the Galaxy S6, Nexus 6 and Nexus 5 is shown in Figure5.2.

The reason why it grows is because PCA behaves exponential but also because of the garbage collector. During the experiment the console of the devices showed a lot of messages which said GC FOR ALLOC which means that the garbage collector is allocating memory to let the application reallocate memory. The execution time of the garbage collector and the feature extraction cannot be determined separately but it is important to keep in mind that the garbage collector has impact on the execution time.

These results lead to the following conclusion for the first experiment. The eigenfaces algorithm can be trained on several iOS and Android devices with 600 images, if the number of images exceeds 600 the iOS prototypes will close automatically because it runs out of memory but the Android prototypes on the other hand keeps on training them while the execution time increases heavily. The number of subjects does not affect the execution time for all devices and it does neither affect the memory usage for iOS devices. The execution time for iOS varies from 55 seconds to 2 minutes and 50 seconds depending on the device. The precise execution time could not be measured for the Android devices

(33)

but with the knowledge gained from the iOS results the conclusion can be made that PCA behaves the same. The Android devices trained more images but the execution time has grown rapidly because of the PCA and the garbage collector. The execution time varies from 1 minute and 45 seconds to 3 minutes and 20 seconds.

5.1.2 Fisherface algorithm

The results from the fisherface algorithm show that linear discriminant analysis also behaves expo-nential but the expoexpo-nential growth is slower than the PCA. See Figure5.3 for the logartihmic scale of LDA and PCA. The execution time of the fisherface algorithm was slightly higher because of the linear discriminant analysis, besides that the execution time of the PCA is also slightly higher. In-vestigation shows that the train method executes an extra method, called PCA::project(InputArray const) on the iOS devices. This method projects the vector(s) to the principal component subspace and is used to prepare the linear discriminant analysis. Again, this cannot be observed in the Android but the assumption is made that this method is also executed during feature extraction.

Figure 5.3: Example of the execution time of LDA and PCA on the iPhone 6 for the fisherface algorithm with training 10 subjects

The memory usage for the iOS devices is about the same as shown in Figure 5.4for the eigenface and fisherface algorithm which means that the application also stops whenever the next run exceeds the usage of 600 megabytes. This means that the differences in execution time for both the eigenface and fisherface algorithm are caused by the extra project(..) method and the LDA. All the iOS devices were able to train up to 600 images except for the iPhone 6. The iPhone 6 only finished the run for 10 subjects where each subject has 60 images and then crashes. The reason why it crashes could not be found, the device stopped while the iPhone 5S and iPhone 5C did continue to train.

The average training time was higher than the eigenface algorithm because of the extra method call in PCA and the LDA itself. The results are shown in Figure5.5.

The Android devices showed the same behavior as the iOS devices. The execution time took longer and the garbage collector was still allocating data for the algorithm while sending the message GC FOR ALLOC. The execution time of the fisherface algorithm on Android devices is shown in Figure5.5.

With these results the following conclusion can be drawn for the fisherface algorithm. The fisherface algorithm uses PCA and LDA to do feature extraction. Even with this algorithm the execution of the training of the fisherface algorithm can still be done on iOS and Android devices.

(34)

Figure 5.4: Average memory usage of 600 images in megabytes for 10, 20 and 30 subjects with the eigenface and fisherface algorithm on iOS devices

Figure 5.5: Average execution time in seconds for 600 images for 10, 20 and 30 subjects on iOS and Android devices for the fisherface algorithm (lower is better)

5.2 Experiment Two

In the second experiment an attempt has been made to measure the TSR of the algorithms. The question asked was How much training is required per algorithm to have a total match rate of 90% for a given training set?. The second experiment has been run according to the experimental setup. The results are shown below.

5.2.1 Eigenface algorithm

The experiments yielded data for the eigenface algorithm which is divided into two graphs. The first graph, shown in Figure5.6, shows the TSR of the ATT database, its subsets and the Yale A database for X training images. On the X-axis of the graph shown in Figure5.6the number of training images are shown. These training images gets trained where the left over images of the whole set of the subject gets predicted. The results of the prediction is shown in percentage on the Y-axis and plotted into the graph.

(35)

Figure 5.6: Total success rates of the ATT database, its subsets and the Yale A database for X number of training images (higher is better)

of 90% or more if a certain number of subjects are trained. For the AT&T database at least 40 subjects, 7 images per subject, must be trained where the total number of images is 280. Reducing the number of subjects to 20 did not impact the TSR a lot but, reducing the number of subjects to 10 yielded better results. The TSR of 90% can already be achieved while training 50 images. The Yale A database also shows a TSR of 90% or can be achieved while training 15 subjects, where each subject has 5 images.

The second graph shown in Figure5.7shows the TSR of the Yale B database and its subsets.

Figure 5.7: Total success rates of the Yale B database and its subsets for X number of training images (higher is better)

Figure5.7shows a different perspective from the eigenface algorithm. Figure5.6shows the results of the Yale A and ATT database which has standard frontal images with a little bit difference in facial expressions and a little bit of lighting difference while the second figure contains a graph with the Yale B database which contains images with different lighting. In this result the impact of lighting and reducing subjects is clearly visible. The graph show that a 90% TSR or higher can be achieved

Performance of Face Recognition Algorithms on Mobile Devices