Generalised density function estimation using moments and the characteristic function

(1)

Generalised density function

estimation using moments

and the characteristic function

Gerhard Esterhuizen

Thesis presented in partial fulfilment

of the requirements for the degree of

Master of Science in Electronic Engineering

at the

University of Stellenbosch

Supervisor:

Prof. J .A. du Preez

(2)

Declaration

I, the undersigned, hereby declare that the work contained in this thesis is my own original work and that I have not previously in its entirety or in part submitted it at any university for a degree.

Signature:

March 2003

(3)

Abstract

Probability density functions (PDFs) and cumulative distribution functions (CDFs) playa central role in statistical pattern recognition and verification systems. They allow observations that do not occur according to deterministic rules to be quantified and mod-elled. An example of such observations would be the voice patterns of a person that is used as input to a biometric security device.

In order to model such non-deterministic observations, a density function estimator is employed to estimate a PDF or CDF from sample data. Although numerous density function estimation techniques exist, all the techniques can be classified into one of two groups, parametric and non-parametric, each with its own characteristic advantages and disadvantages.

In this research, we introduce a novel approach to density function estimation that attempts to combine some of the advantages of both the parametric and non-parametric estimators. This is done by considering density estimation using an abstract approach in which the density function is modelled entirely in terms of its moments or characteristic function. New density function estimation techniques are first developed in theory, after which a number of practical density function estimators are presented.

Experiments are performed in which the performance of the new estimators are com-pared to two established estimators, namely the Parzen estimator and the Gaussian mixture model (GMM). The comparison is performed in terms of the accuracy, computational re-quirements and ease of use of the estimators and it is found that the new estimators does combine some of the advantages of the established estimators without the corresponding disadvantages.

(4)

Opsomming

Waarskynlikheids digtheidsfunksies (WDFs) en Kumulatiewe distribusiefunksies (KDFs) speel 'n sentrale rol in statistiese patroonherkenning en verifikasie stelsels. Hulle maak dit moontlik om nie-deterministiese observasies te kwantifiseer en te modelleer. Die stempa-trone van 'n spreker wat as intree tot 'n biometriese sekuriteits stelsel gegee word, is 'n voorbeeld van so 'n observasie.

Ten einde sulke observasies te modelleer, word 'n digtheidsfunksie afskatter gebruik om die WDF of KDF vanaf data monsters af te skat. Alhoewel daar talryke digtheidsfunksie afskatters bestaan, kan almal in een van twee katagoriee geplaas word, parametries en nie-parametries, elk met hul eie kenmerkende voordele en nadele.

Hierdie werk Ie 'n nuwe benadering tot digtheidsfunksie afskatting voor wat die voordele van beide die parametriese sowel as die nie-parametriese tegnieke probeer kombineer. Dit word gedoen deur digtheidsfunksie afskatting vanuit 'n abstrakte oogpunt te benader waar die digtheidsfunksie uitsluitlik in terme van sy momente en karakteristieke funksie gemo-delleer word. Nuwe metodes word eers in teorie ondersoek en ontwikkel waarna praktiese tegnieke voorgele word. Hierdie afskatters het die vermoe om 'n wye verskeidenheid digt-heidsfunksies af te skat en is nie net ontwerp om slegs sekere families van digtdigt-heidsfunksies optimaal voor te stel nie.

Eksperimente is uitgevoer wat die werkverrigting van die nuwe tegnieke met twee geves-tigde tegnieke, naamlik die Parzen afskatter en die Gaussiese mengsel model (GMM), te vergelyk. Die werkverrigting word gemeet in terme van akkuraatheid, vereiste numeriese verwerkingsvermoe en die gemak van gebruik. Daar word bevind dat die nuwe afskatters weI voordele van die gevestigde afskatters kombineer sonder die gepaardgaande nadele.

(5)

Acknowledgements

I would like to thank the following people:

• Prof. J.A. Du Preez, my supervisor, for his patient guidance.

• Zelda Weitz for her unfailing love and encouragement.

• My parents and family for their education and support.

• My grandmother and Mr and Mrs Weitz for providing a home away from home.

• Dr Dave Weber and the DSP Lab for providing a creative learning environment.

• Chari Botha for his technical advice.

• Pieter Nel and Koos Hugo for False Bay sailing.

(6)

2.4.2 A PDF in terms of moments 29 2.4.3 The anti-derivative . . . 37 2.4.4 Numerical integration techniques 38 2.4.5 Fourier series approximation . . . 41 2.4.6 Estimating moments from sample data 44 2.5 Estimators based on the characteristic function 46 2.5.1 Motivation... 46 2.5.2 A PDF in terms of a characteristic function 47 2.5.3 Fourier series . . . 51 2.5.4 Limiting case where N_x -+ 00 52

2.6 Conclusions . . . 57 2.6.1 Comparison between characteristic function and moments techniques 58 2.6.2 Comparison with the Parzen estimator and Gaussian mixture model

(GMM) 59

3 Novel cumulative distribution function estimators

3.1 Introduction 3.2 Motivation.

3.3 Definitions and background 3.4 Estimators based on moments

3.4.1 A CDF in terms of moments. 3.4.2 Numerical integration techniques 3.4.3 Fourier series approximation . . . 3.5 Estimators based on the characteristic function

3.5.1 A CDF in terms of a characteristic function 3.5.2 Fourier series 3.6 Conclusions . . . . 4 Experimental results 4.1 Introduction . . . . 4.2 Experimental setup 4.2.1 Input data.

4.2.2 Estimation error measure

4.2.3 PDF and CDF estimate using moments.

4.2.4 PDF and CDF estimate using characteristic function

VI 61 61 62 62 63 64 66 69 75 76 79 81 83 83 85 85 86 87 87

(8)

4.2.5 Parzen estimator

....

88

4.2.6 Gaussian Mixture Model 88

4.3 Mean estimation error 88

4.3.1 PDF estimators . . . 88 4.3.2 CDF estimators . . . ₉₇ 4.4 Computational requirements ₁₀₂ 4.4.1 PDF Estimators. 104 4.4.2 CDF Estimators 106 4.5 Training requirements ₁₀₇

4.6 Application to speaker verification. ₁₀₉

4.7 Conclusions .

. .

.

. . . .

₁₁₁

5 Conclusions and recommendations 114

5.1 Conclusions

....

114

5.2 Recommendations. . . 116

A A review of the Fourier transform 120

B Summary of algorithms 123

(9)

List of Tables

4.1 PDF estimators: mllllmum mean estimation errors, minimum combined mean estimation errors and corresponding combined standard deviations (in brackets) from 100 samples (lOOx Kullback-Leibler divergence is indicated). 91 4.2 PDF estimators: minimum mean estimation errors, minimum combined

mean estimation errors and corresponding combined standard deviations (in brackets) from 1000 samples (lOOx Kullback-Leibler divergence is indicated). 95 4.3 PDF estimators: the effect of selecting a sub-optimal working point. . . .. 96 4.4 CDF estimators: minimum mean estimation errors, minimum combined

mean estimation errors and corresponding combined standard deviation (in brackets) from 100 samples (lOx integral absolute difference is indicated). 101 4.5 CDF estimators: minimum mean estimation errors, minimum combined

mean estimation errors and corresponding combined standard deviation (in brackets) from 1000 samples (lOx integral absolute difference is indicated). 102 4.6 Gradient (xl00) characterising the relationship between the computation

time and the parameter value. . . .. 105 4.7 Comparison between GMM and CF technique in a speaker verification

ap-plication.. . . 111 4.8 Feature matrix for all the estimators. 113 A.l Selected Fourier transform properties. .

B.l PDF estimate from moments using Fourier series. B.2 PDF estimate from samples using Fourier series.. B.3 CDF estimate from moments using Fourier series. B.4 CDF estimate from samples using Fourier series..

Vlll 122 124 125 126 127

(10)

List of Figures

2.1 Successive Taylor series approximations to characteristic function of Gaus-sian PDF using 5, 14, and 32 terms. . . .. 31 2.2 The smoothing effect of a Hamming window on the characteristic function. 34 2.3 Leakage introduced by rectangular windowing of the characteristic function. 36 2.4 Discretisation of the characteristic function. 45 2.5 Reconstructing a PDF directly from data samples, using a triangular

win-dowing function. 50

3.1 Relationship between f(x), f'(x)and f"(x) 71

4.1 Mean estimation error: PDF estimate using GMM from 100 samples. 89 4.2 Mean estimation error: PDF estimate using Parzen estimator from 100

sam-ples. . . .. 90 4.3 Mean estimation error: PDF estimate using characteristic function from 100

samples. . . .. 90 4.4 Mean estimation error: PDF estimate using moments from 100 samples. . 91 4.5 Typical PDF estimates.. . . 92 4.6 Examples of over-fitted PDF estimates (obtained from 100 samples). . 94 4.7 Mean estimation error: CDF estimate using GMM from 100 samples. 97 4.8 Mean estimation error: CDF estimate using Parzen estimator from 100

samples. . . .. 98 4.9 Mean estimation error: CDF estimate using characteristic function from 100

samples. . . .. 98 4.10 Mean estimation error: CDF estimate using moments from 100 samples. 99 4.11 Typical CDF estimates. 100 4.12 Comparison of normalised execution times of PDF estimators: Pentium III

700 MHz 104

(11)

(12)

(13)

(14)

(15)

(16)

(17)

(18)

(19)

(20)

(21)

(22)

(23)

(24)

(25)

(26)

(27)

(28)

(29)

(30)

(31)

(32)

(33)

(34)

(35)

(36)

(37)

(38)

(39)

(40)

(41)

(42)

(43)

(44)

(45)

(46)

(47)

(48)

(49)

(50)

(51)

(52)

(53)

(54)

(55)

(56)

(57)

(58)

(59)

(60)

(61)

(62)

(63)

(64)

(65)

(66)

(67)

(68)

(69)

(70)

(71)

(72)

(73)

(74)

(75)

(76)

(77)

(78)

(79)

(80)

(81)

(82)

(83)

(84)

(85)

(86)

(87)

(88)

(89)

(90)

(91)

(92)

(93)

(94)

(95)

(96)

(97)

(98)

(99)

(100)

(101)

(102)

(103)

(104)

(105)

(106)

(107)

(108)

(109)

(110)

(111)

(112)

(113)

(114)

(115)

(116)

(117)

(118)

(119)

(120)

(121)

(122)

(123)

(124)

(125)

(126)

(127)

(128)

(129)

(130)

(131)

(132)

(133)

(134)

(135)

(136)

(137)

(138)

(139)

Generalised density function estimation using moments and the characteristic function