It is obvious that A1= (L)(M+1)xL0M01
+ M
p=1
CpM(L 0 p)(M0p)(L 0 M 0 p)TpxL0M0p01
A
(A.4)
and
A2=
M+1 p=1
Cp01M (L 0 p + 1)(M0p+1)TpxL0M0p01
= TM+1xL +M
p=1
Cp01M (L0p+1)(M0p+1)TpxL0M0p01
A
: (A.5)
UsingA3defined in (A.4) andA4defined in (A.5), we can obtain A3+ A4=
M p=1
CpM(L0M 0p)+CMp01(L 0 p + 1)
2 (L 0 p)(M0p)TpxL0M0p01
=
M p=1
CpM+1(L0p)(M0p+1)TpxL0M0p01: (A.6)
Inserting (A.2)–(A.6) into (A.1) yields
@
@x
@M
@xMJL(x; T )
=exp(0T x01) M+1
p=0
CpM+1(L0p)(M0p+1)TpxL0M0p01 : (A.7)
Thus, (23) also holds true fork = M + 1. The proof is completed.
ACKNOWLEDGMENT
The authors would like to thank the anonymous reviewers and the Associate Editor for their valuable comments and suggestions, which have greatly improved the quality of this paper.
REFERENCES
[1] E. J. Kelly, “An adaptive detection algorithm,” IEEE Trans. Aerosp.
Electron. Syst., vol. 22, no. 1, pp. 115–127, Mar. 1986.
[2] F. C. Robey, D. R. Fuhrmann, E. J. Kelly, and R. Nitzberg, “A CFAR adaptive matched filter detector,” IEEE Trans. Aerosp. Electron. Syst., vol. 28, no. 1, pp. 208–216, Jan. 1992.
[3] E. Conte, M. Lops, and G. Ricci, “Asymptotically optimum radar de- tection in compound-Gaussian clutter,” IEEE Trans. Aerosp. Electron.
Syst., vol. 31, no. 2, pp. 617–625, Apr. 1995.
[4] N. B. Pulsone and R. S. Raghavan, “Analysis of an adaptive CFAR detector in non-Gaussian interference,” IEEE Trans. Aerosp. Electron.
Syst., vol. 35, no. 3, pp. 903–916, Jul. 1999.
[5] G. A. Fabrizio, A. Farina, and M. D. Turley, “Spatial adaptive subspace detection in OTH radar,” IEEE Trans. Aerosp. Electron. Syst., vol. 39, no. 4, pp. 1407–1428, Oct. 2003.
[6] L. L. Scharf and B. Friedlander, “Matched subspace detectors,” IEEE Trans. Signal Process., vol. 42, no. 8, pp. 2146–2157, Aug. 1994.
[7] S. Kraut, L. L. Scharf, and L. T. McWhorter, “Adaptive subspace detec- tors,” IEEE Trans. Signal Process., vol. 49, no. 1, pp. 1–16, Jan. 2001.
[8] S. Kraut, L. L. Scharf, and R. W. Butler, “The adaptive coherent estimator: A uniformly most-powerful-invariant adaptive detection statistic,” IEEE Trans. Signal Process., vol. 53, no. 2, pp. 427–438, Feb. 2005.
[9] S. Kraut and L. L. Scharf, “UMP invariance of the multi-rank adap- tive coherence estimator,” in Proc. 37th Asilomar Conf. Signals Syst.
Comput., Pacific Grove, CA, Nov. 2003, pp. 1863–1867.
[10] J. Liu, Z.-J. Zhang, Y. Yang, and H. Liu, “A CFAR adaptive subspace detector for first-order or second-order Gaussian signals based on a single observation,” IEEE Trans. Signal Process., vol. 59, no. 11, pp.
5126–5140, Nov. 2011.
[11] Y. Jin and B. Friedlander, “A CFAR adaptive subspace detector for second-order Gaussian signals,” IEEE Trans. Signal Process., vol. 53, no. 3, pp. 871–884, Mar. 2005.
[12] C. D. Richmond, “Performance of the adaptive sidelobes blanker de- tection algorithm in homogeneous environments,” IEEE Trans. Signal Process., vol. 48, no. 5, pp. 1235–1247, May 2000.
[13] C. D. Richmond, “Performance of a class of adaptive detection algorithm in nonhomogeneous environments,” IEEE Trans. Signal Process., vol. 48, no. 5, pp. 1248–1262, May 2000.
[14] I. S. Gradshteyn and I. M. Ryzhik, Table of Integrals, Series and Prod- ucts, 7th ed. New York: Academic, 2007.
[15] E. J. Kelly, “Finite-Sum Expression for Signal Detection Probabilities,”
Lincoln Lab., MIT, Tech. Rep. No. 566, 1981.
[16] H. L. Krall and O. Frink, “A new class of orthogonal polynomials:
The bessel polynomials,” Trans. Amer. Math. Soc., vol. 65, no. 1, pp.
100–115, Jan. 1949.
[17] M. M. Ali and M. Obaidullah, “Distribution of linear combination of exponential variates,” Commun. Stat. Theor. Methods, vol. 11, no. 13, pp. 1453–1463, 1982.
[18] F. D. Colavecchia, G. Gasaneo, and J. E. Miraglia, “Numerical evaluation of Appell’sF hypergeometric function,” Comput. Phys.
Commun., vol. 138, no. 1, pp. 29–43, Jul. 2001.
[19] F. D. Colavecchia and G. Gasaneo, “F1: A code to compute Appell’s F hypergeometric function,” Comput. Phys. Commun., vol. 157, no.
1, pp. 32–38, Feb. 2004.
Optimal Stochastic Parameter Design for Estimation Problems
Hamza Soganci, Sinan Gezici, and Orhan Arikan
Abstract—In this study, the aim is to perform optimal stochastic param- eter design in order to minimize the cost of a given estimator. Optimal prob- ability distributions of signals corresponding to different parameters are obtained in the presence and absence of an average power constraint. It is shown that the optimal parameter design results in either a deterministic signal or a randomization between two different signal levels. In addition, sufficient conditions are obtained to specify the cases in which improve- ments over the deterministic parameter design can or cannot be achieved via the stochastic parameter design. Numerical examples are presented in order to provide illustrations of theoretical results.
Index Terms—Bayes risk, randomization, stochastic parameter design.
I. INTRODUCTION
In parametric estimation problems, an unknown parameter is esti- mated based on observations, the probability distribution of which is known as a function of the unknown parameter [1], [2]. In the presence of prior information about the parameter, Bayesian estimators, such as Manuscript received February 17, 2012; revised May 10, 2012; accepted May 11, 2012. Date of publication May 22, 2012; date of current version August 07, 2012. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Ta-Hsin Li. This research was supported in part by the National Young Researchers Career Development Programme (project no. 110E245) of the Scientific and Technological Research Council of Turkey (TUBITAK).
The authors are with the Department of Electrical and Electronics En- gineering, Bilkent University, Bilkent, Ankara 06800, Turkey (e-mail:
hsoganci@ee.bilkent.edu.tr; gezici@ee.bilkent.edu.tr; oarikan@ee.bilkent.
edu.tr).
Color versions of one or more of the figures in this correspondence are avail- able online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TSP.2012.2200892
1053-587X/$31.00 © 2012 IEEE
Fig. 1. System model. Device A transmits a stochastic signals for each value of parameter, and Device B estimates based on the noise corrupted version ofs . One interpretation is to consider the dashed box as a measurement device, in which case n denotes the measurement noise.
the minimum mean-squared error (MMSE) estimator and the minimum mean-absolute error (MMAE) estimator, are commonly employed [1].
On the other hand, in the absence of prior information about the pa- rameter, the minimum variance unbiased estimator (MVUE), if it ex- ists, or the maximum likelihood estimator (MLE) can be used [2]. In these conventional formulations of the parameter estimation problem, the aim is to obtain an optimal estimator that minimizes a certain cost function, such as the mean-squared error. In this study, we consider a different formulation in which the aim is to minimize the cost of a given estimator by performing stochastic parameter design under cer- tain constraints. Motivations for this seemingly counterintuitive formu- lation will be provided in the next section.
Recently, various studies have employed signal randomization in order to improve the performance of detection and estimation systems (e.g., [3]–[7]). For example, an additive noise component that is randomized between two signal values can increase the detection probability of certain detectors under a false alarm constraint [3], [4].
Also, for power constrained communications systems, transmitting stochastic signals that are randomized among at most three different signal values can provide reductions in the average probability of error compared to the conventional case in which deterministic signal values are transmitted for each symbol [5]. In [6], it is shown that performance of some suboptimal estimators can be enhanced via additive “noise”
that is injected into the observations before the estimation process. It is observed that this noise component can be a constant signal or a randomization between two signal values.
Motivated by the investigation of signal randomization in recent works [3]–[7], we consider the concept of stochastic parameter design for es- timation problems in this study. Specifically, we try to answer the fol- lowing question: If a fixed estimator is used at the receiver, what should be the optimal distribution of the signal sent from the transmitter for each possible parameter value? Referring to Fig. 1, the aim is to design the optimal stochastic signalsfor each in order to minimize the cost (specifically, the Bayes risk) of a given estimator, which performs esti- mation based on the noise corrupted version ofs, that is,s+ n. Since there can exist power limits for transmitted signals in practice, this de- sign problem needs to be solved under certain constraints.
As a specific example, consider a scenario in which the receiver em- ploys the sample mean estimator to estimate a parameter based on a number of independent and identically distributed (i.i.d.) observations.
The aim is to find the optimal random variable for each parameter value at the transmitter in order to minimize the Bayes risk of the sample mean estimator at the receiver. For instance, we would like to determine if sending i.i.d. Gaussian or Laplacian random variables with mean and variance 1 results in a lower Bayes risk. Or, more generally, among all continuous and discrete random variables, we would like to determine the one that minimizes the Bayes risk of the sample mean estimator.
In this study, after providing some motivations (Section II), we formulate this optimal stochastic parameter design problem, and prove that the optimalscan be represented by either a deterministic signal value or a randomization between two different signal values (Section III). In addition, a convex relaxation of the optimal parameter design problem (resulting in linearly constrained linear programming)
is presented (Section III), and sufficient conditions under which stochastic parameter design can or cannot provide improvements over the deterministic parameter design are obtained (Section IV). Also, numerical examples are presented to investigate the theoretical results (Section V).
II. MOTIVATION
In conventional estimation problems, the aim is to design an optimal estimator for a given distribution of the observations. However, motiva- tions can also be provided for the stochastic parameter design problem investigated in this study. For example, consider the design of a generic device (Device A in Fig. 1) which needs to output a certain parameter.
This output is to be measured by a measurement device (the dashed box in Fig. 1) that employs a certain estimation algorithm for determining the parameter (e.g., averages various measurements). Then, the aim is to design a stochastic signalsfor each so that the accuracy (i.e., es- timation performance) of the given measurement device is optimized.
In other words, considering a certain type of a measurement device, the estimation performance of the overall system is to be optimized by designing stochastic signals for different parameters. Such a system model, in which estimation is performed based on measurements ob- tained by a number of measurement devices, is considered also in [8].
However, a different problem is considered in that study, and the op- timal linear estimator is obtained in the presence of cost-constrained measurements. It should also be mentioned that most measurement de- vices are designed under a certain measurement noise assumption, such as Gaussian. They are typically nonadaptive devices, hence, in the pres- ence of noise that deviates from the assumed noise distribution, their performance may degrade significantly. To improve the performance, the measurement device can be replaced with a more capable one; how- ever, such a replacement may be very costly in some cases. To avoid the replacement cost and associated complications, the proposed sto- chastic parameter design approach can be used, which designs optimal signals for each parameter so that the performance of the suboptimal measurement device can be improved.
As another motivation of the setup in Fig. 1, a wireless sensor net- work [9], in which a parameter value (such as temperature or pressure) is sent from one device to another, can be considered. When the trans- mitter (Device A) knows the probability distribution of the channel noise,n (which can be obtained via feedback), it can perform stochastic parameter design in order to optimize the performance of the estimator at the receiver (Device B). If the probability distribution ofn is un- known, then the results can be considered to provide a theoretical upper bound on the estimation performance. It is important to note that the ad- ditive noise is used to model all the operations/effects between Device A and Device B in Fig. 1. For example, signal values can be quantized, and encoded symbols can be sent via a specific digital communica- tions method in some cases. Then, the additive noise model in Fig. 1 can be considered to provide an abstraction for all the blocks between Device A and Device B, such as quantizer, encoder/decoder, modu- lator/demodulator, and additive noise channel, as discussed in [11]. It should also be noted that noisen in Fig. 1 is modeled to have generic probability distributions, not necessarily Gaussian, in the theoretical investigations in this study.
III. STOCHASTICPARAMETERDESIGN
Consider a parameter estimation scenario as in Fig. 1, where the aim is to send the information about parameter from Device A to Device B over an additive noise channel. For that purpose, Device A can transmit a (random) function of, say s, to Device B. Then, the received signal (observation) at Device B is expressed as
y = s+ n (1)
wheren denotes the channel noise, which has a probability density function (PDF) represented bypn(1). It is assumed that Device B em- ploys a fixed estimator specified by ^(y) in order to estimate . In ad- dition, the prior distribution of is denoted by w(), and the parameter space in which resides is represented by 3.
In this study, the problem is to find the optimal probability distribu- tion ofsfor each 2 3 in order to minimize the Bayes risk of a given estimator. It should be noted that, in conventional estimation problems, the aim is to design the optimal estimator for a given probability distri- bution of the observation [2]. However, we consider a different problem in which the aim is to optimize the information carrying parameters in order to optimize the performance of a given estimator. Another impor- tant point is that unlike conventional estimation problems,sin (1) is modeled as a random variable for each value of; that is, a stochastic parameter design approach is considered in this study.
A. Unconstrained Optimization
First, no constraints are considered in the selection ofs. Then, the optimal stochastic parameter design problem can be formulated as
fpopts ; 2 3g = arg min
fp ;23gr(^) (2)
wherefps ; 2 3g denotes the set of PDFs for s for all possible values of parameter, and r(^) is the Bayes risk of the estimator. In order to obtain a more explicit formulation of the problem, the Bayes risk can be expressed as
r(^) =
3
w() C ^(y); p(y)dyd (3)
where p(y) denotes the PDF of y, which is indexed by , and C[^(y); ] represents a cost function [2]. For example, C[^(y); ] = k^(y) 0 k2 corresponds to the squared-error cost function, for whichr(^) becomes the mean-squared error (MSE). In this study, a generic cost functionC[^(y); ] is considered in all the derivations.
If s were modeled as a deterministic quantity for each value of, p(y) in (3) could be expressed in terms of the PDF of n as pn(y 0 s) (see (1)). However, we consider a stochastic parameter design framework and model s as a stochastic variable for each
. Then, assuming that the noise and s are independent,p(y) is calculated as ps (x)pn(y 0 x)dx. Therefore, (3) becomes
r(^) =
3
w() ps (x) C ^(y); pn(y 0 x)dydxd: (4)
Defining an auxiliary functiong(x) as
g(x)=1 C ^(y); pn(y 0 x)dy; (5) the relation in (4) can be stated as
r(^) =
3
w()E fg(s)g d (6)
where each expectation operation is over the PDF ofsfor a given value of. From (6), it is observed that r(^) can be minimized if, for each, the PDF of sassigns all the probability to the minimizer of g.1Namely, the solution of the optimization problem in (2) can be expressed as
popts (x) = (x 0 sunc ); sunc = arg min
x g(x) (7)
1If there are multiple minimizers, any (combination) of them can be chosen for the optimal solution.
for all 2 3. Therefore, it is concluded that the optimal stochastic parameter design results in optimal PDFs that have single point masses. Hence, deterministic parameter design is optimal and no sto- chastic modeling is needed when there are no constraints in the design problem. However, in practice, the values of s cannot be chosen without any constraints (such as an average power constraint), and it will be shown in the next section that the stochastic parameter design can result in performance improvements in the presence of constraints on the moments ofs. Another important observation from (7) is that the solution does not require the knowledge of the prior distribution w(), since the optimal solution is obtained for each separately.
B. Constrained Optimization
In practical scenarios, the parameter design cannot be performed without any limitations. For example, in the absence of a power con- straint, it would be possible to reduce the Bayes risk arbitrarily by trans- mitting signals with very high powers compared to the noise power.
In this section, a common design constraint in the form of an av- erage power constraint is considered in the stochastic parameter design problem. Although a specific constraint type is used in the following, it will be discussed that other types of constraints can also be incorpo- rated into the theoretical analysis.
Consider an average power constraint in the form of
E ksk2 A (8)
for 2 3, where ksk is the Euclidean norm of vector s, andA
denotes the average power constraint for. It is noted from (8) that a generic model is considered for the constraintA, which can depend on the value of in general. For the special case in which the average power constraint is the same for all parameters,A= A for 2 3 can be employed.
From (6) and (8), the optimal stochastic parameter design problem can be stated as
fp ;23gmin
3
w()E fg(s)g d
subject to E ksk2 A; 8 2 3 (9) whereg(1) is as defined in (5). The investigation of the constrained optimization problem in (9) reveals that the problem can be solved sep- arately for each as follows:
minp E fg(s)g subject to E ksk2 A (10)
for 2 3. In other words, the optimal PDF of scan be obtained separately for each. Therefore, the result does not depend on the prior distributionw(), and the solution can be obtained in the absence of prior information.
Optimization problems in the form of (10) have been investigated in different studies in the literature [3], [5], [10]. Specifically, [3] and [10] aim to obtain the optimal additive “noise” PDF that maximizes the detection probability under a constraint on the false-alarm probability, and [5] investigates optimal signal PDFs in a power constrained binary communications systems. Based on similar arguments to those in [3], [5], [10], the following result can be obtained.
Proposition 1: Supposegis a continuous function and each com- ponent ofsresides in a finite closed interval. Then, an optimal solution to (10) can be expressed in the following form:
popts (x) = (x 0 s;1) + (1 0 )(x 0 s;2) (11) for 2 [0; 1].
Proof: Consider the set of all(g(s); ksk2) pairs and the set of all(Efg(s)g; Efksk2g) pairs, and denote them as U and W , re- spectively. Namely,U = f(u1; u2) : u1= g(s), u2 = ksk2; 8sg andW = f(w1; w2) : w1 = Efg(s)g;w2 = Efksk2g; 8ps g.
As discussed in [3] and [5], the convex hull ofU can be shown to be equal toW . Then, based on Carathéodory’s theorem [12], it is con- cluded that any point inW can be obtained as a convex combination of at most three points inU. Also, since an optimal PDF should achieve the minimum value, it must correspond to the boundary ofW , which results in a convex combination of at most two points inU. (The as- sumptions in the proposition imply thatW is a closed set; therefore, it contains its boundary [5].) Hence, an optimal solution can be expressed as in (11) [13].
Proposition 1 states that the optimal solution can be achieved by randomization between at most two different values for each. Based on this result, the optimal stochastic parameter design problem in (10) is expressed as
;smin;s g(s;1)+(10)g(s;2)
subject to ks;1k2+(10)ks;2k2A; 2[0; 1] (12) for 2 3. Compared to (10), the formulation in (12) provides a sig- nificant simplification as it requires optimization over a finite number of variables instead of over all possible PDFs. Since generic cost func- tions and noise distributions are considered in the theoretical analysis, gin (5) is quite generic and the optimization problem in (12) can be nonconvex in general. Therefore, global optimization techniques such as particle swarm optimization (PSO) and differential evolution (DE) can be employed to obtain the solution [14], [15].
Remark 1: Although the average power constraint in (8) is consid- ered in obtaining the preceding results, the other types of constraints in the form ofEfhi(s)g A;ifori = 1; . . . ; Nccan also be incor- porated. Specifically, assuming continuoushi, the form of the optimal PDF in Proposition 1 becomespopts (x) = Ni=1;i(x 0 s;i), with
;i 0 for i = 1; . . . ; Ncand Ni=1;i= 1, which can be proven by updating the definitions of setsU and W accordingly in the proof of Proposition 1.
As an alternative approach, a convex relaxation technique can be employed to obtain an approximate solution of (10) in polynomial time [5], [16]. To that aim, it is assumed thatps can be expressed as ps (x) = Nl=1 l(x 0 ~s;l), where l 0 for l = 1; . . . ; Nm,
Nl=1 l = 1, and ~s;1; . . . ; ~sN are known possible values fors. Then, by defining = [ 11 1 1 N ]T,~g= [g(~s;1) 1 1 1 g(~sN )]T andc = [k~s;1k21 1 1 k~sN k2]T, the convex version of (10) can be obtained as
min T~gggsubject to Tc A; T1 = 1; 0 (13) where1 and 0 denote the vectors of ones and zeros, respectively, and 0 means that each element of is greater than or equal to zero.
It is noted that (13) presents a linearly constrained linear optimization problem; hence, it can be solved efficiently in polynomial time [16].
In general, the solution of (13) provides an approximate solution, and the approximation accuracy can be improved by using a large value of Nm.
IV. OPTIMALITYCONDITIONS
The deterministic parameter design can be considered as a special case of the stochastic parameter design whensin (10) is modeled as a deterministic quantity for each. Namely, the deterministic parameter design problem can be formulated as
mins g(s) subject to ksk2 A (14)
for 2 3 (cf. (10)). Let sopt denote the minimizer of the op- timization problem in (14). Then, the minimum Bayes risk achieved by the optimal deterministic parameter design is given byrdet(^) = 3w()g(sopt )d (see (6)). Similarly, let rsto(^) =
3w() g(x)popts (x)dxd represent the minimum Bayes risk achieved by the optimal stochastic parameter design, wherepopts de- notes the optimal solution for. In order for the stochastic parameter design to improve over the deterministic parameter design,rsto(^) should be strictly smaller thanrdet(^). Otherwise, it is concluded that the deterministic parameter design cannot be improved via the stochastic approach; that is, rsto(^) = rdet(^). In the following proposition, sufficient conditions presented for the latter.
Proposition 2: The deterministic parameter design cannot be im- proved via the stochastic approach if at least one of the following is satisfied for each:
1) gis a convex function;
2) the solution of the unconstrained problem (see (7)) satisfies the constraint; i.e.,ksunc k2 A.
Proof: If the second condition is satisfied, that is, ifksunc k2 A, then the solution of (14) coincides with that of the unconstrained problem in Section III-A; namely,sopt = sunc . Therefore, the solution of the optimal stochastic parameter design problem in (10) becomes popts (x) = (x 0 sopt). Hence, the deterministic design is optimal in such a scenario, and the stochastic approach is not needed.
In order to investigate the first condition, it is observed that, for anys,Efksk2g kEfsgk2is satisfied due to Jensen’s inequality since norm is a convex function. Therefore, due to the constraint Efksk2g Ain (10),kEfsgk2 Amust hold for any feasible PDF ofs. LetEfsg be defined as s = Efs1 g. As the minimizer of (14),sopt , achieves the minimumg(s) among all sthat satisfy ksk2 A,kEfsgk2 = ksk2 Aimplies thatg(Efsg) = g(s) g(sopt ) is satisfied. When g is a convex function as specified in the proposition,Efg(s)g g(Efsg) g(sopt ) is obtained from Jensen’s inequality and from the previous relation.
Therefore, for convexg,Efg(s)g can never be smaller than the minimum value of (14),g(sopt), for any PDF of sthat satisfies the average power constraint. Hence, the minimum value of (10) cannot be smaller thang(sopt ), meaning that it is always equal to g(sopt ) (since (10) covers (14) as a special case).
All in all, when at least one of the conditions in the propo- sition are satisfied for all , the deterministic and the stochastic approaches achieve the same minimum values for all parameters;
that is,g(sopt) = g(x)popts (x)dx; 8. Therefore, rdet(^) =
3w()g(sopt )d and rsto(^) = 3w() g(x)popts (x)dxd
become equal.
In order to present an example application of Proposition 2, consider a scenario in which a scalar parameter is to be estimated in the pres- ence of zero-mean additive noise n. The average power constraint is in the generic form ofEfjsj2g A for all, and the estimator is specified by ^(y) = y. In addition, the cost function is modeled as C[^(y); ] = (^(y) 0 )2. In this scenario,gin (5) can be calculated as
g(x) =
1
01
(y 0 )2pn(y 0 x)dy
=
1
01
(y + x 0 )2pn(y)dy = (x 0 )2+ Varfng (15)
whereVarfng denotes the variance of the noise. From (15), it is noted thatgis a convex function for any value of. Therefore, the first condi- tion in Proposition 2 is satisfied for all, meaning that the performance
of the deterministic parameter design cannot be improved via the sto- chastic approach.2Hence, the optimal solution can be obtained from (14), which yields
sopt = arg min
js j A (s0 )2: For example, ifA = 2, thensopt = for all .
In the following proposition, sufficient conditions are presented to specify cases in which the stochastic parameter design provides im- provements over the deterministic one.
Proposition 3: The stochastic parameter design achieves a smaller Bayes risk than the deterministic one if there exists 2 3 for which g(x) is second-order continuously differentiable around sopt and a real vectorz can be found such that
zTsopt zTrg(x)jx=s < 0 (16) and
kzk2< zTsopt zTHz
zTrg(x)jx=s (17)
wheresopt is the solution of (14),rg(x)jx=s denotes the gradient ofg(x) at x = sopt , andHis the Hessian ofg(x) at x = sopt .
Proof: In order to prove that a reduced Bayes risk can be achieved via the stochastic parameter design, consider a specific value of for which the conditions in the proposition are satisfied.
Also consider two values s;1 and s;2 around sopt , which can be expressed ass;i = sopt + i fori = 1, 2. Then, g(s;i) can be approximated asg(s;i) g(sopt ) + Ti~g+ 0:5TiHifori = 1, 2, where~g = rg(x)jx=s is the gradient andHis the Hessian of g(x) at x = sopt [17]. Similarly, ks;ik2 can be expressed as ks;ik2 ksopt k2 + 2Tisopt+ kik2 fori = 1, 2. In order to prove that employingps (x) = (x 0 s;1) + (1 0 )(x 0 s;1) results in a lower risk thang(sopt), which is the one achieved by the deterministic parameter design (see (14)), it is sufficient to show that
g(s;1) + (1 0 )g(s;2) < g sopt
ks;1k2+ (1 0 )ks;2k2< sopt 2 A (18) are satisfied for certain choice of parameters (see (10)). After inserting the expressions forg(s;i) and ks;ik2aroundsoptinto (18), it can be obtained that
T1H1+(10)T2H2+2 (1+(10)2)T ~g<0
k1k2+(10)k2k2+2 (1+(10)2)Tsopt <0: (19) Let1 = z and 2= z. Then, (19) can be manipulated to obtain
zTHz + k(zT~g) < 0 and kzk2+ k zTsopt < 0 (20)
withk =1 2(+(10))
( +(10) ). If the first inequality in (20) is multiplied by (z s(z ~g )), which is always negative due to the condition (16) in the proposition, (20) becomes
(zTHz) zTsopt
(zT~g) + k zTsopt > 0 and
kzk2+ k zTsopt < 0: (21)
2It can be shown thatg is convex for all also for the absolute error cost function; i.e.,C[^(y); ] = j^(y) 0 j.
Sincek can take any real value by adjusting 2 [0; 1] and infinites- imally small and values, it is guaranteed that both inequalities in (21) can be satisfied if(zTHz)(z s(z ~g )) is larger thankzk2, which corresponds to (17).
Remark 2: For the conditions in (16) and (17) to be satisfied, g(x) must be concave at x = sopt(i.e.,Hmust be negative-definite) sincekzk2 is always nonnegative and (z rg (x)j(z s ) ) is negative due to (16).
Proposition 3 provides a simple approach, based on the first and second order derivatives ofg, to determine if the stochastic param- eter design can provide improvements over the deterministic one. If the conditions are satisfied, the improvements are guaranteed and the optimization problem in (12) or (13) can be solved to obtain the optimal solution. However, since the conditions are sufficient but not necessary, there can also exist certain scenarios in which improvements are ob- served although the conditions are not satisfied. Examples for various scenarios are provided in the next section.
V. NUMERICALRESULTS ANDCONCLUSIONS
In order to present examples of the theoretical results in the previous sections, consider an estimation problem in which a scalar parameter is estimated based on observation y that is modeled asy = s+n, with n denoting the additive noise component. (Although a scalar problem is considered for convenience, vector parameter estimation problems can be treated in a similar fashion (per component) when the noise components are independent and the cost function is additive [2].) The noise n is modeled by a Gaussian mixture distribution, specified as pn(n) = Ll=1 lexp 0(n0 )(2 ) =(p
2l), where the parameters are chosen in such a way to generate a zero-mean noise component.
In addition, the estimator is given by ^(y) = y, and the cost func- tion is selected as the uniform cost function, which is expressed as C[^(y); ] = 1 if j^(y) 0 j > 1 and C[^(y); ] = 0 otherwise.
Based on this model,gin (5) can be obtained as
g(x)= L
l=1
l Q x0+l+1
l +Q 0x+0l+1
l (22)
whereQ(x) = p12 x1exp 0t2 dt denotes the Q-function. Re- garding the constraint in (8),Efjsj2g 2is considered for each.
For the numerical examples, parameter is modeled to lie between 010 and 10; that is, the parameter space is specified as 3 = [010; 10].
Also,s can take values in the interval [010, 10] under the average power constraint,Efjsj2g 2. In addition, the parameters of the Gaussian mixture noise n are selected as 1 = 0:33, 2 = 0:13, 3 = 0:08, 4 = 0:07, 5 = 0:11, 6 = 0:28, 1 = 03:8,
2 = 01:6, 3 = 00:51, 4 = 0:4657, 5 = 2:42, 6 = 4:3, andl = 0:5; 8l. With this selection of the parameters, the noise be- comes a zero-mean random variable so that ^(y) = y can be regarded as a practical estimator.3Finally,1 = 1 is considered for the uniform cost function described in the previous paragraph.
In Fig. 2, the conditional risks (i.e.,Efg(s)g in (6)) are plotted versus for various parameter design approaches. For the optimal sto- chastic parameter design, both the exact solution obtained from (12) and the convex relaxation solutions obtained from (13) are plotted.
In the convex relaxation approach, the set of possible values fors
are selected between010 and 10 with an increment of D (in short,
3Although this is not an optimal estimator, it can be used in practice due to its simplicity compared to the optimal estimator, which would have high com- plexity due to the multimodal noise structure.
Fig. 2. Conditional risk versus for various parameter design approaches.
Fig. 3. g (x) in (22) for various values of .
010 : D : 10), and the results for D = 0:25 and D = 0:5 are illus- trated in the figure. The results for the optimal deterministic param- eter design are calculated from (14). In addition, the results obtained from the unconstrained problem (see (7)) and those obtained by using ps (x) = (x 0 ) (labeled as “Conventional”) are shown in the figure to provide performance benchmarks. It is observed that the optimal sto- chastic parameter design achieves the minimum conditional risks for all values in the presence of the average power constraint. It provides performance improvements over the deterministic parameter design for certain range of parameter values, e.g., for > 2:1. In addition, both the stochastic and the deterministic design approaches achieve the same conditional risks as the unconstrained solution for some values, which is due to the fact that the unconstrained solutions satisfy the average power constraint for those values of. Furthermore, the convex relax- ation approaches (which provide low complexity solutions) perform very closely to the exact solutions of the optimal stochastic parameter design problem for small values of D.
In order to provide further explanations of the results in Fig. 2, Fig. 3 illustratesg(x) in (22) for = 05, = 0, and = 5. As expected from the expression in (22), each function in the figure is a shifted ver- sion of the others. Also, this figure can be used to determine when the unconstrained solution coincides with the solutions of the optimal sto- chastic and the optimal deterministic parameter designs. For example, for = 05, the global minimum of g(x) is achieved at 01.223, which
TABLE I
OPTIMALSTOCHASTICSOLUTIONp (x) = (x 0 s ) + (1 0
)(x 0 s ), OPTIMALDETERMINISTICSOLUTIONs ,AND UNCONSTRAINEDSOLUTIONs
already satisfies the constraint. Therefore, all the three approaches yield the same conditional risk for that parameter (see Fig. 2). On the other hand, for = 5, the global minimum is at 8.777; hence, the conditional risk obtained from the unconstrained problem in (7) cannot be achieved by the constrained approaches. Specifically, the optimal deterministic approach in (14) chooses the minimum value in the interval [05, 5], which results in the optimal signal value ofsopt = 0:81. On the other hand, the solution of the optimal stochastic parameter design problem in (12) results in a randomization between 8.741 and 0.809 with prob- abilities of 0.321 and 0.679, respectively, and achieves a lower condi- tional risk than the deterministic approach (see Fig. 2). In Table I, the optimal solutions for the optimal stochastic, the optimal deterministic and the unconstrained parameter design approaches are presented for various values of. Fig. 3 can also be used to explain the oscillatory behavior of the convex relaxation solutions in Fig. 2. Since the convex relaxation approach considers possiblesvalues as010 : D : 10 and sinceg(x) shifts with , the signal values obtained from the convex op- timization problem in (13) move around the optimal values of the exact solution periodically. Finally, the conditions in Proposition 3 are evalu- ated for different values, and it is observed that they provide sufficient but not necessary conditions for specifying improvements via the sto- chastic parameter design over the deterministic one. For example, the calculations show that the conditions in Proposition 3 are satisfied for
2 [01:381; 01:31] and 2 [1:397; 1:536], and improvements are observed in Fig. 2 for those values of.
Future work involves the investigation of the stochastic parameter design problem in the presence of partial knowledge of the noise dis- tribution. The robustness of the stochastic parameter design will be an- alyzed, and various design approaches will be considered.
ACKNOWLEDGMENT
The authors would like to thank the editor, Dr. T.-H. Li, for sug- gesting the example in the fourth paragraph of the introduction.
REFERENCES
[1] S. M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory. Upper Saddle River, NJ: Prentice-Hall, 1993.
[2] H. V. Poor, An Introduction to Signal Detection and Estimation. New York: Springer-Verlag, 1994.
[3] H. Chen, P. K. Varshney, S. M. Kay, and J. H. Michels, “Theory of the stochastic resonance effect in signal detection: Part I—Fixed de- tectors,” IEEE Trans. Signal Process., vol. 55, no. 7, pp. 3172–3184, Jul. 2007.
[4] S. M. Kay, “Noise enhanced detection as a special case of randomiza- tion,” IEEE Signal Process. Lett., vol. 15, pp. 709–712, 2008.
[5] C. Goken, S. Gezici, and O. Arikan, “Optimal stochastic signaling for power-constrained binary communications systems,” IEEE Trans.
Wireless Commun., vol. 9, no. 12, pp. 3650–3661, Dec. 2010.
[6] H. Chen, P. K. Varshney, and J. H. Michels, “Noise enhanced pa- rameter estimation,” IEEE Trans. Signal Process., vol. 56, no. 10, pp.
5074–5081, Oct. 2008.
[7] B. Dulek and S. Gezici, “Detector randomization and stochastic signaling for minimum probability of error receivers,” IEEE Trans.
Commun., vol. 60, no. 4, pp. 923–928, Apr. 2012.
[8] A. Ozcelikkale, H. M. Ozaktas, and E. Arikan, “Signal recovery with cost-constrained measurements,” IEEE Trans. Signal Process., vol. 58, no. 7, pp. 3607–3617, Jul. 2010.
[9] K. Sohraby, D. Minoli, and T. Znati, Wireless Sensor Networks: Tech- nology, Protocols, and Applications. Hoboken, NJ: Wiley, 2007.
[10] A. Patel and B. Kosko, “Optimal noise benefits in Neyman–Pearson and inequality-constrained signal detection,” IEEE Trans. Signal Process., vol. 57, no. 5, pp. 1655–1669, May 2009.
[11] M. Azizoglu, “Convexity properties in binary detection problems,”
IEEE Trans. Inf. Theory, vol. 42, no. 4, pp. 1316–1321, Jul. 1996.
[12] D. P. Bertsekas, A. Nedic, and A. E. Ozdaglar, Convex Analysis and Optimization. Boston, MA: Athena Specific, 2003.
[13] S. Bayram, N. D. Vanli, B. Dulek, I. Sezer, and S. Gezici, “Optimum power allocation for average power constrained jammers in the presence of non-Gaussian noise,” IEEE Commun. Lett., 2012, DOI:
10.1109/LCOMM.2012.052112.120098, preprint.
[14] K. E. Parsopoulos and M. N. Vrahatis, “Particle swarm optimization method for constrained optimization problems,” in Intelligent Tech- nologies-Theory and Applications: New Trends in Intelligent Technolo- gies. Amsterdam, The Netherlands: IOS Press, 2002, pp. 214–220.
[15] K. V. Price, R. M. Storn, and J. A. Lampinen, Differential Evolution:
A Practical Approach to Global Optimization. New York: Springer, 2005.
[16] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge Univ. Press, 2004.
[17] S. Bayram and S. Gezici, “On the improvability and nonimprovability of detection via additional independent noise,” IEEE Signal Process.
Lett., vol. 16, no. 11, pp. 1001–1004, Nov. 2009.
On the Proper Forms of BIC for Model Order Selection Petre Stoica and Prabhu Babu
Abstract—The Bayesian Information Criterion (BIC) is often presented in a form that is only valid in large samples and under a certain condition on the rate at which the Fisher Information Matrix (FIM) increases with the sample length. This form has been improperly used previously in situations in which the conditions mentioned above do not hold. In this correspon- dence, we describe the proper forms of BIC in several practically relevant cases that do not satisfy the above assumptions. In particular, we present a new form of BIC for high signal-to-noise ratio (SNR) cases. The conclusion of this study is that BIC remains one of the most successful existing rules for model order selection, if properly used.
Index Terms—BIC, model order selection, polynomial trend model.
I. INTRODUCTION AND THEPROBLEMFORMULATION
BIC is applicable to a general class of models that are essentially only required to satisfy the regularity conditions under which the max- imum-likelihood estimation (MLE) method is asymptotically statisti-
Manuscript received December 27, 2011; revised March 14, 2012 and May 07, 2012; accepted May 07, 2012. Date of publication June 06, 2012; date of current version August 07, 2012. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Ta-Hsin Li. This work was supported in part by the Swedish Research Council (VR) and the European Research Council (ERC).
The authors are with the Department of Information Technology, Uppsala University, Uppsala, SE 75105, Sweden (e-mail:prabhu.babu@it.uu.se ).
Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TSP.2012.2203128
cally efficient. However in what follows we focus on a linear-regression model, which is sufficient to illustrate the main points we want to make.
Therefore, consider the model:
yyy = 8n n+ eeen (1) whereyyy 2 N21is the data vector (withN being the number of data samples),8n 2 N2(n01)is the regression matrix (which is given),
n 2 (n01)21 is an unknown parameter vector, andeeen 2 N21 is a noise term; we assume thateeenis normally distributed with zero mean and covariance matrix equal to2nIII, where 2nis unknown as well. The integer subindexn in (1) indicates the order (or dimension) of the model, and it is also unknown. LetHn denote the hypothesis that the data satisfy (1). In many cases, but not always, the hypotheses H1; H2; H3; . . . are nested (i.e., 8nis a sub-block of8n, forn < n).
Given the data vectoryyy, the problem associated with (1) is to estimate n; nand2n. UnderHn, the MLEs of nandn2 are well known to be:
^
n= 8Tn8n
018Tnyyy (2)
^2n= kyyy 0 8n ^ nk2=N (3) Hereafter we assume that the inverse matrix in (2) exists forn 2 [1; ~n] (where ~n is a given upper bound on the values of n that are deemed to be of interest). In view of (2), (3) the “only” problem left is the estimation ofn, which is the main topic of the following sec- tions.
II. BIC: THEBASICFORM
Letp(yyy j n; Hn) denote the probability density function (pdf) of the data underHn, where the vectorncomprises all the parameters of the model. For example, in the case of (1),
n= Tn; 2n T (4) and
p(yyy j n; Hn) = e0kyyy08 k =2
(2n2)N=2 (5)
Furthermore, letp(nj Hn) be the prior pdf of the parameter vector.
Then
p(yyy j n; Hn)p(nj Hn) = p(yyy; nj Hn) (6) and therefore
p(yyy j n; Hn)p(nj Hn)dn= p(yyy j Hn) (7) Assuming that the hypothesesfHng are equi-probable, maximizing p(yyy j Hn) with respect to n yields the maximum a-posteriori (MAP) estimate ofn, i.e.,
arg max
n2[1;~n]p(Hnj yyy) (8)
This estimate ofn is well known (see, e.g., [1], [2] and references there) to have the desirable property of maximizing the total probability of correct detection, that is
~ n n=1
prob[(decide Hn) j Hn] (9)
However in general there is hardly any agreement as to how the prior pdfp(nj Hn) should be chosen and, even when a qualified choice of this pdf is possible, the evaluation of the integral in (7) may well be 1053-587X/$31.00 © 2012 IEEE
