LS-SVM based solution for delay differential
equations
Siamak Mehrkanoon and Johan A. K. Suykens
KU Leuven, ESAT-SCD, Kasteelpark Arenberg 10, B-3001 Leuven (Heverlee), Belgium E-mail: {Siamak.Mehrkanoon,Johan.Suykens}@esat.kuleuven.be
Abstract. In this paper a new approach based on Least Squares Support Vector Machines (LS-SVMs) is proposed for solving delay differential equations (DDEs) with single-delay. The proposed method provides a closed form approximate model based solution without using any interpolation techniques. The result of this paper can be extended for DDE with multi-lags. The results of some numerical experiments are presented and compared with analytic solutions to confirm the validity and applicability of the proposed method.
1. Introduction
Delay differential equations (DDEs) can be found in the mathematical formulation of real life phenomena in a wide variety of applications especially in science and engineering such as population dynamics, infectious disease and control problems [1]. In contrast with ordinary differential equations (ODEs) where the unknown function and its derivatives are evaluated at the same time instant, in a DDE the evolution of the system at a certain time, depends on the state of the system at an earlier time. A general form of first order multi-delay DDE is given by:
˙x(t) = f (t, x(t), x(t − τ1), . . . , x(t − τn)), t ≥ tin,
x(t) = g(t), t ≤ tin (1)
where g(t) is the initial function and {τi}ni=1 are the delays or lags which are non-negative and
can be constant, time dependent or state dependent. The term x(t − τi) is called the delay term.
Due to the presence of the delay term, numerical methods that provide only discrete solutions at the grid points are not suitable for DDE and one needs to apply interpolation techniques, for approximating the delay term, in order to advance the solution of the given DDE.
Least Squares Support Vector Machines (LS-SVMs) is a methodology which has been applied in a wide variety of fields such as regression and classification [3]. Like in support vector machines [6], in this method one maps the data into a high dimensional feature space using a feature map. Solving ordinary differential equations (ODEs) and differential algebraic equations (DAEs) using LS-SVMs have been discussed in [4, 5]. In this paper the method developed in [4] is extended to approximate the solution of a scalar DDE. The method provides a closed form approximate model based solution without using any interpolation techniques.
Making use of Mercer’s Theorem [6], derivatives of the feature map can be written in terms of derivatives of the kernel function.
Let us define the following differential operator which will be used in the subsequent section ∇mn ≡ ∂
n+m
∂pn∂qm. (2)
When ϕ(p)Tϕ(q) = K(p, q), one can show then that
[ϕ(n)(p)]Tϕ(m)(q) = ∇mn[ϕ(p)Tϕ(q)] = ∇mn[K(p, q)] = ∂
n+mK(p, q)
∂pn∂qm . (3)
Using formula (3), it is possible to express all derivatives of the feature map in terms of the kernel function itself (provided that the kernel function is sufficiently differentiable).
Suppose that ti∈ ̺ and vj ∈ τ are two arbitrary points and ̺ and τ ⊆ R. Now for notational
convenience let us list the following notations which are used in the following sections. ∇mnK (t, y) = ∇mnK(p, q) p=t,q=y , ̺Ωτ i,j = ∇00[K(p, q)] p=ti,q=vj = K(ti, vj), ̺ nΩτm i,j = ∇mn K(p, q) p=ti,q=vj = ∂ n+mK(p, q) ∂pn∂qm p=ti,q=vj , where ̺ nΩτm i,j
denotes the (i, j)-th entry of matrix ̺nΩτm. In the case that ̺ = τ , we denote
the matrix by̺nΩm.
2. Formulation of the method for DDE
Consider a scalar linear DDE with single-delay of the form
˙x(t) = a(t) + b(t)x(t) + c(t)x(t − τ (t)), tin≤ t ≤ tf,
x(t) = g(t), t ≤ tin, (4)
where ˙g(t−in) = ˙x(t+in). Assume that a general approximate solution to equation (4) is of the form of ˆx(t) = wTϕ(t) + d, where ϕ(·) : R → Rh is the feature map and h is the dimension
of the feature space. To obtain the optimal value of w and d, a collocation method is used [2] with a discretization of the interval [tin, tf] into a set of points {ti}Ni=1. The estimated solution
follows from min w,d,ei 1 2w Tw+ γ 2 m X i=1 e2i s.t. wT ϕ′(ti) − biϕ(ti) − ciϕ(si)− (bi+ ci)d = ai+ ei, i = 2, . . . , N wTϕ(vk) + d = g(vk), k = 1, . . . , M (5)
where si = ti − τ (ti) and vk = {sk| sk ≤ tin}. N is the number of collocation points (which
is equal to the number of training points). Furthermore ai = a(ti), bi = b(ti) and ci = c(ti).
Problem 5 is obtained by combining the LS-SVM cost function with constraints constructed by imposing the approximate solution ˆx(t) = wTϕ(t) + d, given by the LS-SVM model, to
satisfy the given differential equation with corresponding initial function at collocation points {ti}Ni=1. Problem (5) is a quadratic minimization under linear equality constraints, which enables
Lemma 2.1. Given a positive definite kernel function K : R × R → R with K(t, s) = ϕ(t)Tϕ(s)
and a regularization constant γ ∈ R+, the solution to (5) is obtained by solving the following
dual problem: K + IN −1/γ U −H UT V 0Ω0 1M −HT 1 M 0 α β d = Fa G 0 (6) with α= [α2, . . . , αN]T, β = [β2, . . . , βN]T, Fa = [a(t2), . . . , a(tN)]T ∈ RN −1, Fb= [b(t2), . . . , b(tN)]T ∈ RN −1, Fc= [c(t2), . . . , c(tN)]T ∈ RN −1, G = [g(t2), . . . , g(tN)]T ∈ RN −1, K = ¯D ¯Ω ¯DT, U = ¯D ∆T, H = Fb+ Fc ¯ Ω = T 1Ω1 T 0Ω1 S 0Ω T 1 T 1Ω0 T0Ω0 S0ΩT0 T 1Ω S 0 T 0Ω S 0 S 0Ω0 ∈ R3(N −1)×3(N −1), ¯D = [1, −Db, −Dc], ∆ = [ V 0Ω T 1, V 0Ω T 0, V 0Ω S 0]
Db and Dc are diagonal matrices with the elements of Fb and Fc on the main diagonal
respectively. 1 is the identity matrix. In addition S = {si}Ni=2, V = {vi}Mi=1 and T = {ti}Ni=2.
The solution in the dual form becomes: ˆ x(t) = N X i=2 αi [∇01K](ti, t) − b(ti)[∇00K](ti, t) − c(ti)[∇00K](si, t) + M X k=1 βk[∇00K](vk, t) + d,
where [∇00K](t, s) = K(t, s) = ϕ(t)Tϕ(s) is the kernel function and [∇0
1K](t, s) = ϕ ′
(t)Tϕ(s)
is its derivative. Here the Gaussian RBF kernel is used. αi and βk are Lagrange multipliers
associated with (3). The result of this paper can be extended for linear DDE with multi-lags. 3. Numerical Result
Problem 3.1: Consider the following scalar DDE,
˙x(t) = −x(t − 1 + exp(−t)) + sin(t − 1 + exp(−t)) + cos(t), t ≥ 0, x(t) = sin(t), t ≤ 0
The problem is solved on domain t ∈ [0, 40] for N = 200. Figure 1 shows the residuals e(t) = x(t) − bx(t).
Problem 3.2: Consider the delay equation with asymptotically vanishing-lag:
˙x(t) = t
4− 3
(t5+ t) ln(t − t−3+ [t − t−3]−3)x(t − t
−3), t ∈ [2, 30],
x(t) = ln(t + t−3), t ∈ [1.5, 2] whose exact solution is x(t) = ln(t + t−3
). The result obtained by the proposed method method, with N = 400, is depicted in Figure 2.
4. Conclusion
In this paper, we have presented a method based on LS-SVMs for the numerical solution of delay differential equations. The proposed method provides a close form approximate solution for the problem without using any interpolation techniques.
0 10 20 30 40 −1 −0.5 0 0.5 1 1.5 Exact solution Approximate solution t x (t ) (a) 0 10 20 30 40 −3 −2 −1 0 1x 10 −6 t e( t) (b)
Figure 1. (a) Numerical results for Problem 3.1, when 200 equidistant points in [0,40] are used for training. (b) Obtained model errors for problem 3.1.
5 10 15 20 25 30 0.5 1 1.5 2 2.5 3 3.5 Exact solution Approximate solution t x (t ) (a) 5 10 15 20 25 30 10−7 10−6 10−5 10−4 10−3 t e( t) (b)
Figure 2. (a) Numerical results for Problem 3.2, when 400 equidistant points in [2,30] are used for training. (b) Obtained model errors for problem 3.2.
Acknowledgments
This work was supported by: • Research Council KUL: GOA/10/09 MaNet, PFV/10/002 (OPTEC), several PhD/postdoc & fellow grants • Flemish Government: ◦ IOF: IOF/KP/SCORES4CHEM; ◦ FWO: PhD/postdoc grants, projects: G.0320.08 (convex MPC), G.0558.08 (Robust MHE), G.0557.08 (Glycemia2), G.0588.09 (Brain-machine), G.0377.09 (Mechatronics MPC); G.0377.12 (Structured systems) research community (WOG: MLDM); ◦ IWT: PhD Grants, projects: Eureka-Flite+, SBO LeCoPro, SBO Climaqs, SBO POM, O&O-Dsquare • Belgian Federal Science Policy Office: IUAP P7/ (DYSCO, Dynamical systems, control and optimization, 2012-2017) • IBBT • EU: ERNSI, EMBOCON (ICT-248940), FP7-SADCO (MC ITN-264735), ERC ST HIGHWIND (259 166), ERC AdG A-DATADRIVE-B • COST: Action ICO806: IntelliCIS • Contract Research: AMINAL • Other: ACCM. Johan Suykens is a professor at the KU Leuven, Belgium.
References
[1] Driver R D 1977 Ordinary and Delay Differential Equations (New York: Springer-Verlag New York Inc). [2] Kincaid D R and Cheney E W 2002 Numerical Analysis, (Mathematics of Scientific Computing, third ed.,
Brooks/Cole, Pacific Grove, CA).
[3] Suykens J A K, Van Gestel T, De Brabanter J, De Moor B and Vandewalle J 2002 Least Squares Support Vector Machines(World Scientific, Singapore.)
[4] Mehrkanoon S, Falck T and Suykens J A K 2012 Approximate Solutions to Ordinary Differential Equations Using Least Squares Support Vector Machines IEEE Transactions on Neural Networks and Learning Systems, 23 1356-67.
[5] Mehrkanoon S and Suykens J A K 2012 LS-SVM approximate solution to linear time varying descriptor systems Automatica, 48, 2502-2511.
