DOI 10.1007/s10589-010-9359-x
Neural networks for solving second-order cone constrained variational inequality problem
Juhe Sun· Jein-Shan Chen · Chun-Hsu Ko
Received: 25 April 2010 / Published online: 6 October 2010
© Springer Science+Business Media, LLC 2010
Abstract In this paper, we consider using the neural networks to efficiently solve the second-order cone constrained variational inequality (SOCCVI) problem. More specifically, two kinds of neural networks are proposed to deal with the Karush-Kuhn- Tucker (KKT) conditions of the SOCCVI problem. The first neural network uses the Fischer-Burmeister (FB) function to achieve an unconstrained minimization which is a merit function of the Karush-Kuhn-Tucker equation. We show that the merit function is a Lyapunov function and this neural network is asymptotically stable.
The second neural network is introduced for solving a projection formulation whose solutions coincide with the KKT triples of SOCCVI problem. Its Lyapunov stability and global convergence are proved under some conditions. Simulations are provided to show effectiveness of the proposed neural networks.
Keywords Second-order cone· Variational inequality · Fischer-Burmeister function· Neural network · Lyapunov stable · Projection function
J. Sun is also affiliated with Department of Mathematics, National Taiwan Normal University.
J.-S. Chen is member of Mathematics Division, National Center for Theoretical Sciences, Taipei Office. The author’s work is partially supported by National Science Council of Taiwan.
J. Sun
School of Science, Shenyang Aerospace University, Shenyang 110136, China e-mail:juhesun@abel.math.ntnu.edu.tw
J.-S. Chen (
)Department of Mathematics, National Taiwan Normal University, Taipei 11677, Taiwan e-mail:jschen@math.ntnu.edu.tw
C.-H. Ko
Department of Electrical Engineering, I-Shou University, Kaohsiung 840, Taiwan e-mail:chko@isu.edu.tw
1 Introduction
Variational inequality (VI) problem, which was introduced by Stampacchia and his collaborators [19,29,30,35,36], has attracted much attention from researchers of engineering, mathematics, optimization, transportation science, and economics com- munities, see [1,25,26]. It is well known that VIs subsume many other mathematical problems, including the solution of systems of equations, complementarity problems, and a class of fixed point problems. For a complete discussion and history of the finite VI problem and its associated solution methods, we refer the interested readers to the excellent survey text by Facchinei and Pang [13], the monograph by Patriksson [34], the survey article by Harker and Pang [18], the Ph.D. thesis of Hammond [16] and the references therein.
In this paper, we are interested in solving the second-order cone constrained varia- tional inequality (SOCCVI) problem whose constraints involve the Cartesian product of second-order cones (SOCs). The problem is to find x∈ C satisfying
F (x), y − x ≥ 0, ∀y ∈ C, (1)
where the set C is finitely representable as
C= {x ∈ Rn: h(x) = 0, −g(x) ∈ K}. (2) Here·, · denotes the Euclidean inner product, F : Rn→ Rn, h: Rn→ Rl and g: Rn→ Rmare continuously differentiable functions andK is a Cartesian product of second-order cones (or Lorentz cones), expressed as
K = Km1× Km2× · · · × Kmp, (3) where l≥ 0, m1, m2, . . . , mp≥ 1, m1+ m2+ · · · + mp= m, and
Kmi:= {(xi1, xi2, . . . , ximi)T ∈ Rmi | (xi2, . . . , ximi) ≤ xi1}
with· denoting the Euclidean norm and K1the set of nonnegative realsR+.A spe- cial case of (3) isK = Rn+,namely the nonnegative orthant inRn,which corresponds to p= n and m1= · · · = mp= 1. When h is affine, an important special case of the SOCCVI problem corresponds to the KKT conditions of the convex second-order cone program (CSOCP):
min f (x)
s.t. Ax= b, −g(x) ∈ K, (4)
where A∈ Rl×n has full row rank, b∈ Rl, g: Rn→ Rmand f : Rn→ R. Further- more, when f is a convex twice continuously differentiable function, problem (4) is equivalent to the following SOCCVI problem: Find x∈ C satisfying
∇f (x), y − x ≥ 0, ∀y ∈ C, where
C= {x ∈ Rn: Ax − b = 0, −g(x) ∈ K}.
For solving the constrained variational inequalities and complementary problems (CP), many computational methods have been proposed, see [3,4,6,8,13,39] and references therein. These methods include the method based on merit function, in- terior method, Newton method, nonlinear equation method, projection method and its variant versions. Another class of techniques for solving the VI problem exploits the fact that the KKT conditions of a VI problem comprise a mixed complementar- ity problem (MiCP), involving both equations and nonnegativity constraints. In other words, the SOCCVI problem can be solved by analyzing its KKT conditions:
⎧⎪
⎨
⎪⎩
L(x, μ, λ)= 0,
g(x), λ = 0, −g(x) ∈ K, λ ∈ K, h(x)= 0,
(5)
where L(x, μ, λ)= F (x) + ∇h(x)μ + ∇g(x)λ is the variational inequality La- grangian function, μ∈ Rland λ∈ Rm.However, in many scientific and engineering applications, it is desirable to have a real-time solution for the VI and CP problems.
Thus, at present, for solving the VI and CP problems, many researchers employ the neural network method which is a promising way to overcome this problem.
Neural networks for optimization were first introduced in the 1980s by Hopfield and Tank [20,38]. Since then, neural networks have been applied to various optimiza- tion problems, including linear programming, nonlinear programming, variational inequalities, and linear and nonlinear complementarity problems; see [5,9–11,17, 21,22,24,28,40–44]. The main idea of the neural network approach for optimiza- tion is to construct a nonnegative energy function and establish a dynamic system that represents an artificial neural network. The dynamic system is usually in the form of first order ordinary differential equations. Furthermore, it is expected that the dynamic system will approach its static state (or an equilibrium point), which corresponds to the solution for the underlying optimization problem, starting from an initial point. In addition, neural networks for solving optimization problems are hardware-implementable; that is, the neural networks can be implemented by using integrated circuits. In this paper, we focus on neural network approach to the SOCCVI problem. Our neural networks will be aimed to solve the system (5) whose solutions are candidates of SOCCVI problem (1).
The rest of this paper is organized as follows. Section2introduces some prelim- inaries. In Sect.3, the first neural network based on the Fischer-Burmeister function is proposed and studied. In Sect.4, we show that the KKT system (5) is equivalent to a nonlinear projection formulation. Then, the model of neural network for solv- ing the projection formulation is introduced and its stability is analyzed. In Sect.5, illustrative examples are discussed. Section6gives the conclusion of this paper.
2 Preliminaries
In this section, we recall some preliminary results that will be used later and back- ground materials of ordinary differential equations that will play an important role in the subsequent analysis. We begin with some concepts for a nonlinear mapping.
Definition 2.1 Let F = (F1, . . . , Fn)T : Rn→ Rn. Then, the mapping F is said to be
(a) monotone if
F (x) − F (y), x − y ≥ 0, ∀x, y ∈ Rn. (b) strictly monotone if
F (x) − F (y), x − y > 0, ∀x, y ∈ Rn. (c) strongly monotone with constant η > 0 if
F (x) − F (y), x − y ≥ ηx − y2, ∀x, y ∈ Rn. (d) F is said to be Lipschitz continuous with constant γ if
F (x) − F (y) ≤ γ x − y, ∀x, y ∈ Rn.
Definition 2.2 Let X be a closed convex set inRn. Then, for each x∈ Rn,there is a unique point y∈ X such that x − y ≤ x − z, ∀z ∈ X. Here y is known as the projection of x onto the set X with respect to Euclidean norm, that is,
y= PX(x)= arg min
z∈Xx − z.
The projection function PX(x)has the following property, called projection theo- rem [2], which is useful in our subsequent analysis.
Property 2.1 Let X be a nonempty closed convex subset ofRn. Then, for each z∈ Rn, PX(z)is the unique vector¯z ∈ X such that (y − ¯z)T(z− ¯z) ≤ 0, ∀y ∈ X.
Next, we recall some materials about first order differential equations (ODE):
˙w(t) = H(w(t)), w(t0)= w0∈ Rn, (6) where H: Rn→ Rnis a mapping. We also introduce three kinds of stability that will be discussed later. These materials can be found in usual ODE textbooks, e.g. [31].
Definition 2.3 A point w∗= w(t∗)is called an equilibrium point or a steady state of the dynamic system (6) if H (w∗)= 0. If there is a neighborhood ∗⊆ Rnof w∗ such that H (w∗)= 0 and H(w) = 0 ∀w ∈ ∗\ {w∗}, then w∗is called an isolated equilibrium point.
Lemma 2.1 Assume that H: Rn→ Rnis a continuous mapping. Then, for any t0>0 and w0∈ Rn, there exists a local solution w(t) for (6) with t∈ [t0, τ )for some τ > t0. If, in addition, H is locally Lipschitz continuous at w0, then the solution is unique; if His Lipschitz continuous inRn, then τ can be extended to∞.
If a local solution defined on[t0, τ )cannot be extended to a local solution on a larger interval[t0, τ1), τ1> τ,then it is called a maximal solution, and the interval [t0, τ )is the maximal interval of existence. Clearly, any local solution has an exten- sion to a maximal one. We denote[t0, τ (w0))by the maximal interval of existence associated with w0.
Lemma 2.2 Assume that H: Rn→ Rnis continuous. If w(t) with t∈ [t0, τ (w0))is a maximal solution and τ (w0) <∞, then limt↑τ(w0)w(t) = ∞.
Definition 2.4 (Lyapunov Stability) Let w(t) be a solution for (6). An isolated equi- librium point w∗is Lyapunov stable if for any w0= w(t0)and any ε > 0, there exists a δ > 0 such that ifw(t0)− w∗ < δ, then w(t) − w∗ < ε for all t ≥ t0.
Definition 2.5 (Asymptotic Stability) An isolated equilibrium point w∗is said to be asymptotic stable if in addition to being Lyapunov stable, it has the property that if
w(t0)− w∗ < δ, then w(t) → w∗as t→ ∞.
Definition 2.6 (Lyapunov function) Let ⊆ Rn be an open neighborhood of ¯w.
A continuously differentiable function V: Rn→ R is said to be a Lyapunov function at the state ¯w over the set for (6) if
V (¯w) = 0, V (w) > 0, ∀w ∈ \ { ¯w},
˙V (w) ≤ 0, ∀w ∈ \ { ¯w}. (7)
Lemma 2.3
(a) An isolated equilibrium point w∗ is Lyapunov stable if there exists a Lyapunov function over some neighborhood ∗of w∗.
(b) An isolated equilibrium point w∗ is asymptotically stable if there exists a Lya- punov function over some neighborhood ∗ of w∗ such that ˙V (w) <0, ∀w ∈
∗\ {w∗}.
Definition 2.7 (Exponential Stability) An isolated equilibrium point w∗is exponen- tially stable if there exists a δ > 0 such that arbitrary point w(t) of (6) with the initial condition w(t0)= w0andw(t0)− w∗ < δ is well defined on [t0,+∞) and satisfies
w(t) − w∗ ≤ ce−ωtw(t0)− w∗ ∀t ≥ t0, where c > 0 and ω > 0 are constants independent of the initial point.
3 Neural network model based on smoothed Fischer-Burmeister function The smoothed Fischer-Burmeister function over the second-order cone defined be- low is used to construct a merit function by which the KKT system of SOCCVI is reformulated as an unconstrained smooth minimization problem. Furthermore, based
on the minimization problem, we propose a neural network and study its stability in this section.
For any a= (a1; a2), b= (b1; b2)∈ R × Rn−1,we define their Jordan product as a· b = (aTb; b1a2+ a1b2).
We denote a2= a · a and |a| =√
a2, where for any b∈ Kn,√
bis the unique vector inKnsuch that b=√
b·√ b.
Definition 3.1 A function φ: Rn× Rn→ Rn is called an SOC-complementarity function if it satisfies
φ (a, b)= 0 ⇐⇒ a · b = 0, a ∈ Kn, b∈ Kn.
A popular SOC-complementarity function is the Fischer-Burmeister function, which is semismooth [33] and defined as
φFB(a, b)=
a2+ b21/2
− (a + b).
Then the smoothed Fischer-Burmeister function is given by
φFBε (a, b)=
a2+ b2+ ε2e
1/2
− (a + b) (8)
with ε∈ R+and e= (1, 0, . . . , 0)T ∈ Rn.
The following lemma gives the gradient of φFBε . Since the proofs can be found in [14,33,37], we here omit them.
Lemma 3.1 Let φFBε be defined as in (8) and ε = 0. Then, φFBε is continuously differ- entiable everywhere and
∇εφFBε (a, b)= eTL−1z Lεe, ∇aφFBε (a, b)= L−1z La− I,
∇bφFBε (a, b)= L−1z Lb− I,
where z=
a2+ b2+ ε2e1/2
, I is identity mapping and La= a1 aT
2
a2a1In−1
for a= (a1; a2)∈ R × Rn−1.
Using Definition3.1and KKT condition described in [37], we can see that the KKT system (5) is equivalent to the following unconstrained smooth minimization problem:
min (w):=1
2S(w)2. (9)
Here (w), w= (ε, x, μ, λ) ∈ R1+n+l+m, is a merit function, and S(w) is defined by
S(w)=
⎛
⎜⎜
⎜⎜
⎜⎜
⎜⎝
ε L(x, μ, λ)
−h(x) φε
FB(−gm1(x), λm1) ...
φε
FB(−gmp(x), λmp)
⎞
⎟⎟
⎟⎟
⎟⎟
⎟⎠ ,
with gmi(x), λmi∈ Rmi.In other words, (w) given in (9) is a smooth merit function for the KKT system (5).
Based on the above smooth minimization problem (9), it is natural to propose the first neural network for solving the KKT system (5) of SOCCVI problem:
dw(t )
dt = −ρ∇ (w(t)), w(t0)= w0, (10) where ρ > 0 is a scaling factor.
Remark 3.1 In fact, we can also adopt another merit function which is based on the FB function without the element ε. That is, we can define
S(x, μ, λ)=
⎛
⎜⎜
⎜⎜
⎜⎝
L(x, μ, λ)
−h(x) φFB(−gm1(x), λm1)
...
φFB(−gmp(x), λmp)
⎞
⎟⎟
⎟⎟
⎟⎠
. (11)
Then, the neural network model (10) could be obtained as well becauseφFB2 is smooth [7]. However, it is observed that the gradient mapping∇ has more com- plicated formulas because (−gmi(x))2+ λ2mi may lie on the boundary of SOC, or interior of SOC, see [7,33], which will cost more expensive numerical computa- tions. Thus, the one dimensional parameter ε in use not only has no influence on the main result, but also will simplify the computational work.
To discuss properties of the neural network model (10), we make the following assumption which is used to avoid the singularity of∇S(w), see [37].
Assumption 3.1
(a) the gradients{∇hj(x)|j = 1, . . . , l} ∪ {∇gi(x)|i = 1, . . . , m} are linear indepen- dent.
(b) ∇xL(x, μ, λ)is positive definite on the null space of the gradients{∇hj(x)|j = 1, . . . , l}.
When SOCCVI problem corresponds to the KKT conditions of a convex second- order cone program (CSOCP) problem as (4) where both h and g are linear, the above
Assumption3.1(b) is indeed equivalent to the well-used condition∇2f (x)is positive definite, e.g. [42, Corollary 1].
Proposition 3.1 Let : R1+n+l+m→ R+be defined as in (9). Then, (w)≥ 0 for w= (ε, x, μ, λ) ∈ R1+n+l+m and (w)= 0 if and only if (x, μ, λ) solves the KKT system (5).
Proof The proof is straightforward.
Proposition 3.2 Let : R1+n+l+m→ R+be defined as in (9). Then, the following results hold.
(a) The function is continuously differentiable everywhere with
∇ (w) = ∇S(w)S(w), where
∇S(w) =
⎡
⎢⎢
⎣
1 0 0 diag{∇εφε
FB(−gmi(x), λmi)}pi=1 0 ∇xL(x, μ, λ)T −∇h(x) −∇g(x)diag{∇gmiφε
FB(−gmi(x), λmi)}pi=1
0 ∇h(x)T 0 0
0 ∇g(x)T 0 diag{∇λmiφε
FB(−gmi(x), λmi)}pi=1
⎤
⎥⎥
⎦ .
(b) Suppose that Assumption3.1holds. Then, ∇S(w) is nonsingular for any w ∈ R1+n+l+m. Moreover, if (0, x, μ, λ)∈ R1+n+l+mis a stationary point of , then (x, μ, λ)∈ Rn+l+mis a KKT triple of the SOCCVI problem.
(c) (w(t)) is nonincreasing with respect to t .
Proof Part(a) follows from the chain rule. For part(b), we know that∇S(w) is non- singular if and only if the following matrix
⎡
⎣∇xL(x, μ, λ)T −∇h(x) −∇g(x)diag{∇gmiφFBε (−gmi(x), λmi)}pi=1
∇h(x)T 0 0
∇g(x)T 0 diag{∇λmiφFBε (−gmi(x), λmi)}pi=1
⎤
⎦
is nonsingular. In fact, from [37, Theorem 3.1] and [37, Proposition 4.1], the above matrix is nonsingular and (x, μ, λ)∈ Rn+l+m is a KKT triple of the SOCCVI prob- lem if (0, x, μ, λ)∈ R1+n+l+mis a stationary point of . It remains to show part(c).
By the definition of (w) and (10), it is not difficult to compute d (w(t ))
dt = ∇ (w(t))Tdw(t )
dt = −ρ∇ (w(t))2≤ 0. (12) Therefore, (w(t)) is a monotonically decreasing function with respect to t . Now, we are ready to analyze the behavior of the solution trajectory of (10) and establish properties of three kinds of stability for an isolated equilibrium point.
Proposition 3.3
(a) If (x, μ, λ)∈ Rn+l+m is a KKT triple of SOCCVI problem, then (0, x, μ, λ)∈ R1+n+l+mis an equilibrium point of (10).
(b) If Assumption3.1holds and (0, x, μ, λ)∈ R1+n+l+m is an equilibrium point of (10), then (x, μ, λ)∈ Rn+l+mis a KKT triple of SOCCVI problem.
Proof (a) From Proposition3.1and (x, μ, λ)∈ Rn+l+mbeing a KKT triple of SOC- CVI problem, it is clear that S(0, x, μ, λ)= 0. Hence, ∇ (0, x, μ, λ) = 0. Besides, by Proposition3.2, we know that if ε = 0, then ∇ (ε, x, μ, λ) = 0. This shows that (0, x, μ, λ) is an equilibrium point of (10).
(b) It follows from (0, x, μ, λ)∈ R1+n+l+mbeing an equilibrium point of (10) that
∇ (0, x, μ, λ) = 0. In other words, (0, x, μ, λ) is the stationary point of . Then, the result is a direct consequence of Proposition3.2(b). Proposition 3.4
(a) For any initial state w0= w(t0), there exists exactly one maximal solution w(t) with t∈ [t0, τ (w0))for the neural network (10).
(b) If the level set L(w0)= {w ∈ R1+n+l+m| (w) ≤ (w0)} is bounded, then τ (w0)= +∞.
Proof (a) Since S is continuous differentiable,∇S is continuous, and therefore, ∇S is bounded on a local compact neighborhood of w. That means∇ (w) is locally Lipschitz continuous. Thus, applying Lemma2.1leads to the desired result.
(b) This proof is similar to the proof of Case(i) in [5, Proposition 4.2]. Remark 3.2 We wish to obtain the result that the level sets
L( , γ ) := {w ∈ R1+n+l+m| (w) ≤ γ }
are bounded for all γ ∈ R. However, we are not able to complete the argument. We suspect that there needs more subtle properties of F , h and g to finish it.
Next, we investigate the convergence of the solution trajectory of neural net- work (10).
Theorem 3.1
(a) Let w(t) with t∈ [t0, τ (w0))be the unique maximal solution to (10). If τ (w0)= +∞ and {w(t)} is bounded, then limt→∞∇ (w(t)) = 0.
(b) If Assumption3.1holds and (ε, x, μ, λ)∈ R1+n+l+mis the accumulation point of the trajectory w(t ), then (x, μ, λ)∈ Rn+l+mis a KKT triple of SOCCVI problem.
Proof With Proposition3.2(b) and (c) and Proposition3.3, the arguments are exactly the same as those for [28, Corollary 4.3]. Thus, we omit them. Theorem 3.2 Let w∗ be an isolated equilibrium point of the neural network (10).
Then the following results hold.
(a) w∗is asymptotically stable.
(b) If Assumption3.1holds, then it is exponentially stable.
Proof Since w∗is an isolated equilibrium point of (10), there exists a neighborhood
∗⊆ R1+n+l+mof w∗such that
∇ (w∗)= 0 and ∇ (w) = 0 ∀w ∈ ∗\ {w∗}.
Next, we argue that (w) is indeed a Lyapunov function at x∗ over the set ∗for (10) by showing that the conditions in (7) are satisfied. First, notice that (w)≥ 0.
Suppose that there is an ¯w ∈ ∗\ {w∗} such that ( ¯w) = 0. Then, we can easily obtain that∇ ( ¯w) = 0, i.e., ¯w is also an equilibrium point of (10), which clearly contradicts the assumption that w∗is an isolated equilibrium point in ∗. Thus, we prove that (w) > 0 for any w∈ ∗\ {w∗}. This together with (12) shows that the condition in (7) are satisfied. Because w∗is isolated, from (12), we have
d (w(t ))
dt <0, ∀w(t) ∈ ∗\ {w∗}.
This implies that w∗is asymptotically stable. Furthermore, if Assumption3.1holds, we can obtain that∇S is nonsingular. In addition, we have
S(w)= S(w∗)+ ∇S(w∗)(w− w∗)+ o(w − w∗), ∀w ∈ ∗\ {w∗}. (13) From S(w(t)) being a monotonically decreasing function with respect to t and (13), we can deduce that
w(t) − w∗ ≤ (∇S(w∗))−1S(w(t)) − S(w∗) + o(w(t) − w∗)
≤ (∇S(w∗))−1S(w(t0))− S(w∗) + o(w(t) − w∗)
≤ (∇S(w∗))−1
(∇S(w∗))w(t0)− w∗ + o(w(t0)− w∗) + o(w(t) − w∗).
That is,
w(t) − w∗ − o(w(t) − w∗) ≤ (∇S(w∗))−1
(∇S(w∗))w(t0)− w∗ + o(w(t0)− w∗)
.
The above inequality implies that the neural network (10) is also exponentially sta-
ble.
4 Neural network model based on projection function
In this section, we present that the KKT triple of SOCCVI problem is equivalent to the solution of a projection formulation. Based on this, we introduce another neural network model for solving the projection formulation and analyze the stability con- ditions and convergence.
Define the function U: Rn+l+m→ Rn+l+mand vector w in the following form:
U (w)=
⎛
⎝L(x, μ, λ)
−h(x)
−g(x)
⎞
⎠ , w =
⎛
⎝x μ λ
⎞
⎠ , (14)
where L(x, μ, λ)= F (x) + ∇h(x)μ + ∇g(x)λ is the Lagrange function. To avoid confusion, we emphasize that, for any w∈ Rn+l+m, we have
wi ∈ R, if 1 ≤ i ≤ n + l,
wi ∈ Rmi−(n+l), if n+ l + 1 ≤ i ≤ n + l + p.
Then, we may write (14) as
Ui= (U(w))i= (L(x, μ, λ))i, wi= xi, i= 1, . . . , n, Un+j= (U(w))n+j= −hj(x), wn+j= μj, j= 1, . . . , l,
Un+l+k= (U(w))n+l+k= −gk(x)∈ Rmk, wn+l+k= λk∈ Rmk, k= 1, . . . , p,
p k=1
mk= m.
With this, the KKT conditions (5) can be recast as Ui= 0, i = 1, 2, . . . , n, n + 1, . . . , n + l,
UJ, wJ = 0, UJ= (Un+l+1, Un+l+2, . . . , Un+l+p)T ∈ K, (15) wJ= (wn+l+1, wn+l+2, . . . , wn+l+p)T ∈ K.
Thus, (x∗, μ∗, λ∗) is a KKT triple for (1) if and only if (x∗, μ∗, λ∗)is a solution to (15).
It is well known that the nonlinear complementarity problem, which is denoted by NCP(F, K) and to find an x∈ Rnsuch that
x∈ K, F (x) ∈ K and F (x), x = 0
where K is a closed convex set ofRn, is equivalent to the following VI(F, K) prob- lem: finding an x∈ K such that
F (x), y − x ≥ 0 ∀y ∈ K.
Furthermore, if K= Rn, then NCP(F, K) becomes the system of nonlinear equations F (x)= 0.
Based on the above, solution of (15) is equivalent to solution of the following VI problem: find w∈ K such that
U(w), v − w ≥ 0, ∀v ∈ K, (16)
whereK = Rn+l× K. In addition, by applying the Property2.1, its solution is equiv- alent to solution of below projection formulation
PK(w− U(w)) = w with K = Rn+l× K, (17) where function U and vector w are defined in (14). Now, according to (17), we give the following neural network:
dw
dt = ρ{PK(w− U(w)) − w}, (18)
where ρ > 0. Note thatK is a closed and convex set. For any w ∈ Rn+l+m, PKmeans PK(w)= [PK(w1), PK(w2), . . . , PK(wn+l), PK(wn+l+1), PK(wn+l+2), . . . ,
PK(wn+l+p)], where
PK(wi)= wi, i= 1, . . . , n + l,
PK(wn+l+j)= [λ1(wn+l+j)]+· u(1)wn+l+j + [λ2(wn+l+j)]+· u(2)wn+l+j, j= 1, . . . , p.
Here, for the sake of simplicity, we denote the vector wn+l+j by v for the moment, and[·]+ is the scalar projection, λ1(v), λ2(v)and u(1)v , u(2)v are the spectral values and the associated spectral vectors of v= (v1; v2)∈ R × Rmj−1, respectively, given
by ⎧
⎨
⎩
λi(v)= v1+ (−1)iv2, u(i)v =12(1, (−1)i vv22), for i= 1, 2, see [7,33].
The dynamic system described by (18) can be recognized as a recurrent neural network with a single-layer structure. To analyze the stability conditions of (18), we need the following lemmas and proposition.
Lemma 4.1 If the gradient of L(x, μ, λ) is positive semi-definite (respectively, pos- itive definite), then the gradient of U in (14) is positive semi-definite (respectively, positive definite).
Proof Since we have
∇U(x, μ, λ) =
⎡
⎣∇xLT(x, μ, λ) −∇h(x) −∇g(x)
∇Th(x) 0 0
∇Tg(x) 0 0
⎤
⎦ ,
for any nonzero vector d= (pT, qT, rT)T ∈ Rn+l+m, we can obtain that
dT∇U(x, μ, λ)d =
pT qT rT⎡
⎣∇xLT(x, μ, λ) −∇h(x) −∇g(x)
∇Th(x) 0 0
∇Tg(x) 0 0
⎤
⎦
⎛
⎝p q r
⎞
⎠
= pT∇xL(x, μ, λ)p.
This leads to the desired results.
Proposition 4.1 For any initial point w0= (x0, μ0, λ0)with λ0:= λ(t0)∈ K, there exist a unique solution w(t )= (x(t), μ(t), λ(t)) for neural network (18), Moreover, λ(t )∈ K.
Proof For simplicity, we assumeK = Km. The analysis can be carried over to the general case whereK is the Cartesian product of second-order cones. Since F, h, g are continuous differentiable, the function
F (w):= PK(w− U(w)) − w with K = Rn+l× Km (19) is semi-smooth and Lipschitz continuous. Thus, there exists a unique solution w(t )= (x(t), μ(t), λ(t)) for neural network (18). Therefore, it remains to show that λ(t )∈ Km. For convenience, we denote λ(t):= (λ1(t ), λ2(t ))∈ R × Rm−1. To com- plete the proof, we need to verify two things: (i) λ1(t )≥ 0 and (ii) λ2(t ) ≤ λ1(t ).
First, from (18), we have dλ
dt + ρλ(t) = ρPKm(λ+ g(x)).
The solution of the above first-order ordinary differential equation is λ(t )= e−ρ(t−t0)λ(t0)+ ρe−ρt
t t0
ρeρsPKm(λ+ g(x))ds. (20)
If we let λ(t0):= (λ1(t0), λ2(t0))∈ R × Rm−1and denote PKm(λ+ g(x)) as z(t0):=
(z1(t0), z2(t0)), then (20) leads to
λ1(t )= e−ρ(t−t0)λ1(t0)+ ρe−ρt
t
t0
ρeρsz1(s)ds, (21)
λ2(t )= e−ρ(t−t0)λ2(t0)+ ρe−ρt
t
t0
ρeρsz2(s)ds. (22) Due to both λ(t0)and z(t) belong to Km, there have λ1(t0)≥ 0, λ2(t0) ≤ λ1(t0) andz2(t ) ≤ z1(t ). Therefore, λ1(t )≥ 0 since both terms in the right-hand side of (21) are nonnegative. In addition,
λ2(t ) ≤ e−ρ(t−t0)λ2(t0) + ρe−ρt
t
t0
ρeρsz2(s)ds
≤ e−ρ(t−t0)λ1(t0)+ ρe−ρt
t
t0
ρeρsz1(s)ds
= λ1(t ),
which implies that λ(t)∈ Km.
Lemma 4.2 Let U (w), F (w) be defined as in (14) and (19), respectively. Suppose w∗= (x∗, μ∗, λ∗)is an equilibrium point of neural network (18) with (x∗, μ∗, λ∗) being an KKT triple of SOCCVI problem. Then, the following inequality holds:
(F (w)+ w − w∗)T(−F (w) − U(w)) ≥ 0. (23) Proof Notice that
(F (w)+ w − w∗)T(−F (w) − U(w))
= [−w + PK(w− U(w)) + w − w∗]T[w − PK(w− U(w)) − U(w)]
= [−w∗+ PK(w− U(w))]T[w − PK(w− U(w)) − U(w)]
= −[w∗− PK(w− U(w))]T[w − U(w) − PK(w− U(w))].
Since w∗∈ K, applying Property2.1gives
[w∗− PK(w− U(w))]T[w − U(w) − PK(w− U(w))] ≤ 0.
Thus, we have
(F (w)+ w − w∗)T(−F (w) − U(w)) ≥ 0.
This completes the proof.
We now show the stability and convergence issues regarding neural network (18).
Theorem 4.1 If∇xL(w)is positive semi-definite (respectively, positive definite), the solution of neural network (18) with initial point w0= (x0, μ0, λ0)where λ0∈ K is Lyapunov stable (respectively, asymptotically stable). Moreover, the solution trajec- tory of neural network (18) is extendable to the global existence.
Proof Again, for simplicity, we assumeK = Km. From Proposition4.1, there exists a unique solution w(t)= (x(t), μ(t), λ(t)) for neural network (18) and λ(t)∈ Km. Let w∗= (x∗, μ∗, λ∗)be an equilibrium point of neural network (18). We define a Lyapunov function as below:
V (w):= V (x, μ, λ) := −U(w)TF (w)−1
2F (w)2+1
2w − w∗2. (24) From [12, Theorem 3.2], we know that V is continuously differentiable with
∇V (w) = U(w) − [∇U(w) − I]F (w) + (w − w∗).
It is also trivial that V (w∗)= 0. Then, we have
dV (w(t ))
dt = ∇V (w(t))Tdw dt
= {U(w) − [∇U(w) − I]F (w) + (w − w∗)}TρF (w)
= ρ{[U(w) + (w − w∗)]TF (w)+ F (w)2− F (w)T∇U(w)F (w)}.
Inequality (23) in Lemma4.2implies
[U(w) + (w − w∗)]TF (w)≤ −U(w)T(w− w∗)− F (w)2, which yields
dV (w(t )) dt
≤ ρ{−U(w)T(w− w∗)− F (w)T∇U(w)F (w)}
= ρ{−U(w∗)T(w− w∗)− (U(w) − U(w∗))T(w− w∗)
− F (w)T∇U(w)F (w)}. (25)
Note that w∗ is the solution of the variational inequality (16). Since w∈ K, we therefore obtain −U(w∗)T(w− w∗)≤ 0. Because U(w) is continuous differen- tiable and ∇U(w) is positive semi-definite, by [32, Theorem 5.4.3], we obtain that U (w) is monotone. Hence, we have−(U(w) − U(w∗))T(w− w∗)≤ 0 and
−F (w)T∇U(w)F (w) ≤ 0. The above discussions lead todV (w(t ))dt ≤ 0. Also, by [32, Theorem 5.4.3], we know that if∇U(w) is positive definite, then U(w) is strictly monotone, which impliesdV (w(t ))dt <0 in this case.
In order to obtain V (w) is a Lyapunov function and w∗ is Lyapunov stable, we will show the following inequality:
−U(w)TF (w)≥ F (w)2. (26) To see this, we first observe that
F (w)2+ U(w)TF (w)= [w − PK(w− U(w))]T[w − U(w) − PK(w− U(w))].
Since w∈ K, applying Property2.1again, there holds
[w − PK(w− U(w))]T[w − U(w) − PK(w− U(w))] ≤ 0, which yields the desired inequality (26). By combining (24) and (26), we have
V (w)≥1
2F (w)2+1
2w − w∗2,
which says V (w) > 0 if w = w∗. Hence V (w) is indeed a Lyapunov function and w∗ is Lyapunov stable. Furthermore, if∇xL(w)is positive definite, we have w∗ is asymptotically stable. Moreover, it holds that
V (w0)≥ V (w) ≥1
2w − w∗2 for t≥ t0, (27)
which tells us the solution trajectory w(t) is bounded. Hence, it can be extended to
global existence.
Theorem 4.2 Let w∗= (x∗, μ∗, λ∗) be an equilibrium point of (18). If ∇xL(w) is positive definite, the solution of neural network (18) with initial point w0= (x0, μ0, λ0)where λ0∈ K is globally convergent to w∗ and has finite convergence time.
Proof From (27), the level set
L(w0):= {w|V (w) ≤ V (w0)}
is bounded. Then, the Invariant Set Theorem [15] implies the solution trajectory w(t) converges to θ as t→ +∞ where θ is the largest invariant set in
=
w∈ L(w0)
dV (w(t ))
dt = 0
.
We will show that dw/dt= 0 if and only if dV (w(t))/dt = 0 which yields that w(t) converges globally to the equilibrium point w∗= (x∗, μ∗, λ∗). Suppose dw/dt = 0, then it is clear that dV (w(t))/dt= ∇V (w)T(dw/dt )= 0. Let ˆw = ( ˆx, ˆμ, ˆλ) ∈ which says dV (ˆw(t))/dt = 0. From (23), we know that
dV (ˆw(t))/dt ≤ ρ{(−U( ˆw) − U(w∗))T(ˆw − w∗)− F ( ˆw)T∇U( ˆw)F ( ˆw)}.
Both terms inside the big parenthesis are nonpositive as shown in Theorem4.1, so (U (ˆw) − U(w∗))T(ˆw − w∗)= 0, F ( ˆw)T∇U( ˆw)F ( ˆw) = 0. The condition ∇xL(w) being positive definite leads to∇U( ˆw) being positive definite. Hence,
F (ˆw) = − ˆw + PK(ˆw − U( ˆw)) = 0,
which is equivalent to dˆw/dt = 0. From the above, w(t) converges globally to the equilibrium point w∗= (x∗, μ∗, λ∗). Moreover, with Theorem4.1and following the same argument as in [42, Theorem 2], the neural network (18) has finite convergence
time.
5 Simulations
To demonstrate effectiveness of the proposed neural networks, some illustrative SOC- CVI problems are tested. The numerical implementation is coded by Matlab 7.0 and the ordinary differential equation solver adopted is ode23, which uses Runge-Kutta (2), (3) formula. In the following tests, the parameter ρ in both neural networks is set to be 1000.
Example 5.1 Consider the SOCCVI problem (1)–(2) where
F (x)=
⎛
⎜⎜
⎜⎜
⎜⎜
⎜⎜
⎜⎜
⎜⎜
⎜⎜
⎝
2x1+ x2+ 1 x1+ 6x2− x3− 2
−x2+ 3x3−65x4+ 3
−65x3+ 2x4+12sin x4cos x5sin x6+ 6
1
2cos x4sin x5sin x6+ 2x5−52
−12cos x4cos x5cos x6+ 2x6+14cos x6sin x7cos x8+ 1
1
4sin x6cos x7cos x8+ 4x7− 2
−14sin x6sin x7sin x8+ 2x8+12
⎞
⎟⎟
⎟⎟
⎟⎟
⎟⎟
⎟⎟
⎟⎟
⎟⎟
⎠
and
C= {x ∈ R8: −g(x) = x ∈ K3× K3× K2}.
This problem has an approximate solution
x∗= (0.3820, 0.1148, −0.3644, 0.0000, 0.0000, 0.0000, 0.5000, −0.2500)T. It can be verified that the Lagrangian function for this example is
L(x, μ, λ)= F (x) − λ and the gradient of the Lagrangian function is
∇L(x, μ, λ) =
∇F (x) I8×8
,
where I is the identity mapping and∇F (x) means the gradient of F (x). We use the proposed neural networks with smoothed FB and projection functions, respectively, to solve the problem whose trajectories are depicted in Figs.1and2. The simulation results show that both trajectories are globally convergent to x∗ and the neural net- work with projection function converges to x∗quicker than that with smoothed FB function.
Example 5.2 [37, Example 5.1] We consider the following SOCCVI problem:
1
2Dx, y− x
!
≥ 0, ∀y ∈ C
where
C= {x ∈ Rn: Ax − a = 0, Bx − b 0},
Dis an n× n symmetric matrix, A and B are l × n and m × n matrices, respectively, dis an n× 1 vector, a and b are l × 1 and m × 1 vectors with l + m ≤ n, respectively.
Fig. 1 Transient behavior of neural network with smoothed FB function in Example5.1
Fig. 2 Transient behavior of the neural network with projection function in Example5.1
In fact, we can determine the data a, b, A, B and D randomly. However, as in [37, Example 5.1], we set the data as follows:
D= [Dij]n×n, where Dij=
⎧⎨
⎩
2, i= j, 1, |i − j| = 1, 0, otherwise,
A= [Il×l 0l×(n−l)]l×n, B= [0m×(n−m) Im×m]m×n, a= 0l×1, b= (em1, em2, . . . , emp)T, where emi= (1, 0, . . . , 0)T ∈ Rmi and l+ m ≤ n. Clearly, A and B are full row rank and rank([AT BT]) = l + m.
In the simulation, the parameters l, m, and n are set to be 3, 3, and 6, respec- tively. The problem has an solution x∗= (0, 0, 0, 0, 0, 0)T. It can be verified that the Lagrangian function for this example is
L(x, μ, λ)=1
2Dx+ ATμ+ BTλ.
Note that∇xL(x, μ, λ)is positive definite. We know from Theorems3.1and4.2that both proposed neural networks are globally convergent to the KKT triple of the SOC- CVI problem. Figures3and4depict the trajectories of Example5.2obtained using the proposed neural networks. The simulation results show that both neural networks are effective in the SOCCVI problem and the neural network with projection function converges to x∗quicker than that with smoothed FB function.
Example 5.3 Consider the SOCCVI problem (1)–(2) where
F (x)=
⎛
⎜⎜
⎜⎜
⎜⎜
⎝
x3exp(x1x3)+ 6(x1+ x2) 6(x1+ x2)+√2(2x2−x3)
1+(2x2−x3)2
x1exp(x1x3)−√ 2x2−x3
1+(2x2−x3)2
x4
x5
⎞
⎟⎟
⎟⎟
⎟⎟
⎠
and
C= {x ∈ R5: h(x) = 0, −g(x) ∈ K3× K2}, with
h(x)= −62x13+ 58x2+ 167x33− 29x3− x4− 3x5+ 11,
g(x)=
⎛
⎜⎜
⎜⎜
⎝
−3x13− 2x2+ x3− 5x33 5x13− 4x2+ 2x3− 10x33
−x3
−x4
−3x5
⎞
⎟⎟
⎟⎟
⎠.
This problem has an approximate solution x∗= (0.6287, 0.0039, −0.2717, 0.1761, 0.0587)T.
Fig. 3 Transient behavior of neural network with smoothed FB function in Example5.2
Fig. 4 Transient behavior of neural network with projection function in Example5.2
Example 5.4 Consider the SOCCVI problem (1)–(2) where
F (x)=
⎛
⎜⎜
⎝
4x1− sin x1cos x2+ 1
− cos x1sin x2+ 6x2+95x3+ 2
9
5x2+ 8x3+ 3 2x4+ 1
⎞
⎟⎟
⎠
and
C=
x∈ R4: h(x) =
"
x12−101x2x3+ x3
x32+ x4
#
= 0, −g(x) =
$x1
x2
%
∈ K2
&
.
This problem has an approximate solution x∗ = (0.2391, −0.2391, −0.0558,
−0.0031)T.
The neural network (10) based on smoothed FB function can solve Examples5.3–
5.4successfully, see Figs.5,6, whereas the neural network (18) based on projection function fails to solve them. This is because that∇xL(x, μ, λ)is not always positive definite in Examples5.3and5.4. Hence, the neural network with projection function is not effective in these two problems. To the contrast, though there is no guarantee that the Assumption 3.1(b) holds, the neural network with smoothed FB function is asymptotically stable from Theorem3.2. Figures5 and6 depict the trajectories obtained using the neural network with the smoothed FB function for Examples5.3 and5.4, respectively. The simulation results show that each trajectory converges to the desired isolated equilibrium point which is exactly the approximate solution of Examples5.3and5.4, respectively.
Example 5.5 Consider the nonlinear convex SOCP [23] given by min exp(x1− x3)+ 3(2x1− x2)4+'
1+ (3x2+ 5x3)2
s.t. −g(x) =
⎛
⎜⎜
⎜⎜
⎝
4x1+ 6x2+ 3x3− 1
−x1+ 7x2− 5x3+ 2 x1
x2
x3
⎞
⎟⎟
⎟⎟
⎠∈ K2× K3.
The approximate solution of this problem is x∗= (0.2324, −0.07309, 0.2206)T. As mentioned in Sect.1, the CSOCP in Example5.5can be transformed into an equiv- alent SOCCVI problem. There are neural network models proposed for CSOCP in [27]. We here try a different approach for it. In other words, we use the proposed neural networks with the smoothed FB and projection functions, respectively, to solve the problem whose trajectories are depicted in Figs.7 and8. From the simulation results, we see that the neural network with smoothed FB function converges very slowly and it is not clear whether it converges to the solution in finite time or not. In view of these, to solve CSOCP, it seems better to apply the models introduced in [27]
directly instead of transforming CSOCP into SOCCVI problem.
Fig. 5 Transient behavior of neural network with smoothed FB function in Example5.3
Fig. 6 Transient behavior of neural network with smoothed FB function in Example5.4
Fig. 7 Transient behavior of neural network with smoothed FB function in Example5.5
Fig. 8 Transient behavior of neural network with projection function in Example5.5
