Information Sciences

A smoothed NR neural network for solving nonlinear convex programs with second-order cone constraints

Xinhe Miao

^a,1

, Jein-Shan Chen

^b,⇑,2

, Chun-Hsu Ko

^c

aDepartment of Mathematics, School of Science, Tianjin University, Tianjin 300072, PR China

bDepartment of Mathematics, National Taiwan Normal University, Taipei 11677, Taiwan

cDepartment of Electrical Engineering, I-Shou University, Kaohsiung 84001, Taiwan

a r t i c l e i n f o

Article history:

Received 2 January 2012

Received in revised form 22 June 2013 Accepted 16 October 2013

Available online 24 October 2013

Keywords:

Merit function Neural network NR function Second-order cone Stability

a b s t r a c t

This paper proposes a neural network approach for efficiently solving general nonlinear convex programs with second-order cone constraints. The proposed neural network model was developed based on a smoothed natural residual merit function involving an uncon- strained minimization reformulation of the complementarity problem. We study the exis- tence and convergence of the trajectory of the neural network. Moreover, we show some stability properties for the considered neural network, such as the Lyapunov stability, asymptotic stability, and exponential stability. The examples in this paper provide a further demonstration of the effectiveness of the proposed neural network. This paper can be viewed as a follow-up version of[20,26]because more stability results are obtained.

Ó 2013 Elsevier Inc. All rights reserved.

1. Introduction

In this paper, we are interested in finding a solution to the following nonlinear convex programs with second-order cone constraints (henceforth SOCP):

min f ðxÞ s:t: Ax ¼ b

gðxÞ 2K

ð1Þ

where A 2 R^mnhas full row rank, b 2 R^m, f : Rⁿ! R; g ¼ ½g₁; . . . ;g_l^T:Rⁿ! R^lwith f and gi’s being two order continuous differentiable and convex on Rⁿ, andK is a Cartesian product of second-order cones (also called Lorentz cones), expressed as

K ¼ Kⁿ¹Kⁿ²    K^nN

with N, n1, . . . , nNP1, n1+    + nN= l and

Kⁿⁱ :¼ fðxi1;xi2; . . . ;xin_iÞ^T2 Rⁿⁱj kðxi2; . . . ;xin_iÞk 6 xi1g:

0020-0255/$ - see front matter Ó 2013 Elsevier Inc. All rights reserved.

http://dx.doi.org/10.1016/j.ins.2013.10.017

⇑Corresponding author. Address: National Center Theoretical Sciences, Taipei Office, Taiwan. Tel.: +886 2 77346641; fax: +886 2 29332342.

E-mail addresses:[email protected](X. Miao),[email protected](J.-S. Chen),[email protected](C.-H. Ko).

1The author’s work is also supported by National Young Natural Science Foundation (No. 11101302) and The Seed Foundation of Tianjin University (No.

60302041).

2The author’s work is supported by National Science Council of Taiwan.

Contents lists available atScienceDirect

Information Sciences

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / i n s

Here, k  k denotes the Euclidean norm andK¹means the set of nonnegative reals Rþ. In fact, the problem(1)is equivalent to the following variational inequality problem, which is to find x 2 D satisfying

hrf ðxÞ; y  xi P 0; 8y 2 D;

where D ¼ fx 2 RⁿjAx ¼ b; gðxÞ 2Kg. Many problems in the engineering, transportation science, and economics commu- nities can be solved by transforming the original problems into the mentioned convex optimization problems or variational inequality problems, see[1,7,10,17,23].

Many studies have proposed computational approaches to solve convex optimization problems. Examples of these meth- ods include the interior-point method[29], merit function method[5,16], Newton method[18,25], and projection method [10]. However, real-time solutions are imperative in many applications, such as force analysis in robot grasping and control applications. The traditional optimization methods may not be suitable for these applications because of stringent compu- tational time requirements. Therefore, a feasible and efficient method is required to solve real-time optimization problems.

The neural network method is an ideal method for solving real-time optimization problems. Compared with previous meth- ods, the neural network method has an advantage in solving real-time optimization problems. Hence, researchers have developed many continuous-time neural networks for constrained optimization problems. The literature contains many studies on neural networks for solving real-time optimization problems, please see[4,9,12,14,15,19–22,27,30–34,36]and references therein.

Neural networks stemmed back from McCulloch and Pitts’ pioneering work a half century ago, and neural networks were first introduced to the optimization domain in the 1980s[13,28]. The essence of neural network method for optimization[6]

is to establish a nonnegative Lyapunov function (or energy function) and a dynamic system that represents an artificial neu- ral network. This dynamic system usually adopts the form of a first-order ordinary differential equation. For an initial point, the neural network is likely to approach its equilibrium point, which corresponds to the solution to the considered optimi- zation problem.

This paper presents a neural network method to solve general nonlinear convex programs with second-order cone con- straints. In particular, we consider the Karush–Kuhn–Tucker (KKT) optimality conditions of the problem(1), which can be transformed into a second-order cone complementarity problems (SOCCP), as well as some equality constraints. Following a reformulation of the complementarity problem, an unconstrained optimization problem is formulated. A smoothed natural residual (NR) complementarity function is then used to construct a Lyapunov function and a neural network model. At the same time, we show the existence and convergence of the solution trajectory for the dynamic system. This study also inves- tigates the stability results, such as the Lyapunov stability, the asymptotic stability, and the exponential stability. We want to point out that the optimization problem considered in this paper is more general than the one studied in[20]where g(x) = x is investigated therein. From[20], for solving the specific SOCP (i.e., g(x) = x), we know that the neural network based on the cone projection function has better performance than the one based on the Fischer–Burmeister function in most cases (ex- cept for some oscillating cases). In light of considering this phenomenon, we employ a neural network model based on the cone projection function for a more general SOCP. Thus, this paper can be viewed as a follow-up of[20]in this sense. Nev- ertheless, the neural network model studied here is not exactly the same as the one considered in[20]. More specifically, we consider a neural network based on the smoothed NR function which was studied in[16]. Why do we make such a change?

As in Section4, we can establish various stability results, including exponential stability for the proposed neural network that were not achieved in[20]. In addition, the second neural network studied in[27](for various types of problems) is also similar to the proposed network. Again, the stability is not guaranteed in that study, but three stabilities are proved here.

The remainder of this paper is organized as follows. Section2presents stability concepts and provides related results.

Section3describes the neural network architecture, which is based on the smoothed NR function, to solve the problem (1). Section4presents the convergence and stability results of the proposed neural network. Section5shows the simulation results of the new method. Finally, Section6gives the conclusion of this paper.

2. Preliminaries

In this section, we briefly recall background materials of the ordinary differential equation (ODE) and some stability con- cepts regarding the solution of ODE. We also present some related results that play an essential role in the subsequent analysis.

Let H : Rⁿ! Rⁿbe a mapping. The first order differential equation (ODE) means du

dt¼ HðuðtÞÞ; uðt0Þ ¼ u02 Rⁿ: ð2Þ

We start with the existence and uniqueness of the solution of Eq.(2). Then, we introduce the equilibrium point of(2)and define various stabilities. All of these materials can be found in a typical ODE textbook, such as[24].

Lemma 2.1 (The existence and uniqueness 21, Theorem 2.5). Assume that H : Rⁿ! Rⁿ is a continuous mapping. Then for arbitrary t0P0 and u02 Rⁿ, there exists a local solution u(t), t 2 [t0,

s

) to(2)for some

s

> t0. Furthermore, if H is locally Lipschitz continuous at u0, then the solution is unique; if H is Lipschitz continuous in Rⁿ, then

s

can be extended to 1.

Remark 2.1. For Eq.(2), if a local solution defined on [t₀,

s

) cannot be extended to a local solution on a larger interval [t₀,

s

1), where

s

1>

s

, then it is called a maximal solution, and this interval [t0,

s

) is the maximal interval of existence. It is obvious that an arbitrary local solution has an extension to a maximal one.

Lemma 2.2 (21, Theorem 2.6). Let H : Rⁿ! Rⁿbe a continuous mapping. If u(t) is a maximal solution, and [t0,

s

) is the maximal interval of existence associated with u0and

s

< +1, then limt"sku(t)k = +1.

For the first-order differential Eq.(2), a point u^2 Rⁿis called an equilibrium point of(2)if H(u^⁄) = 0. If there is a neigh- borhoodX#Rⁿof u^⁄such that H(u^⁄) = 0 and H(u) – 0 for any u 2Xn{u^⁄}, then u^⁄is called an isolated equilibrium point. The following are definitions of various stabilities, and related materials can be found in[21,24,27].

Definition 2.1 (Lyapunov stability and Asymptotic stability). Let u(t) be a solution to Eq.(2).

(a) An isolated equilibrium point u^⁄is Lyapunov stable (or stable in the sense of Lyapunov) if for any u0= u(t0) and

e

> 0, there exists a d > 0 such that

ku0 u^k < d ) kuðtÞ  u^k <

e

for t P t0:

(b) Under the condition that an isolated equilibrium point u^⁄is Lyapunov stable, u^⁄is said to be asymptotically stable if it has the property that if ku0 u^⁄k < d, then u(t) ? u^⁄as t ? 1.

Definition 2.2 (Lyapunov function). Let X#Rⁿ be an open neighborhood of u. A continuously differentiable function g : Rⁿ! R is said to be a Lyapunov function (or energy function) at the state u (over the setX) for Eq.(2)if

gðuÞ ¼ 0;

gðuÞ > 08u 2Xn fug;

dgðuðtÞÞ

dt 60; 8u 2X: 8>

<

>:

The following Lemma shows the relationship between stabilities and a Lyapunov function, see[3,8,35].

Lemma 2.3.

(a) An isolated equilibrium point u^⁄is Lyapunov stable if there exists a Lyapunov function over some neighborhoodXof u^⁄. (b) An isolated equilibrium point u^⁄is asymptotically stable if there exists a Lyapunov function over some neighborhoodXof u^⁄

that satisfies dgðuðtÞÞ

dt <0; 8u 2Xn fu^g:

Definition 2.3 (Exponential stability). An isolated equilibrium point u^⁄is exponentially stable for Eq.(2)if there existx< 0,

j

> 0, d > 0 such that arbitrary solution u(t) to Eq.(2), with the initial condition u(t0) = u0, ku0 u^⁄k < d, is defined on [0, 1) and satisfies

kuðtÞ  u^k 6

j

e^xtkuðt0Þ  u^k; t P t0:

From the above definitions, it is obvious that exponential stability is asymptotic stable.

3. NR neural network model

This section shows how the dynamic system in this study was formed. As mentioned previously, the key steps in the neu- ral network method lie in constructing the dynamic system and Lyapunov function. To this end, we first look into the KKT conditions of the problem(1)which are presented as below:

rf ðxÞ  A^Ty þrgðxÞz ¼ 0;

z 2K; gðxÞ 2 K; z^TgðxÞ ¼ 0;

Ax  b ¼ 0;

8>

<

>: ð3Þ

where y 2 R^m,rg(x) denotes the gradient matrix of g. According to the KKT condition, it is well known that if the problem(1) satisfies Slater’s condition, which means there exists a strictly feasible point for(1), i.e., there exists an x 2 Rⁿsuch that

gðxÞ 2 intðKÞ and Ax = b. Then x^⁄is a solution of the problem(1)if and only if there exist y^⁄, z^⁄such that (x^⁄, y^⁄, z^⁄) satisfies the KKT conditions(3). Hence, we assume that the problem(1)satisfies the Slater’s condition in this paper.

The following paragraphs provide a brief review of particular properties of the spectral factorization with respect to a sec- ond-order cone, which will be used in the subsequent analysis. Spectral factorization is one of the basic concepts in Jordan algebra. For more details, see[5,11,25].

For any vector z ¼ ðz1;z2Þ 2 R  R^l1ðl P 2Þ, its spectral factorization with respect to the second-order coneK is defined as z ¼ k1e1þ k2e2;

where ki= z1+ (1)ⁱkz2k, (i = 1, 2) are the spectral values of z, and

ei¼

1

2ð1; ð1Þ^{i z}_kzⁱ

ikÞ; z2–0

1

2ð1; ð1ÞⁱwÞ; z2¼ 0 (

with w 2 R^l1such that kwk = 1. The terms e1, e2are called the spectral vectors of z. The spectral values of z and the vector z have the following properties: for any z 2 R^l, there have k16k2and

k1P0 () z 2K:

Now we review the concept of metric projection ontoK. For arbitrary element z 2 R^l, the metric projection of z ontoK is de- noted by P_KðzÞ and defined as

P_KðzÞ :¼ arg min

w2Kkz  wk:

Combining the spectral decomposition of z with the metric projection of z ontoK yields the expression of metric projection P_KðzÞ in[11]:

P_KðzÞ ¼ maxf0; k1ge1þ maxf0; k2ge2:

The projection function P_Khas the following property, which is called the Projection Theorem (see[2]).

Lemma 3.1. LetXbe a closed convex set of Rⁿ. Then, for all x; y 2 Rⁿand any z 2X, ðx  PXðxÞÞ^TðPXðxÞ  zÞ P 0 and kPXðxÞ  PXðyÞk 6 kx  yk:

Given the definition of the projection, suppose z₊denotes the metric projection P_KðzÞ of z 2 R^lontoK. Then, the natural residual (NR) function is given as follows[11]:

UNRðx; yÞ :¼ x  ðx  yÞ_þ 8x; y 2 R^l:

The NR function is a popular SOC-complementarity function, i.e., UNRðx; yÞ ¼ 0 () x 2K; y 2 K and hx; yi ¼ 0:

Because of the non-differentiability ofUNR, we consider a class of smoothed NR complementarity function. To this end, we employ a continuously differentiable convex function ^g : R ! R such that

a!1limgðaÞ ¼ 0;^ lim

a!1ð^gðaÞ  aÞ ¼ 0 and 0 < ^g⁰ðaÞ < 1: ð4Þ

What kind of functions satisfies the condition(4)? Here we present two examples:

^gðaÞ ¼ ffiffiffiffiffiffiffiffiffiffiffiffiffiffi a²þ 4

p þ a

2 and ^gðaÞ ¼ lnðe^aþ 1Þ:

Suppose z = k1e1+ k2e2, where kiand ei(i = 1, 2) are the spectral values and spectral vectors of z, respectively. By applying the function ^g, we define the following function:

PlðzÞ :¼

l

^{^}g k1

l

 

e1þ

l

^{^}g k2

l

 

e2: ð5Þ

Fukushima et al.[11]show that Plis smooth for any

l

> 0; moreover Plis a smoothing function of the projection P_K, i.e., liml#0Pl¼ P_K. Hence, a smoothed NR complementarity function is given in the form of

Ulðx; yÞ :¼ x  Plðx  yÞ:

In particular, from[11, Proposition 5.1], there exists a positive constant

c

> 0 such that kUlðx; yÞ UNRðx; yÞk 6

cl

for any

l

> 0 and ðx; yÞ 2 Rⁿ Rⁿ.

Now we look into the KKT conditions(3)of the problem(1). Let

Lðx; y; zÞ ¼rf ðxÞ  A^Ty þrgðxÞz; HðuÞ :¼

l

Ax  b Lðx; y; zÞ U_{lðz; gðxÞÞ} 2

66 64

3 77 75

and

WlðuÞ :¼ 1

2kHðuÞk²¼1

2kUlðz; gðxÞÞk²þ1

2kLðx; y; zÞk²þ1

2kAx  bk²þ1 2

l

²;

where u ¼ ð

l

;x^T;y^T;z^TÞ^T2 Rþ Rⁿ R^m R^l. It is known thatWl(u) serves as a smoothing function of the merit function WNRwhich means the KKT conditions(3)are shown to be equivalent to the following unconstrained minimization problem via the merit function approach:

minWlðuÞ :¼ 1

2kHðuÞk²: ð6Þ

Theorem 3.1.

(a) Let Plbe defined by(5). Then,rPl(z) and I rPl(z) are positive definite for any

l

> 0 and z 2 R^l.

(b) Let Wl be defined as in (6). Then, the smoothed merit function Wl is continuously differentiable everywhere with rWl(u) =rH(u)H(u) where

rHðuÞ ¼

1 0 0 ^{@PlðzþgðxÞÞ}_@l T

0 A^T r²f ðxÞ þr²g₁ðxÞ þ    þr²g_lðxÞ rxPlðz þ gðxÞÞ

0 0 A 0

0 0 rgðxÞ^T I rzPlðz þ gðxÞÞ

2 66 66 64

3 77 77 75

: ð7Þ

Proof. Form the proof of[16, Proposition 3.1], it is clear thatrPl(z) and I rPl(z) are positive definite for any

l

> 0 and z 2 R^l. With the help of the definition of the smoothed merit functionWl, part (b) easily follows from the chain rule. h

In light of the main ideas for constructing artificial neural networks (see[6]for details), we establish a specific first order ordinary differential equation, i.e., an artificial neural network. More specifically, based on the gradient of the objective func- tionWlin minimization problem(6), we propose the neural network for solving the KKT system(3)of nonlinear SOCP(1) with the following differential equation:

duðtÞ

dt ¼ 

q

rWlðuÞ; uðt0Þ ¼ u0; ð8Þ

where

q

> 0 is a time scaling factor. In fact, if

s

=

q

t, then^duðtÞ_dt ¼

q

^duð_ds^sÞ. Hence, it follows from(8)that^duð_ds^sÞ¼ rWlðuÞ. In view of this, for simplicity and convenience, we set

q

= 1 in this paper. Indeed, the dynamic system(8)can be realized by an archi- tecture with the cone projection function shown inFig. 1. Moreover, the architecture of this artificial neural network is cat- egorized as a ‘‘recurrent’’ neural network according to the classifications of artificial neural networks as in[6, Chapter 2.3.1].

The circuit for (8)requires n + m + l + 1 integrators, n processors forrf(x), l processors for g(x), ln processors for rg(x), (l + 1)²n processors forr²f ðxÞ þPl

i¼1r²g_iðxÞ, 1 processor forUl, 1 processor for^@Pl_@l;n processors forrxPl,l processors for rzPl, n²+ 4mn + 3ln + l²+ l connection weights and some summers.

4. Stability analysis

In this section, in order to study the stability issues of the proposed neural network(8)for solving the problem(1), we first make an assumption that will be required in our subsequent analysis.

Assumption 4.1.

(a) The problem(1)satisfies the Slater’s condition.

(b) The matrixr²f(x) +r²g1(x) +    +r²gl(x) is positive definite for each x.

Here we say a few words aboutAssumption 4.1(a and b). The Slater’s condition is a standard condition that is widely used in optimization field.Assumption 4.1(b) seems stringent at first glance. Indeed, since f and g_i’s are two order continuously differentiable and convex functions on Rⁿ, if there exists at least one function which is strictly convex among these functions, thenAssumption 4.1(b) is guaranteed.

Lemma 4.1.

(a) For any u, we have

kHðuÞ  Hðu^Þ  Vðu  u^Þk ¼ oðku  u^kÞ for u ! u^and V 2 @HðuÞ where @H(u) denotes the Clarke generalized Jacobian at u.

(b) UnderAssumption 4.1,rH(u)^Tis nonsingular for any u ¼ ð

l

;x; y; zÞ 2 Rþþ Rⁿ R^m R^l, where Rþþ denotes the set {

l

j

l

> 0}.

(c) UnderAssumption 4.1and V 2 @P0(w) being a positive definite matrix where @P0(w) denotes the Clarke generalized Jacobian of the project function P at w, there has

T 2 @HðuÞ ¼

1 0 0 ^{@PlðzþgðxÞÞ}_@l T

j_l¼0 0 A^T r²f ðxÞ þr²g1ðxÞ þ    þr²glðxÞ V^TrgðxÞ

0 0 A 0

0 0 rgðxÞ^T I  V

2 66 66 64

3 77 77 75

jV 2 @P0ðWÞ 8>

>>

>>

<

>>

>>

>:

9>

>>

>>

=

>>

>>

>;

is nonsingular for any u ¼ ð0; x; y; zÞ 2 f0g  Rⁿ R^m R^l. (d)Wl(u(t)) is nonincreasing with respect to t.

Proof. (a) This result follows directly from the definition of semismoothness of H, see[26]for more details.

(b) From the expression ofrH(u) inTheorem 3.1, it follows thatrH(u)^Tis nonsingular if and only if the following matrix Fig. 1. Block diagram of the proposed neural network with smoothed NR function.

M :¼

A 0 0

r²f ðxÞ þr²g₁ðxÞ þ    þr²g_lðxÞ A^T rgðxÞ

rxPlðz þ gðxÞÞ^T 0 ðI rzPlðz þ gðxÞÞÞ^T 2

64

3 75

is nonsingular. Suppose

v

¼ ðx; y; zÞ 2 Rⁿ R^m R^l. To show the nonsingularity of M, it is enough to prove that M

v

¼ 0 ) x ¼ 0; y ¼ 0 and z ¼ 0:

Because rxPl(z + g(x))^T= rPl(w)^Trg(x)^T, where w ¼ z þ gðxÞ 2 R^l, from Mv = 0, we have

Ax ¼ 0; ðr²f ðxÞ þr²g₁ðxÞ þ    þr²g_lðxÞÞx  A^Ty þrgðxÞz ¼ 0 ð9Þ and

rPlðwÞ^TrgðxÞ^Tx þ ðI rPlðwÞÞ^Tz ¼ 0: ð10Þ

From(9), it follows that

x^Tðr²f ðxÞ þr²g1ðxÞ þ    þr²glðxÞÞx þ ðrgðxÞ^TxÞ^Tz ¼ 0: ð11Þ Moveover, Eq.(10)andTheorem 3.1yield

rgðxÞ^Tx ¼ ðrPlðwÞ^TÞ^1ðI rPlðwÞÞ^Tz: ð12Þ

Combining(11) and (12)andTheorem 3.1, under the condition ofAssumption 4.1, it is not hard to obtain that x = 0 and z = 0.

By looking at Eq.(9)again, since A is full row rank, we have y = 0. Therefore,rH(u)^Tis nonsingular.

(c) The proof of Part (c) is similar to that of Part (b), in which the only option is to replacerP_l(w) with V 2 @ P0(w).

(d) According to the definition ofWl(u(t)) and Eq.(8), it is clear that dWlðuðtÞÞ

dt ¼rWlðuðtÞÞduðtÞ

dt ¼ 

q

krWlðuðtÞÞk²60:

Consequently,Wl(u(t)) is nonincreasing with respect to t. h

Proposition 4.1. Assume thatrH(u) is nonsingular for any u 2 R_þ Rⁿ R^m R^l. Then,

(a) (x^⁄, y^⁄, z^⁄) satisfies the KKT conditions(3)if and only if (0, x^⁄, y^⁄, z^⁄) is an equilibrium point of the neural network(8);

(b) Under the Slater’s condition, x^⁄is a solution to the problem(1)if and only if (0, x^⁄, y^⁄, z^⁄) is an equilibrium point of the neural network(8).

Proof. (a) BecauseU0=UNRwhen

l

= 0, it follows that (x^⁄, y^⁄, z^⁄) satisfies the KKT conditions(3)if and only if H(u^⁄) = 0, where u^⁄= (0, x^⁄, y^⁄, z^⁄)^T. SincerH(u) is nonsingular, we have that H(u^⁄) = 0 if and only ifr Wl(u^⁄) =rH(u^⁄)^TH(u^⁄) = 0. Thus, the desired result follows.

(b) Under the Slater’s condition, it is well known that x^⁄is a solution of the problem(1)if and only if there exist y^⁄and z^⁄ such that (x^⁄, y^⁄, z^⁄) satisfying the KKT conditions(3). Hence, according to Part (a), it follows that (0, x^⁄, y^⁄, z^⁄) is an equilibrium point of the neural network(8). h

The next result addresses the existence and uniqueness of the solution trajectory of the neural network(8).

Theorem 4.1.

(a) For any initial point u0= u(t0), there exists a unique continuously maximal solution u(t) with t 2 [t0,

s

) for the neural net- work(8), where [t0,

s

) is the maximal interval of existence.

(b) If the level setLðu0Þ :¼ fujWlðuÞ 6Wlðu0Þg is bounded, then

s

can be extended to +1.

Proof. This proof is exactly the same as the proof of[27, Proposition 3.4], and therefore omitted here. h

Theorem 4.2. Assume thatrH(u) is nonsingular and that u^⁄is an isolated equilibrium point of the neural network(8). Then the solution of the neural network(8)with any initial point u0is Lyapunov stable.

Proof. FromLemma 2.3, we only need to argue that there exists a Lyapunov function over some neighborhoodXof u^⁄. Now, we consider the smoothed merit function

WlðuÞ ¼1 2kHðuÞk²:

Since u^⁄is an isolated equilibrium point of(8), there is a neighborhoodXof u^⁄such that rWlðu^Þ ¼ 0 and rWlðuðtÞÞ – 0; 8uðtÞ 2Xn fu^g:

By the nonsingularity ofrH(u) and the definition ofWl, it is easy to obtain thatWl(u^⁄) = 0. From the definition ofWl, we claim thatWl(u(t)) > 0 for any u(t) 2Xn{u^⁄}, whereXis a neighborhood of u^⁄. Suppose not, namely,Wl(u(t)) = 0. It follows that H(u(t)) = 0. Then, we haverWl(u(t)) = 0 which contradicts with the assumption that u^⁄is an isolated equilibrium point of(8). Thus,Wl(u(t)) > 0 for any u(t) 2Xn{u^⁄}. Furthermore, by the proof ofLemma 4.1(d), we know that for any u(t) 2X

dWlðuðtÞÞ

dt ¼rWlðuðtÞÞduðtÞ

dt ¼ 

q

krWlðuðtÞÞk²60: ð13Þ

Consequently, the functionWlis a Lyapunov function overX. This implies that u^⁄is Lyapunov stable for the neural network (8). h

Theorem 4.3. Assume thatrH(u) is nonsingular and that u^⁄is an isolated equilibrium point of the neural network(8). Then u^⁄is asymptotically stable for neural network(8).

Proof. From the proof ofTheorem 4.2, we consider again the Lyapunov functionWl. ByLemma 2.3again, we only need to verify that the Lyapunov functionWlover some neighborhoodXof u^⁄satisfies

dWlðuðtÞÞ

dt <0; 8uðtÞ 2X_{n fu}^_{g: ð14Þ}

In fact, by using(13)and the definition of the isolated equilibrium point, it is not hard to check that the Eq.(14)is true.

Hence, u^⁄is asymptotically stable. h

Theorem 4.4. Assume that u^⁄is an isolated equilibrium point of the neural network (8). If rH(u)^Tis nonsingular for any u ¼ ð

l

;x; y; zÞ 2 Rþ Rⁿ R^m R^l, then u^⁄is exponentially stable for the neural network(8).

Proof. From the definition of H(u), we know that H is semismooth. Hence, byLemma 4.1, we have

HðuÞ ¼ Hðu^Þ þrHðuðtÞÞ^Tðu  u^Þ þ oðku  u^kÞ; 8u 2Xn fu^g; ð15Þ whererH(u(t))^T2 @H(u(t)) andXis a neighborhood of u^⁄. Now, we let

gðuðtÞÞ ¼ kuðtÞ  u^k²; t 2 ½t0;1Þ:

Then, we have dgðuðtÞÞ

dt ¼ 2ðuðtÞ  u^Þ^TduðtÞ

dt ¼ 2

q

ðuðtÞ  u^Þ^TrWlðuðtÞÞ ¼ 2

q

ðuðtÞ  u^Þ^TrHðuÞHðuÞ: ð16Þ Substituting Eq.(15)into Eq.(16)yields

dgðuðtÞÞ

dt ¼ 2

q

ðuðtÞ  u^Þ^TrHðuðtÞÞðHðu^Þ þrHðuðtÞÞ^TðuðtÞ  u^Þ þ oðkuðtÞ  u^kÞÞ

¼ 2

q

ðuðtÞ  u^Þ^TrHðuðtÞÞrHðuðtÞÞ^TðuðtÞ  u^Þ þ oðkuðtÞ  u^k²Þ:

BecauserH(u) andrH(u)^Tare nonsingular, we claim that there exists an

j

> 0 such that

ðuðtÞ  u^Þ^TrHðuÞrHðuÞ^TðuðtÞ  u^Þ P

j

kuðtÞ  u^k²: ð17Þ

Otherwise, if (u(t)  u^⁄)^TrH(u(t))rH(u(t))^T(u(t)  u^⁄) = 0, it implies that rHðuðtÞÞ^TðuðtÞ  u^Þ ¼ 0:

Indeed, from the nonsingularity of H(u), we have u(t)  u^⁄= 0, i.e., u(t) = u^⁄which contradicts with the assumption of u^⁄being an isolated equilibrium point. Consequently, there exists an

j

> 0 such that(17)holds. Moreover, for o(ku(t)  u^⁄k²), there is

e

> 0 such that o(ku(t)  u^⁄k²) 6

e

ku(t)  u^⁄k². Hence, dgðuðtÞÞ

dt 6ð2

qj

þ

e

ÞkuðtÞ  u^k²¼ ð2

qj

þ

e

ÞgðuðtÞÞ:

This implies

gðuðtÞÞ 6 e^ð2qjþeÞtgðuðt0ÞÞ

which means

kuðtÞ  u^k 6 e^qjþ2ekuðt0Þ  u^k:

Thus, u^⁄is exponentially stable for the neural network(8). h

To show the contribution of this paper, we present the stability comparisons of neural networks considered in the current paper,[20,27]inTable 1. More convergence comparisons will be presented in the next section. Generally speaking, we estab- lish three stabilities for the proposed neural network, whereas not all three stabilities for the similar neural networks studied in[20,27]are guaranteed. Why do we choose to investigate the proposed neural network? Indeed, in[20], two neural net- works based on NR function and FB function are considered which does not reach exponential stability. Our target optimi- zation problem is a wider class than the one studied in[20]. In contrast, the smoothed FB has good performance as is shown in[27], but not all the three stabilities are established even though exponential stability is good enough. In light of these observations, we decide to look into the smoothed NR function for our problem which turns out to have better theoretical results. We summarize their differences in problems format, dynamical model, and stability issues inTable 1.

5. Numerical examples

In order to demonstrate the effectiveness of the proposed neural network, in this section we test several examples for our neural network(8). The numerical implementation is coded by Matlab 7.0 and the ordinary differential equation solver adopted here is ode23, which uses the Ruge–Kutta (2; 3) formula. As mentioned earlier, the parameter

q

is set to be 1.

How is

l

chosen initially? FromTheorem 4.2in previous section, we know the solution converges with any initial point, we set initial

l

= 1 in the codes (and of course

l

?0, as seen in the trajectory behavior).

Example 5.1. Consider the following nonlinear convex programming problem:

min e^{ðx13Þ2þx22þðx31Þ2þðx42Þ2þðx5þ1Þ2} s:t: x 2K⁵;

Here, we denote f ðxÞ :¼ e^{ðx13Þ2þx22þðx31Þ2þðx42Þ2þðx5þ1Þ2}and g(x) = x. Then, we compute

Lðx; zÞ ¼rf ðxÞ þrgðxÞz ¼ 2f ðxÞ x1 3

x2

x3 1 x4 2 x5þ 1 2 66 66 66 4

3 77 77 77 5

 z1

z2

z3

z4

z5

2 66 66 66 4

3 77 77 77 5

: ð18Þ

Moreover, let x :¼ ðx1; xÞ 2 R  R⁴and z :¼ ðz1; zÞ 2 R  R⁴. Then, the element z  x can be expressed as z  x :¼ k1e1þ k2e2

where ki¼ z1 x1þ ð1Þⁱkz  xk and ei¼¹₂1; ð1Þ^{i zx}_kzxk

ði ¼ 1; 2Þ if z  x – 0, otherwise ei¼¹₂ð1; ð1ÞⁱwÞ with w being any vector in R⁴satisfying kwk = 1. This implies that

Ulðz; gðxÞÞ ¼ z  Plðz þ gðxÞÞ ¼ z 

l

^{^}g k1

l

 

e1þ

l

^{^}g k2

l

 

e2 ð19Þ

with ^gðaÞ ¼ ffiffiffiffiffiffiffiffi

a²þ4

p

þa

2 or ^gðaÞ ¼ lnðe^aþ 1Þ. Therefore, by Eqs.(18) and (19), we obtain the expression of H(u) as follows:

HðuÞ ¼

l

Lðx; zÞ Ulðz; gðxÞÞ 2

64

3 75:

Table 1

Stability comparisons of neural networks considered in current paper,[20,27].

Current paper [20] [27]

Problem min f ðxÞ s:t: Ax ¼ b

gðxÞ 2K

min f ðxÞ s:t: Ax ¼ b

x 2K

hFðxÞ; y  xi P 0; 8y 2 C C ¼ fxj hðxÞ ¼ 0; gðxÞ 2Kg

ODE Based on smoothed NR-function Based on NR-function and FB-function Based on NR-function and smoothed FB-function

Stability Lyapunov (smoothed NR) Lyapunov (NR) Lyapunov (NR)

Asymptotical (smoothed NR) Lyapunov (FB) Asymptotical (NR)

Exponential (smoothed NR) asymptotical (FB) Exponential (smoothed FB)

This problem has an optimal solution x^⁄= (3, 0, 1, 2, 1)^T. We use the proposed neural network to solve the above problem whose trajectories are depicted inFig. 2. All simulation results show that the state trajectories with any initial point are al- ways convergent to an optimal solution of the above problem x^⁄.

Example 5.2. Consider the following nonlinear second-order cone programming problem:

min f ðxÞ ¼ x²₁þ 2x²₂þ 2x1x2 10x1 12x2

s:t: gðxÞ ¼ 8  x1þ 3x2

3  x²₁ 2x1þ 2x2 x²₂

 

2K²: For this example, we compute that

Lðx; zÞ ¼rf ðxÞ þrgðxÞz ¼ 2x1 2x2 10 4x2þ 2x1 12

 

 z1 2ðx1þ 1Þz2

3z3þ 2ð1  x2Þz2

 

: ð20Þ

Since

z  gðxÞ ¼ z1 8 þ x1 3x2

z2 3 þ x²1þ 2x1 2x2þ x²2

 

; the vector z  g(x) can be expressed as

z  x :¼ k1e1þ k2e2

where ki¼ z1 8 þ x1 3x2þ ð1Þⁱjz2 3 þ x²₁þ 2x1 2x2þ x²₂j and

ei¼1

2 1; ð1Þⁱ z2 3 þ x²1þ 2x1 2x2þ x²2

jz2 3 þ x²₁þ 2x1 2x2þ x²₂j

 

ði ¼ 1; 2Þ; if z2 3 þ x²₁þ 2x1 2x2þ x²₂–0;

otherwise, ei¼¹₂ð1; ð1ÞⁱwÞ with w being any element in R satisfying jwj = 1. This implies that Ulðz; gðxÞÞ ¼ z  Plðz þ gðxÞÞ ¼ z 

l

g^{^} k1

l

 

e1þ

l

g^{^} k2

l

 

e2 ð21Þ

with ^gðaÞ ¼ ffiffiffiffiffiffiffiffi

a²þ4

p

þa

2 or ^gðaÞ ¼ lnðe^aþ 1Þ. Therefore, by(20) and (21), we obtain the expression of H(u) as follows:

HðuÞ ¼

l

Lðx; zÞ Ulðz; gðxÞÞ 2

64

3 75:

This problem has an approximate solution x^⁄= (2.8308, 1.6375)^T. Note that the objective function is convex and the Hes- sian matrixr²f(x) is positive definite. Using the proposed neural network in this paper, we can easily obtain the approximate solution x^⁄of the above problem, seeFig. 3.

0 1 2 3 4 5 6

−1.5

−1

−0.5 0 0.5 1 1.5 2 2.5 3 3.5

Time (ms)

Trajectories of x(t)

x1

x2 x3 x4

x5

Fig. 2. Transient behavior of the neural network with the smoothed NR function inExample 5.1.

Example 5.3. Consider the following nonlinear convex program with second-order cone constraints[18]:

min e^ðx1x3Þþ 3ð2x1 x2Þ⁴þ

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þ ð3x2þ 5x3Þ² q

s:t: gðxÞ ¼ Ax þ b 2K²; x 2K³

where

A :¼ 4 6 3

1 7 5

 

;b :¼ 1 2

 

:

For this example, f ðxÞ :¼ e^ðx1x3Þþ 3ð2x1 x2Þ⁴þ

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þ ð3x2þ 5x3Þ² q

, from which we have

Lðx; y; zÞ ¼rf ðxÞ þrgðxÞy rxz ¼

e^ðx1x3Þþ 24ð2x1 x2Þ³

12ð2x1 x2Þ³þ ^3ð3xffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi^2þ5x3Þ

1þð3x2þ5x3Þ²

p

e^ðx1x3Þþ ^5ð3xffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi^2þ5x3Þ

1þð3x2þ5x3Þ²

p 2

66 64

3 77 75

4y1 y2

6y1þ 7y2

3y₁ 5y2

2 64

3 75 

z1

z2

z3

2 64

3

75: ð22Þ

Since

y þ gðxÞ z  x

 

¼

y1 4x1 6x2 3x3þ 1 y2þ x1 7x2þ 5x3 2

z1 x1

z2 x2

z3 x3

2 66 66 66 4

3 77 77 77 5

;

y + g(x) and z  x can be expressed as follows, respectively, y þ gðxÞ :¼ k1e1þ k2e2

and

z  x :¼

j

1f1þ

j

2f2

where ki= y1 4x1 6x2 3x3+ 1 + (1)ⁱjy2+ x1 7x2+ 5x3 2j and ei¼1

2 1; ð1Þⁱ y₂þ x1 7x2þ 5x3 2 jy2þ x1 7x2þ 5x3 2j

 

ði ¼ 1; 2Þ if y2þ x1 7x2þ 5x3 2 – 0;

otherwise, ei¼¹₂ð1; ð1ÞⁱwÞ with w being any element in R satisfying jwj = 1. Moveover, let x :¼ ðx1; xÞ 2 R  R² and z :¼ ðz1; zÞ 2 R  R². Then, we obtain that

j

i¼ z1 x1þ ð1Þⁱkz  xk and fi¼¹₂ð1; ð1Þ^{i zx}_kzxkÞði ¼ 1; 2Þ if z  x – 0; otherwise f_i¼¹₂ð1; ð1Þⁱ

t

Þ withtbeing any vector in R²satisfying ktk = 1. This implies that

0 5 10 15 20 25 30

−0.5 0 0.5 1 1.5 2 2.5 3

Time (ms)

Trajectories of x(t)

x1

x2

Fig. 3. Transient behavior of the neural network with the smoothed NR function inExample 5.2.

U_{lðÞ ¼} ^{y  Plðy þ gðxÞÞ} z  Plðz  xÞ

 

¼

y 

l

^{^}g ^k_l¹

e1þ

l

^{^}g ^k_l² e2

z 

l

^{^}g ^j_l¹

f1þ

l

^{^}g ^j_l² f2

2 64

3

75 ð23Þ

with ^gðaÞ ¼ ffiffiffiffiffiffiffiffi

a²þ4

p

þa

2 or ^gðaÞ ¼ lnðe^aþ 1Þ. Therefore, by(22) and (23), we obtain the expression of H(u) as below:

HðuÞ ¼

l

Lðx; y; zÞ UlðÞ 2 64

3 75:

The approximate solution to this problem is x^⁄= (0.2324, 0.07309, 0.2206)^T. The trajectories are depicted inFig. 4. We want to point out on thing:Assumption 4.1(a and b) are not both satisfied in this example. More specifically, for this example, the Assumption 4.1(a) is satisfied, which is obvious. However,r²f(x) +r²g1(x) +    +r²gl(x) is not positive semidefinite for each x. To see this, we compute

r²f ðxÞ þr²g₁ðxÞ þ    þr²g_lðxÞ ¼r²f ðxÞ ¼

e^ðx1x3Þþ 144ð2x1 x2Þ² 72ð2x1 x2Þ² e^ðx1x3Þ

72ð2x1 x2Þ² 36ð2x1 x2Þ²þ ⁹

ð1þð3x2þ5x3Þ²Þ 3 2

15 ð1þð3x2þ5x3Þ²Þ

3 2

e^{ðx1x3Þ 15}

ð1þð3x₂þ5x₃Þ²Þ³2

e^ðx1x3Þþ ²⁵

ð1þð3x₂þ5x₃Þ²Þ³2

2 66 66 4

3 77 77 5;

which is not positive semidefinite when 2x1 x2= 0 (because the determinant equals zero). Hence, H(u) is not guaranteed to be nonsingular and all the theorems in Section4do not apply for this example. Nonetheless, the solution trajectory does converge as depicted inFig. 4. This phenomenon also occurs when it is solved by the second neural network studied in [27](the stability is not guaranteed theoretically, but the solution trajectory does converges).

In addition, forExample 5.3, we also do comparisons among three neural networks based on FB function (considered in [20]), smoothed NR function (considered in this paper), and smoothed FB function (considered in [27]), respectively.

AlthoughExample 5.3can be solved by all three neural networks, the neural network based on FB function does not behave as good as the other two neural networks, seeFig. 5.

Example 5.4. Consider the following nonlinear second-order cone programming problem:

min f ðxÞ ¼ e^x1x3þ 3ðx1þ x2Þ²

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þ ð2x2 x3Þ² q

þ¹₂x²₄þ¹₂x²₅ s:t: hðxÞ ¼ 24:51x1þ 58x2 16:67x3 x4 3x5þ 11 ¼ 0

g1ðxÞ ¼

3x³₁þ 2x2 x3þ 5x²3

5x³1þ 4x2 2x3þ 10x³3

x3

0 B@

1 CA 2 K³;

g2ðxÞ ¼ x4

3x5

 

2K²:

0 5 10 15 20 25 30

−0.2

−0.15

−0.1

−0.05 0 0.05 0.1 0.15 0.2 0.25 0.3

Time (ms)

Trajectories of x(t)

x1

x2 x3

Fig. 4. Transient behavior of the neural network with the smoothed NR function inExample 5.3.

For this example, we compute

Lðx; y; zÞ ¼rf ðxÞ þ rg1ðxÞy rg2ðxÞz

 

¼

x3e^ðx1x3Þþ 6ðx1þ x2Þ 6ðx1þ x2Þ þ ^2ð2xffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi^2x3Þ

1þð2x2x3Þ²

p x1e^ðx1x3Þþ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi^2x2x3

1þð2x2x3Þ²

p x4

x5

2 66 66 66 66 4

3 77 77 77 77 5



9x²₁y₁ 15x²1y₂ 2y₁þ 4y₂

ð10x3 1Þy1þ ð30x²₃Þy2þ y3

z1

3z2

2 66 66 66 4

3 77 77 77 5

: ð24Þ

Moreover, we know

y þ g1ðxÞ z þ g₂ðxÞ

 

¼

y1 3x³₁ 2x2þ x3 5x²₃ y2þ 5x³₁ 4x2þ 2x3 10x³₃

y3 x3

z1 x4

z2 x5

2 66 66 66 4

3 77 77 77 5 :

Let y þ g₁ðxÞ :¼ ðu; uÞ 2 R  R²and z þ g₂ðxÞ :¼ ð

v

^{; }

v

Þ 2 R  R, where u ¼ y₁ 3x³₁ 2x2þ x3 5x²₃, u ¼ y₂þ 5x³1 4x2þ 2x3 10x³3

y₃ x3

" #

and

v

¼ z1 x4;

v

¼ z2 x5:

Then, y + g1(x) and z + g2(x) can be expressed as follows:

y þ g₁ðxÞ :¼ k1e1þ k2e2

and

z þ g₂ðxÞ :¼

j

1f1þ

j

2f2

where k_i¼ u þ ð1Þⁱkuk; e_i¼¹₂1; ð1Þ^{i }_k^u_uk

and

j

_i¼

v

þ ð1Þⁱj

v

j; f_i¼¹₂1; ð1Þ^{i }_j^v_vj

(i = 1, 2) if u – 0 and 

v

^–0, otherwise ei¼¹₂ð1; ð1ÞⁱwÞ with w being any element in R²satisfying kwk = 1, and fi¼¹₂ð1; ð1Þⁱ

t

Þ withtbeing any vector in R satis- fying jtj = 1. This implies that

UlðÞ ¼ y  Plðy þ g1ðxÞÞ z  Plðz þ g2ðxÞÞ

 

¼

y 

l

^{^}g ^k_l¹

e1þ

l

^{^}gð^k_l²Þe2

z 

l

^{^}g ^j_l¹

f1þ

l

g^{^} ^j_l² f2

2 64

3

75 ð25Þ

0 20 40 60 80 100

10⁻⁶ 10⁻⁵ 10⁻⁴ 10⁻³ 10⁻² 10⁻¹ 10⁰

Time (ms)

Norm of error

Smoothed NR Smoothed FB FB

Fig. 5. Comparisons of three neural networks based on the FB function, smoothed NR function, and smoothed FB function inExample 5.3.

with ^gðaÞ ¼ ffiffiffiffiffiffiffiffi

a²þ4

p

þa

2 or ^gðaÞ ¼ lnðe^aþ 1Þ. Consequently, by Eqs.(24) and (25), we obtain the expression of H(u) as follows:

HðuÞ ¼

l

Lðx; y; zÞ UlðÞ 2 64

3 75:

This problem has an approximate solution x^⁄= (0.0903, 0.0449, 0.6366, 0.0001, 0)^TandFig. 6displays the trajectories obtained by using the proposed new neural network. All simulation results show that the state trajectory with any initial point are always convergent to the solution x^⁄. As observed, the neural network with the smoothed NR function has a fast convergence rate.

Furthermore, we also do comparisons between two neural networks based on smoothed NR function (considered in this paper) and smoothed FB function (considered in[27]) for Example 5.5. Note that Example 5.5 cannot be solved by the neural networks studied in[20]. Both neural networks possess exponential stability as shown inTable 1, which means the solution trajectories have the same order of convergence. This phenomenon is reflected inFig. 7.

6. Conclusion

In this paper, we have studied a neural network approach for solving general nonlinear convex programs with second- order cone constraints. The proposed neural network is based on the gradient of the merit function derived from smoothed

0 20 40 60 80 100 120 140 160

−0.2

−0.1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Time (ms)

Trajectories of x(t)

x1 x2 x3

x4, x5

Fig. 6. Transient behavior of the neural network with the smoothed NR function inExample 5.4.

0 50 100 150 200

10⁻⁶ 10⁻⁵ 10⁻⁴ 10⁻³ 10⁻² 10⁻¹ 10⁰

Time (ms)

Norm of error

Smoothed NR Smoothed FB

Fig. 7. Comparisons of two neural network based on smoothed NR function and smoothed FB function inExample 5.4.

NR complementarity function. In particular, fromDefinition 2.1andLemma 2.3, we know that there exists a stable equilib- rium point u^⁄as long as there exists a Lyapunov function over some neighborhood of u^⁄, and the stable equilibrium point u^⁄is exactly the solution of our considering problem. In addition to studying the existence and convergence of the solution tra- jectory of the neural network, this paper shows that the merit function is a Lyapunov function. Furthermore, the equilibrium point of the neural network(8)is stable, including the stability in the sense of Lyapunov, asymptotic stability, and exponen- tial stability under suitable conditions.

Indeed, this paper can be viewed as a follow-up of[20,27]because we establish three stabilities for the proposed neural network, but not all three stabilities for the similar neural networks studied in[20,27]are guaranteed. The numerical exper- iments presented in this study demonstrate the efficiency of the proposed neural network.

Acknowledgment

The authors are grateful to the reviewers for their valuable suggestions, which have considerably improved the paper a lot.

Appendix A

For the function Pl(z) defined as in(5), the following Lemma provides the gradient matrix of Pl(z), which will be used in numerical computation and coding.

Lemma 6.1. For any z ¼ ðz1;z^T₂Þ^T2 Rⁿ, the gradient matrix of Pl(z) is written as

rPlðzÞ ¼

^g^{0 z} _l¹

I if z2¼ 0;

bl

clz^T 2 kz2k clz2

kz2k alI þ ðbl alÞ^z2z

T 2 kz2k²

2 64

3

75 if z²–0;

8>

>>

><

>>

>>

: where

al¼

^g ^k_l²

 ^g ^k_l¹

k2

l^k_l¹ ; bl¼1

2 g^⁰ k2

l

  þ ^g⁰ k1

l

 

 

; cl¼1

2 g^⁰ k2

l

 

 ^g⁰ k1

l

 

 

; and I denotes the identity matrix.

Proof. See Proposition 5.2 in[11]. h

References

[1]F. Alizadeh, D. Goldfarb, Second-order cone programming, Mathematical Programming 95 (2003) 3–52.

[2]D.P. Bertsekas, Nonlinear Programming, Athena Scientific, 1995.

[3]Y.-H. Chen, S.-C. Fang, Solving convex programming problems with equality constraints by neural networks, Computers and Mathematics with Applications 36 (1998) 41–68.

[4]J.-S. Chen, C.-H. Ko, S.-H. Pan, A neural network based on the generalized Fischer–Burmeister function for nonlinear complementarity problems, Information Sciences 180 (2010) 697–711.

[5]J.-S. Chen, P. Tseng, An unconstrained smooth minimization reformulation of the second-order cone complementarity problem, Mathematical Programming 104 (2005) 293–327.

[6]A. Cichocki, R. Unbehauen, Neural Networks for Optimization and Signal Processing, John wiley, New York, 1993.

[7]C. Dang, Y. Leung, X. Gao, K. Chen, Neural networks for nonlinear and mixed complementarity problems and their applications, Neural Networks 17 (2004) 271–283.

[8]S. Effati, A. Ghomashi, A.R. Nazemi, Application of projection neural network in solving convex programming problems, Applied Mathematics and Computation 188 (2007) 1103–1114.

[9]S. Effati, A.R. Nazemi, Neural network and its application for solving linear and quadratic programming problems, Applied Mathematics and Computation 172 (2006) 305–331.

[10] F. Facchinei, J. Pang, Finite-dimensional Variational Inequalities and Complementarity Problems, Springer, New York, 2003.

[11]M. Fukushima, Z.-Q. Luo, P. Tseng, Smoothing functions for second-order-cone complementarity problems, SIAM Journal on Optimization 12 (2002) 436–460.

[12]Q. Han, L.-Z. Liao, H. Qi, L. Qi, Stability analysis of gradient-based neural networks for optimization problems, Journal of Global Optimization 19 (2001) 363–381.

[13]J.J. Hopfield, D.W. Tank, Neural computation of decision in optimization problems, Biological Cybernetics 52 (1985) 141–152.