Recurrentneuralnetworksforsolvingsecond-orderconeprograms Neurocomputing

Recurrent neural networks for solving second-order cone programs

Chun-Hsu Ko

^a

, Jein-Shan Chen

^b,1,

^ , Ching-Yu Yang

^b

aDepartment of Electrical Engineering, I-Shou University, Kaohsiung County 840, Taiwan

bDepartment of Mathematics, National Taiwan Normal University, Taipei 11677, Taiwan

a r t i c l e i n f o

Article history:

Received 6 January 2011 Received in revised form 1 June 2011

Accepted 14 July 2011 Communicated by Y. Liu Available online 10 August 2011 Keywords:

SOCP Neural network Merit function

Fischer–Burmeister function Cone projection function Lyapunov stable

a b s t r a c t

This paper proposes using the neural networks to efficiently solve the second-order cone programs (SOCP). To establish the neural networks, the SOCP is first reformulated as a second-order cone complementarity problem (SOCCP) with the Karush–Kuhn–Tucker conditions of the SOCP. The SOCCP functions, which transform the SOCCP into a set of nonlinear equations, are then utilized to design the neural networks. We propose two kinds of neural networks with the different SOCCP functions. The first neural network uses the Fischer–Burmeister function to achieve an unconstrained minimization with a merit function. We show that the merit function is a Lyapunov function and this neural network is asymptotically stable. The second neural network utilizes the natural residual function with the cone projection function to achieve low computation complexity. It is shown to be Lyapunov stable and converges globally to an optimal solution under some condition. The SOCP simulation results demonstrate the effectiveness of the proposed neural networks.

&2011 Elsevier B.V. All rights reserved.

1. Introduction

Second-order cone program (SOCP) has been widely applied in engineering optimization[1]. It requires solving the optimization problem subject to the linear equality and second-order cone inequality constraints [2]. Numerical approaches such as the interior-point method[1] or the merit function method[3] can effectively solve the SOCP. However, many engineering dynamic systems, such as force analysis in robot grasping[1,4] and control applications [5,6], require the real-time SOCP solutions. As a result, efficient approaches for solving the real-time SOCP are needed. Prior research[7–18] indicates that the neural networks can be used to solve various optimization problems. Furthermore, the neural networks based on circuit implementation exhibit the real-time processing ability. We consider that it is appropriate to utilize the neural networks for efficiently solving the SOCP problems.

The recurrent neural network was introduced by Hopfield and Tank[7]for solving linear programming problems. Kennedy and Chua[8]proposed an extended neural network for solving non- linear convex programming problems thereafter, while their

approach involves the penalty parameter which affects the neural network accuracy. To find the exact solutions, more neural networks for optimization have been further developed. Among them, the primal-dual neural network [9–11] with the global stability is proposed for providing the exact solutions of the linear and quadratic programming problems. The projection neural network, developed by Xia and Wang[12,14,15], was proposed to efficiently solve many optimization problems and variational inequalities. Since the SOCP is a nonlinear convex problem, both primal-dual neural network[16]and projection neural network [17]can be used to provide the SOCP solution. However, they require many state variables, leading to high model complexity. It thus motivates the development of more compact neural networks for SOCP.

The SOCP can be solved by analyzing its Karush–Kuhn–Tucker (KKT) optimality conditions which leads to the second-order cone complementarity problem (SOCCP) [3,19,20]. The approaches [3,20] based on the SOCCP functions, such as Fischer–Burmeister (FB) and natural residual functions, can be further utilized for solving the SOCCP. In the merit function approach [3], an unconstrained smooth minimization with the FB function is achieved in finding the SOCCP solution. On the other hand, the semi-smooth approach [20] uses the natural residual function with the cone projection (CP) function to reformulate the SOCCP as a set of nonlinear equations and then apply the non-smooth Newton method to obtain the solution. Previous studies have demonstrated the feasibility of these SOCCP functions in solving the SOCP problems. We also use them in our neural network Contents lists available atScienceDirect

journal homepage:www.elsevier.com/locate/neucom

Neurocomputing

0925-2312/$ - see front matter & 2011 Elsevier B.V. All rights reserved.

doi:10.1016/j.neucom.2011.07.009

Corresponding author. Member of Mathematics Division, National Center for Theoretical Sciences, Taipei Office.

E-mail addresses: [email protected] (C.-H. Ko), [email protected] (J.-S. Chen), [email protected] (C.-Y. Yang).

1The author’s work is partially supported by National Science Council of Taiwan.

design. In this paper, we propose two novel neural networks for efficiently solving the SOCP problems. One is based on the gradient of the smooth merit function derived from the FB function[18]. The other is an extended projection neural network by replacing the scalar projection function[12,14,15] with the CP function. These neural networks are with less state variables than those previously proposed[16,17] for solving the SOCP. Further- more, they are shown to be stable and globally convergent to the SOCP solutions.

This paper is organized as follows.Section 2introduces the second-order cone program and its SOCCP formulation. InSection 3, the neural network based on the Fischer–Burmeister function is proposed and analyzed. InSection 4, the second neural network based on the cone projection function is proposed. Its global stability is also verified. InSection 5, several SOCP examples are presented to demonstrate the effectiveness of the proposed neural networks. Finally, the conclusions are given inSection 6.

2. Problem formulation

In this section, we introduce the second-order cone program and reformulate it as a second-order cone complementarity problem. The second-order cone program is in the form of minimize f ðxÞ

subject to Ax ¼ b, x A K: ð1Þ

Here f :Rⁿ_-R is a nonlinear continuously differentiable func- tion, A AR^mnis a full row rank matrix, bAR^mis a vector, and K is a Cartesian product of second-order cones (or Lorentz cones), expressed as

K ¼ Kⁿ¹Kⁿ²    K^nN, ð2Þ

where N,n1, . . . ,nNZ1,n1þ    þnN¼n, and Kⁿⁱ:¼ fðx_i1,x_i2, . . . ,x_iniÞ^TARⁿⁱ_{9 Jðx}i2, . . . ,x_iniÞJrxi1g

with J  J denoting the Euclidean norm and K¹ the set of non- negative realsRþ. A special case of Eq. (2) is K ¼Rⁿ_þ, namely the nonnegative orthant in Rⁿ, which corresponds to N ¼n and n1¼    ¼nN¼1. When f is linear, i.e., f ¼ c^Tx with c ARⁿ, SOCP (1) reduces to the following linear SOCP:

minimize c^Tx

subject to Ax ¼ b, x A K: ð3Þ

The KKT optimality conditions for (1) are given by

r

f ðxÞA^Tyl¼0, x^Tl¼0, x A K, l^AK, Ax ¼ b,

8>

<

>: ð4Þ

where y AR^mandl^ARⁿ. When f is convex, these conditions are sufficient for optimality. It also can be written as

x^Tð

r

f ðxÞA^TyÞ ¼ 0, x A K,

r

f ðxÞA^Ty A K, Ax ¼ b:

(

ð5Þ

By solving the system (5), we may obtain a primal-dual optimal solution of SOCP (1). Note that system (5) involves the SOCCP. To efficiently solve it, we propose using the neural network approaches with the FB function and CP function, respectively, described below.

3. Neural network design with Fischer–Burmeister function

It is known that the merit function approach[3]can be used for solving system (5). Motivated by this approach, we propose a

neural network with the Fischer–Burmeister function to find the minimal of the merit function and study its global stability.

In [3], system (5) is shown to be equivalent to an uncon- strained smooth minimization problem via the merit function approach, described as

min Eðx,yÞ ¼CFBðx,

r

f ðxÞA^TyÞ þ¹₂JAxbJ², ð6Þ where Eðx,yÞ is a merit function, CFBðx,yÞ ¼¹₂PN

i ¼ 1Jf_FBðxi,yiÞJ², x ¼ ðx1, . . . ,xNÞ^T, y ¼ ðy1, . . . ,yNÞ^TARⁿ¹    R^nN, and f_FB is the Fischer–Burmeister function defined as

f_FBðxi,yiÞ:¼ ðx²_iþy²_iÞ¹⁼²xiyi: ð7Þ Based on the gradient of the objective Eðx,yÞ in minimization problem (6), we propose the first neural network for solving the SOCP, with the following dynamic equation:

d dt

x y !

¼

r

^

r

xEðx,yÞ



r

yEðx,yÞ

!

, ð8Þ

where

r

is a positive scaling factor and

r

xEðx,yÞ ¼

r

xCFBðx,

r

f ðxÞA^TyÞ

þ

r

²f ðxÞ 

r

yCFBðx,

r

f ðxÞA^TyÞ þ A^TðAxbÞ,

r

yEðx,yÞ ¼ A 

r

yCFBðx,

r

f ðxÞA^TyÞ:

8>

><

>>

:

ð9Þ

For linear SOCP (3), the above equations reduce to

r

xEðx,yÞ ¼

r

xCFBðx,cA^TyÞ þ A^TðAxbÞ,

r

yEðx,yÞ ¼ A 

r

yCFBðx,cA^TyÞ:

(

ð10Þ

Note that the Jordan product[3]is required for calculating

r

xCFB

and

r

yCFBwhich are introduced in Appendix. And, the dynamic equation (8) can be realized by a recurrent neural network with FB function as shown inFig. 1. The circuit for the neural network realization requires n þ m integrators, n processors for

r

f ðxÞ, n² processors for

r

²f ðxÞ, n processors for

r

xCFB, m processors for

r

yCFB, 4mn connection weights and some summers. Further- more, the neural network (8) is asymptotically stable, as proven in the following theorem.

Fig. 1. Block diagram of the proposed neural network with FB function.

Theorem 3.1. If uⁿ¼ ðxⁿ,yⁿÞ is an isolated equilibrium point of neural network (8), then uⁿ¼ ðxⁿ,yⁿÞis asymptotically stable for (8).

Proof. We assume that uⁿ¼ ðxⁿ,yⁿÞ is an isolated equilibrium point of neural network (8) over a neighborhoodOnDRⁿ of uⁿ such that

r

Eðxⁿ,yⁿÞ ¼0 and

r

Eðx,yÞa0, 8ðx,yÞAOn\fðxⁿ,yⁿÞg. First we show that Eðx,yÞ is a Lyapunov function for uⁿatOn. Since

r

yEðxⁿ,yⁿÞ ¼ A 

r

yCFBðxⁿ,

r

f ðxⁿÞA^TyⁿÞ ¼0, from Lemma 3 and Proposition 1 of[3], we have

r

xCFBðxⁿ,

r

f ðxⁿÞA^TyⁿÞ ¼

r

yCFBðxⁿ,

r

f ðxⁿÞA^TyⁿÞ ¼0:

Moreover, from Proposition 1 of[3], this says CFBðxⁿ,

r

f ðxⁿÞA^TyⁿÞ ¼0:

Then from Eq. (9),

r

xEðxⁿ,yⁿÞ ¼

r

xCFBðxⁿ,

r

f ðxⁿÞA^TyⁿÞ

þ

r

²f ðxⁿÞ 

r

yCFBðxⁿ,

r

f ðxⁿÞA^TyⁿÞ þA^TðAxⁿbÞ ¼ 0,

which implies that A^TðAxⁿbÞ ¼ 0. Because A AR^mn is a full row rank matrix, we must have Axⁿb ¼ 0, which yields

Eðxⁿ,yⁿÞ ¼CFBðxⁿ,

r

f ðxⁿÞA^TyⁿÞ þ¹₂JAxⁿbJ²¼0:

Next, we claim that Eðx,yÞ 4 0, 8ðx,yÞ AOn\fðxⁿ,yⁿÞg. If not, there is an ðx,yÞ AOn\fðxⁿ,yⁿÞg such that Eðx,yÞ ¼ 0, this says that CFBðx,

r

f ðxÞA^TyÞ ¼ 0 and Ax ¼b, then

r

xEðx,yÞ ¼ 0 and

r

yEðx,yÞ ¼ 0. Hence, (x,y) is an equilibrium point of neural net- work (8), contradicting with that uⁿ¼ ðxⁿ,yⁿÞis an isolate equili- brium point. Finally,

dEðxðtÞ,yðtÞÞ

dt ¼ ½

r

ðxðtÞ,yðtÞÞEðxðtÞ,yðtÞÞ^Tð

rr

ðxðtÞ,yðtÞÞEðxðtÞ,yðtÞÞÞ

¼ 

r

_J

r

ðxðtÞ,yðtÞÞEðxðtÞ,yðtÞÞJ²r0:

Therefore, the function Eðx,yÞ is a Lyapunov function for neural network (8) over the set On. Since uⁿ¼ ðxⁿ,yⁿÞ is an isolated equilibrium point of neural network (8), we have

dEðxðtÞ,yðtÞÞ

dt o0, 8ðxðtÞ,yðtÞÞAOn\fðxⁿ,yⁿÞg:

Thus, uⁿis asymptotically stable for neural network (8). &

4. Neural network design with cone projection function In this section, we propose another neural network associated with the cone projection function to solve system (5) for obtain- ing the SOCP solution and study its stability. In fact, from [24, Proposition 3.3], we know that such cone projection onto K has a special formula given as

PKðzÞ ¼ ½l1ðzÞ_þu^ð1Þ_z þ ½l2ðzÞ_þu^ð2Þ_z ,

where ½_þmeans the scalar projection,l1ðzÞ,l2ðzÞ and u^ð1Þz , u^ð2Þz are the spectral values and the associated spectral vectors of z ¼ ðz1,z2Þ ARR^n1, respectively, given by

liðzÞ ¼ z1þ ð1ÞⁱJz2J, u^ðiÞz ¼1

2 1,ð1Þⁱ z2

Jz²J

 

, 8>

<

>:

for i¼ 1,2. The CP function PK(z) has the following property, called projection theorem [21], which is useful in our subsequent analysis.

Property 4.1. Let K be a nonempty closed convex subset ofRⁿ. Then, for each z ARⁿ, PK(z) is the unique vector z A K satisfying ðyzÞ^TðzzÞr0, 8yAK.

Employing the natural residual function with the CP function [19,20], system (5) can be equivalently written as

xPKðx

r

f ðxÞ þ A^TyÞ ¼ 0, Axb ¼ 0,

(

ð11Þ

where x ¼ ðx1, . . . ,xNÞ^TARⁿ¹    R^nNwith xi¼ ðxi1,xi2, . . . ,xin_iÞ^T, i ¼ 1, . . . ,N, and PKðxÞ ¼ ½PKðx1Þ, . . . ,PKðxNÞ^T.

Based on the equivalent formulation in (11) and employing the ideas for networks used in[12,13], we consider the second neural network for solving the SOCP, with the following dynamic equations:

d dt

x y !

¼

r

^{x þ PKðx}

r

f ðxÞ þA^TyÞ

Ax þ b

!

, ð12Þ

where

r

is a positive scaling factor. The dynamic can be realized by a recurrent neural network with the cone projection function as shown inFig. 2. The circuit for the neural network realization requires n þ m integrators, n processors for

r

f ðxÞ, N processors for cone projection mapping PK, 2mn connection weights and some summers. Compared with the first neural network in (8), the second neural network (12) dose not require to calculate

r

²f ðxÞ, resulting in lower model complexity.

To analyze the stability of the neural network in Eq. (12), we first give three lemmas and one proposition.

Lemma 4.1. Let F(u) be defined as

FðuÞ :¼ Fðx,yÞ :¼ x þ PKðx

r

f ðxÞ þA^TyÞ

Ax þb

!

: ð13Þ

Then, F(u) is semi-smooth. Moreover, F(u) is strongly semi-smooth if

r

²f ðxÞ is locally Lipschitz continuous.

Proof. This is an immediate consequence of[20, Theorem 1]. &

Proposition 4.1. For any initial point u0¼ ðx0,y0Þ where x0:¼ xðt0Þ AK, there exists a unique solution uðtÞ ¼ ðxðtÞ,yðtÞÞ for neural network (12). Moreover, xðtÞ A K.

Proof. For simplicity, we assume K ¼ Kⁿ. The analysis can be carried over to the general case where K is the Cartesian product of second-order cones. From Lemma 4.1, FðuÞ :¼ Fðx,yÞ is semi- smooth and Lipschitz continuous. Thus, there exists a unique solution uðtÞ ¼ ðxðtÞ,yðtÞÞ for neural network (12). Therefore, it remains to show that xðtÞ A Kⁿ. For convenience, we denote xðtÞ :¼ ðx1ðtÞ,x2ðtÞÞ ARR^n1. To complete the proof, we need to verify two things: (i) x1ðtÞ Z 0 and (ii) Jx²ðtÞJrx¹ðtÞ. First, from (12), we have

dx

dtþ

r

xðtÞ ¼

r

PKðx

r

f ðxÞ þ A^TyÞ:

The solution of the first-order ordinary differential equation above is

xðtÞ ¼ e^rðtt0Þxðt0Þ þ

r

e^rt Z t

t0

e^rsPKðx

r

f ðxÞ þ A^TyÞds:

If we let xðt0Þ:¼ ðx1ðt0Þ,x2ðt0ÞÞ ARR^n1 and denote zðtÞ :¼ ðz1ðt0Þ,z2ðt0ÞÞas the term PKðxð

r

f ðxÞA^TyÞÞ, which leads to x1ðtÞ ¼ e^rðtt0Þx1ðt0Þ þ

r

e^rt

Z t t₀

e^rsz1ðsÞds,

x2ðtÞ ¼ e^rðtt0Þx2ðt0Þ þ

r

e^rt Z t

t0

e^rsz2ðsÞds:

Due to both x0ðtÞ and z(t) belong to Kⁿ, there have x1ðt0Þ Z0, Jx²ðt0ÞJrx¹ðt0Þ and z1ðtÞ Z0, Jz²ðtÞJrz¹ðtÞ. Therefore, x1ðtÞ Z0 since both terms in the right-hand side are nonnegative.

In addition,

Jx²ðtÞJre^rðtt0ÞJx²ðt0ÞJþ

r

e^rt Z t

t₀

e^rsJz²ðsÞJds

re^rðtt0Þx1ðt0Þ þ

r

e^rt Zt

t0

e^rsz1ðsÞds ¼ x1ðtÞ,

which implies that xðtÞ A Kⁿ. &

Lemma 4.2. Let H(u) be defined as

HðuÞ :¼ Hðx,yÞ :¼

r

f ðxÞA^Ty Axb

!

: ð14Þ

Then, H is a monotone function if f is a convex function. Moreover,

r

HðuÞ is positive semi-definite if and only if

r

²f ðxÞ is positive semi- definite.

Proof. Let u ¼ ðx,yÞ and ~u ¼ ð ~x, ~yÞ. Then, the monotonicity of H holds since

ðu ~uÞ^TðHðuÞHð ~uÞÞ ¼ ðx ~xÞ^Tð

r

f ðxÞ

r

f ð ~xÞÞðx ~xÞ^TðA^Tðy ~yÞÞ þ ðy ~yÞ^TðAðx ~xÞÞ ¼ ðx ~xÞ^Tð

r

f ðxÞ

r

f ð ~xÞÞZ 0,

where the last inequality is due to the convexity of f(x), see[22, Theorem 3.4.5]. Furthermore, we observe that

r

HðuÞ ¼

r

²f ðxÞ A^T

A 0

" #

:

Thus, we have

u^T

r

HðuÞu ¼ ½x^T y^T

r

²f ðxÞ A^T

A 0

" #

x y

" #

¼x^T

r

²f ðxÞx,

which indicates that the positive semi-definiteness of

r

HðuÞ is equivalent to the positive semi-definiteness of

r

²f ðxÞ. &

Lemma 4.3. Let F(u) and H(u) be defined as in (13) and (14), respectively. Also, let uⁿ¼ ðxⁿ,yⁿÞbe an equilibrium point of neural network (12) with xⁿbeing an optimal solution of SOCP. Then, the following inequalities hold:

ðFðuÞ þuuⁿÞ^TðFðuÞHðuÞÞ Z0: ð15Þ

Proof. First, we denotel:¼

r

f ðxÞA^Ty. Then, we obtain ðFðuÞ þuuⁿÞ^TðFðuÞHðuÞÞ

Fig. 2. Block diagram of the proposed neural network with CP function.

¼ x þ PKðxlÞ þ ðxxⁿÞ ðAx þ bÞ þ ðyyⁿÞ

" #T

xPKðxlÞl ðAxbÞðAxbÞ

" #

¼ xⁿþPKðxlÞ ðAx þ bÞ þ ðyyⁿÞ

" #T

ðxlÞPKðxlÞ 0

 

¼ ðxⁿPKðxlÞÞ^TððxlÞPKðxlÞÞ:

Since xⁿAK, applyingProperty 4.1gives ðxⁿPKðxlÞÞ^TððxlÞPKðxlÞÞr0:

Thus, inequality (15) is proved. &

We now investigate the stability and convergence issues of neural network (12). First, we analyze the behavior of the solution trajectory of neural network (12) including existence and con- vergence. We then establish two kinds of stability for an isolated equilibrium point.

We know that every solution uⁿ to SOCP is an equilibrium point of neural network (12). If further uⁿis an isolated equili- brium point of neural network (12), we show that uⁿis Lyapunov stable.

Theorem 4.1. If f is convex and twice differentiable, then the solution of neural network (12), with initial point u0¼ ðx0,y0Þwhere x0AK, is Lyapunov stable. Moreover, the solution trajectory of neural network (12) is extendable to the global existence.

Proof. Again, for simplicity, we assume K ¼ Kⁿ. FromProposition 4.1, there exists a unique solution uðtÞ ¼ ðxðtÞ,yðtÞÞ for neural network (12) and xðtÞ A Kⁿ. Let uⁿ¼ ðxⁿ,yⁿÞbe an equilibrium point of neural network (12) with xⁿbeing an optimal solution of SOCP.

We define a Lyapunov function as below:

EðuÞ :¼ Eðx,yÞ :¼ HðuÞ^TFðuÞ¹₂JFðuÞJ²þ¹₂JuuⁿJ², ð16Þ where F(u) and H(u) are given as in (13) and (14), respectively.

From[23, Theorem 3.2], we know that E is continuously differ- entiable with

r

EðuÞ ¼ HðuÞ½

r

HðuÞIFðuÞ þ ðuuⁿÞ:

It is also trivial that EðuⁿÞ ¼0. Then, we have dEðuðtÞÞ

dt ¼

r

EðuðtÞÞ^Tdu

dt ¼ fHðuÞ½

r

HðuÞIFðuÞ þ ðuuⁿÞg^T

r

FðuÞ

¼

r

f½HðuÞ þðuuⁿÞ^TFðuÞ þ JFðuÞJ²FðuÞ^T

r

HðuÞFðuÞg:

Hence, inequality (15) inLemma 4.3implies ðHðuÞ þ uuⁿÞ^TFðuÞrHðuÞ^TðuuⁿÞJFðuÞJ², which yields

dEðuðtÞÞ

dt r

r

fHðuÞ^TðuuⁿÞFðuÞ^T

r

HðuÞFðuÞg

¼

r

fHðuⁿÞ^TðuuⁿÞðHðuÞHðuⁿÞÞ^TðuuⁿÞ

FðuÞ^T

r

HðuÞFðuÞg: ð17Þ

On the other hand, we know that

ðFðuⁿÞ þuⁿuÞ^TðFðuⁿÞHðuⁿÞÞ ¼ ðxP_KðxⁿlⁿÞÞ^TððxⁿlⁿÞP_KðxⁿlⁿÞÞ:

Since x A Kⁿ, applyingProperty 4.1gives ðxPKðxⁿlⁿÞÞ^TððxⁿlⁿÞPKðxⁿlⁿÞÞr0:

Thus, we have ðFðuⁿÞ þuⁿuÞ^TðFðuⁿÞHðuⁿÞÞ Z0. Note that FðuⁿÞ ¼0, we therefore obtain HðuⁿÞ^TðuuⁿÞ^Tr0. Also the mono- tonicity of H implies ðHðuÞHðuⁿÞÞ^TðuuⁿÞr0. In addition, f is convex and twice differentiable if and only if

r

²f ðxÞ is positive semidefinite and hence

r

H is positive semidefinite byLemma 4.2, i.e., the second term FðuÞ^T

r

HðuÞFðuÞr0. The above discussions lead to dEðuðtÞÞ=dtr0.

In order to obtain E(u) is a Lyapunov function and uⁿ is Lyapunov stable, we will show the following inequality:

HðuÞ^TFðuÞ Z JFðuÞJ²: ð18Þ

To see this, we first observe that

JFðuÞJ²þHðuÞ^TFðuÞ ¼ ðxPKðxlÞÞ^TððxlÞPKðxlÞÞ:

Since x A K, applyingProperty 4.1again, there holds ðxPKðxlÞÞ^TððxlÞPKðxlÞÞr0,

which yields the desired inequality (18). By combining Eq. (16) and inequality (18), we have

EðuÞ Z¹₂JFðuÞJ²þ¹₂JuuⁿJ²,

which says EðuÞ 4 0 if uauⁿ. Hence E(u) is indeed a Lyapunov function and uⁿis Lyapunov stable. Moreover, it holds that Eðu0Þ ZEðuÞ Z¹₂JuuⁿJ² for t Z t0, ð19Þ which means the solution trajectory u(t) is bounded. Hence, it can be extended to global existence. &

Theorem 4.2. Let uⁿ¼ ðxⁿ,yⁿÞbe an equilibrium point of (12) with xⁿbeing an optimal solution of SOCP. If f is twice differentiable and

r

²f ðxÞ is positive definite, the solution of neural network (12), with initial point u0¼ ðx0,y0Þwhere x0AK, is globally convergent to uⁿ and has finite convergence time.

Proof. From (19), the level set Lðu0Þ:¼ fu 9 EðuÞrEðu0Þg

is bounded. Then, the Invariant Set Theorem [25] implies the solution trajectory u(t) converges to yas t-1 where yis the largest invariant set in

P¼ u A Lðu0Þ dEðuðtÞÞ dt ¼0









:

We will show that du=dt ¼ 0 if and only if dEðuðtÞÞ=dt ¼ 0 which yields that u(t) converges globally to the equilibrium point uⁿ¼ ðxⁿ,yⁿÞ. Suppose du=dt ¼ 0, then it is clear that dEðuðtÞÞ=dt ¼

r

EðuÞ^Tðdu=dtÞ ¼ 0. Let ^u ¼ ð ^x, ^yÞ AP which says dEð ^uðtÞÞ=dt ¼ 0.

From (17), we know that dEð ^uðtÞÞ

dt r

r

fðHð ^uÞHðuⁿÞÞ^Tð ^uuⁿÞFð ^uÞ^T

r

Hð ^uÞFð ^uÞg:

Both terms inside the big parenthesis are nonpositive as shown in Lemma 4.2, so ðHð ^uÞHðuⁿÞÞ^Tð ^uuⁿÞ ¼0, Fð ^uÞ^T

r

Hð ^uÞFð ^uÞ ¼ 0, and Fð ^uÞ^T

r

Hð ^uÞFð ^uÞ ¼ f ^x þ PKð ^x

r

f ð ^xÞ þ A^TyÞg^ ^T

r

²f ð ^xÞf ^x

þPKð ^x

r

f ð ^xÞ þ A^TyÞg ¼ 0:^

The condition of

r

²f ð ^xÞ being positive definite leads to

 ^x þPKð ^x

r

f ð ^xÞ þA^TyÞ ¼ 0,^

which is equivalent to d ^x=dt ¼ 0. On the other hand, similar to the arguments inLemma 4.2, we have

ð ^uuⁿÞ^TðHð ^uÞHðuⁿÞÞ ¼ ð ^xxⁿÞ^Tð

r

f ð ^xÞ

r

f ðxⁿÞÞ

¼ ð ^xxⁿÞ^T

r

²f ðxsÞð ^xxⁿÞ ¼0,

where xsA½xⁿ, ^x. Again, the condition of

r

²f ðxsÞ being positive definite yields ^x ¼ xⁿ. Hence d ^y=dt ¼ 0 and therefore d ^uðtÞ=dt ¼ 0.

From above, u(t) converges globally to the equilibrium point uⁿ¼ ðxⁿ,yⁿÞ. Moreover, with Theorem 4.1 and following the same arguments as in[12, Theorem 2], the neural network (12) has finite convergence time. &

5. Simulations

To demonstrate the effectiveness of the proposed neural net- works, three illustrative SOCP problems are tested, described as below.

Example 5.1. Consider the nonlinear convex SOCP[20]given by minimize exp ðx1x3Þ þ3ð2x1x2Þ⁴þ

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1 þð3x2þ5x3Þ² q

subject to Ax ¼ b, x A K³K², where

A ¼ 4 6 3 1 0

1 7 5 0 1

 

and b ¼ 1

2

 

:

This problem has an approximate solution xⁿ¼ ½0:2324,

0:07309,0:2206,0:153,0:153^T. We use the proposed neural net- works with the FB and CP functions, respectively, to solve the problem with the trajectories obtained by them shown in Figs. 3 and 4. From the simulation results, we found that both trajectories are globally convergent to xⁿ and the neural network with the CP function converged to xⁿquicker than that with the FB

function. On the other hand, the neural network with the CP function also has lower model complexity than that with the FB function as mentioned in Section 4. Hence, the neural network with the CP function is preferable to the neural network with the FB function when both can globally converge to the optimal solution.

Example 5.2. Consider the following linear SOCP given by minimize x1þx2þx3þx4þx5þx6

subject to Ax ¼ b, x A K³K³, where

A ¼

1 2 0 0 0 1

1 0 0 1 4 0

0 1 1 0 1 0

1 1 0 0 0 0

0 0 1 0 2 0

2 66 66 66 4

3 77 77 77 5

and b ¼ 9 20

6 4 8 2 66 66 66 4

3 77 77 77 5

This problem has an optimal solution xⁿ¼ ½3,1,2,5,3,4^T. Note that, its objective function is convex and the Hessian matrix

r

²f ðxÞ is a zero matrix. Hence, the neural network with the FB function is asymptotically stable from Theorem 3.1 while the neural network with the CP function is Lyapunov stable from Theorem 4.1.Figs. 5 and 6display the trajectories obtained using the neural networks with the FB and CP functions, respectively.

The simulation results show that both trajectories are convergent to xⁿ. Coinciding with above results ofTheorems 3.1and4.1, the neural network with the CP function yields the oscillating trajectory and has longer convergence time than the neural network with the FB function.

Example 5.3. Consider the grasping force optimization problem for the multi-fingered robotic hand[1,4,17]. Its goal is to find the minimum grasping force for moving an object. For the robotic hand with m fingers, the optimization problem can be formulated as minimize ¹₂f^Tf

subject to Gf ¼ fext

Jðfⁱ¹,fi2ÞJr

m

fi3 ði ¼ 1, . . . ,mÞ,

where f ¼ ½f11,f12, . . . ,fm3^T is the grasping force, G the grasping transformation matrix, fextthe time-varying external wrench, and

m

the friction coefficient.

0 0.05 0.1 0.15 0.2

−0.2 0 0.2 0.4 0.6 0.8 1

Time (sec)

Trajectories of x (t)

x1

x2 x3

x4, x5

Fig. 3. Transient behavior of the neural network with FB function inExample 5.1.

0 0.005 0.01 0.015 0.02 0.025 0.03

−1

−0.5 0 0.5 1 1.5 2 2.5 3

Time (sec)

Trajectories of x (t)

x1

x2 x3 x4, x5

Fig. 4. Transient behavior of the neural network with CP function inExample 5.1.

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

0 1 2 3 4 5 6

Time (sec)

Trajectories of x (t)

x1, x5

x2 x3 x4

x6

Fig. 5. Transient behavior of the neural network with FB function inExample 5.2.

Letting ½x_i1,x_i2,x_i3 ¼ ½

m

f_i3,f_i1,f_i2,i ¼ 1, . . . ,m, and x ¼ ½x11,x12, . . . , xm3^T, the problem can be reformulated as a nonlinear convex SOCP. For the three-finger grasp example in[17], the robot hand grasps a polyhedral with the grasp points ½0,1,0^T, ½1,0:5,0^T, and

½0,1,0^T, and the robot hand moves along a vertical circular trajectory of radius r with a constant velocity

n

. We reformulate the example as

minimize ¹₂x^TQx

subject to Ax ¼ b, x A K³K³K³, ð20Þ where Q ¼ diagð1=

m

²,1,1,1=

m

²,1,1,1=

m

²,1,1Þ

A ¼

0 0 1 1=

m

0 0 0 1 0

1=

m

0 0 0 0 1 1=

m

0 0

0 1 0 0 1 0 0 0 1

0 1 0 0 0:5 0 0 0 1

0 0 0 0 1 0 0 0 0

0 0 1 0:5=

m

0 1 0 1 0

2 66 66 66 66 64

3 77 77 77 77 75

and

b ¼

0

fcsinyðtÞ MgfccosyðtÞ

0 0 0 2 66 66 66 66 4

3 77 77 77 77 5 ,

where M is the mass of the polyhedral, g ¼9.8 m/s², fc¼M

n

²=r the centripetal force, t the time, and y¼

n

t=r A ½0,2

p

. Note that problem (20) is a nonlinear convex SOCP and the matrix Q is positive definite. We know fromTheorems 3.1and4.2that both the proposed neural networks are globally convergent to the optimal solution. Under the conditions M¼0.1 kg, r ¼0.2 m,

n

¼0:4

p

m=s, and

m

¼0:6, the time-varying grasping force obtained from the proposed neural networks is shown inFig. 7.

We found that the maximum grasping force occurs at the position y¼

p

(t ¼0.5 s) which corresponds to the maximum downward wrench. The simulation results demonstrate that the neural networks are effective in the SOCP applications.

6. Conclusion

In this paper, we have proposed two neural networks for efficiently solving the SOCP. The first neural network is based on gradient of the merit function derived from the FB function and was shown to be asymptotically stable. The second neural network with the CP function has low model complexity, and has been shown to be Lyapunov stable and converge globally to the SOCP solution under the positive definite condition of Hessian matrix of the objective function. The convergence of the neural networks has been validated with the simulation results of the SOCP examples.

When the second neural network with the CP functions yields oscillating trajectory, we can employ the neural network based on FB function instead, though it has higher model complexity. The proposed neural networks are thus ready for the SOCP applications.

During the reviewing process of this paper, we published another paper[26] which focuses on second-order cone constrained varia- tional inequality problem. Since the KKT conditions of second-order cone programs can be recast as variational inequality problem, the paper [26] indeed deals with a broader class of optimization problems. However, the two neural networks considered therein are different from the two neural networks studied in this paper.

More specifically, the FB method used in [26] is based on the smoothed FB function while the one studied here is based on regular FB function; the CP method in[26]is based on a Lagrangian model which is, even when it reduces to SOCP, not the same as the one investigated here. Due to the essential difference, the assumptions used to establish stability are also different. In view of this, it will be an interesting topic to do numerical comparison among these neural networks for SOCP.

Acknowledgement

The work was supported by National Science Council of Taiwan under the Grant NSC 97-2221-E-214-034.

Appendix

In this appendix, we introduce the Jordan product and its properties used in the neural network with the FB function, which are needed when we write codes for simulations.

0 0.2 0.4 0.6 0.8 1

0 1 2 3 4 5 6

Time (sec)

Trajectories of x (t)

x1, x5

x2 x3 x4

x6

Fig. 6. Transient behavior of the neural network with CP function inExample 5.2.

0 0.2 0.4 0.6 0.8 1

−1

−0.5 0 0.5 1 1.5 2

Time (sec)

Grasping force (N)

f11,f32

f₁₂ f13

f21

f₂₂ f₂₃

f₃₁ f₃₃

Fig. 7. Grasping force obtained by using proposed neural networks inExample 5.3.

For any x ¼ ðx1,x2Þ ARR^n1, their Jordan product is defined as

xJy ¼ ðx^Ty,y1x2þx1y2Þ:

Their sum of square is calculated by x²þy²¼ ðJxJ²þ JyJ²,2x1x2þ2y1y2Þ:

The square root of x is

x¹⁼²¼ s,x2

2s



, s ¼

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1

2ðx1þ

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi x²₁Jx2J² q

Þ r

if x ¼ 0, x¹⁼²¼0 and the determinant of x is det ðxÞ ¼ x²₁Jx²J². Furthermore, a matrix Lxis defined as

Lx¼ x1 x^T₂ x2 x1I

" # ,

and when detðxÞa0, Lxis invertible with L^1_x ¼ 1

det ðxÞ

x1 x^T₂

x2 detðxÞ x₁ I þ_x¹

1x2x^T₂ 2

4

3 5:

Based on the properties of the Jordan product described above, the formulae of

r

xCFBðx,yÞ and

r

yCFBðx,yÞ in neural network (8) are calculated (see[3]) as

r

xCFBðx,yÞ ¼ ðLxL^1_ðx2þy2Þ1=2IÞf_FBðx,yÞ and

r

yCFBðx,yÞ ¼ ðLyL^1_ðx2þy2Þ1=2IÞfFBðx,yÞ:

References

[1] M.S. Lobo, L. Vandenberghe, S. Boyd, H. Lebret, Applications of second-order cone programming, Linear Algebra and its Applications 284 (1) (1998) 193–228.

[2] F. Alizadeh, D. Goldfarb, Second-order cone programming, Mathematical Programming 95 (1) (2003) 3–51.

[3] J.-S. Chen, P. Tseng, An unconstrained smooth minimization reformulation of the second-order cone complementarity problem, Mathematical Program- ming 104 (2005) 293–327.

[4] S.P. Boyd, B. Wegbreit, A Fast computation of optimal contact forces, IEEE Transactions on Robotics 23 (6) (2007) 1117–1132.

[5] S. Boyd, C. Crusius, A. Hansson, Control applications of nonlinear convex programming, Journal of Control Process 8 (5) (1998) 313–324.

[6] D. Bertsimas, D.B. Brown, Constrained stochastic LQC: a tractable approach, IEEE Transactions on Automatic Control 52 (10) (2007) 1826–1841.

[7] D.W. Tank, J.J. Hopfield, Simple neural optimization networks: an A/D converter, signal decision circuit, and a linear programming circuit, IEEE Transactions on Circuits and Systems 33 (5) (1986) 533–541.

[8] M.P. Kennedy, L.O. Chua, A Neural network for nonlinear programming, IEEE Transaction on Circuits and Systems 35 (5) (1988) 554–562.

[9] Y.S. Xia, A new neural network for solving linear and quadratic programming problems, IEEE Transactions on Neural Networks 7 (6) (1996) 1544–1547.

[10] Q. Tao, J.D. Cao, M.S. Xue, H. Qiao, A high performance neural network for solving nonlinear programming problems with hybrid constraints, Physics Letters A 288 (2) (2001) 88–94.

[11] J. Wang, Q. Hu, D. Jiang, A Lagrangian neural network for kinematic control of redundant robot manipulators, IEEE Transactions on Neural Networks 10 (5) (1999) 1123–1132.

[12] Y. Xia, J. Wang, A recurrent neural network for solving nonlinear convex programs subject to linear constraints, IEEE Transactions on Neural Networks 16 (3) (2005) 379–386.

[13] Y. Xia, H. Leung, J. Wang, A projection neural network and its application to constrained optimization problems, IEEE Transactions on Circuits and Sys- tems – Part I 49 (2002) 447–458.

[14] Y. Xia, J. Wang, A recurrent neural network for nonlinear convex optimization subject to nonlinear inequality constraints, IEEE Transactions on Circuits and Systems I: Regular Papers 51 (7) (2004) 1385–1394.

[15] Y. Xia, H. Leung, J. Wang, A general projection neural network for solving monotone variational inequalities and related optimization problems, IEEE Transactions on Neural Networks 15 (2) (2004) 318–328.

[16] X. Mu, S. Liu, Y. Zhang, A neural network algorithm for second-order conic programming, in: Proceedings of the Second International Symposium on Neural Networks, Chongqing, China, Part II, 2005, pp. 718–724.

[17] Y. Xia, J. Wang, L.M. Fok, Grasping-force optimization for multifingered robotic hands using a recurrent neural network, IEEE Transactions on Robotics and Automation 20 (3) (2004) 549–554.

[18] L.Z. Liao, H.D. Qi, A neural network for the linear complementarity problem, Mathematical and Computer Modeling 29 (3) (1999) 9–18.

[19] J.S. Chen, X. Chen, P. Tseng, Analysis of nonsmooth vector-valued function associated with second-order cone, Mathematical Programming 101 (1) (2004) 95–117.

[20] C. Kanzow, I. Ferenczi, M. Fukushima, On the local convergence of semi- smooth Newton methods for linear and nonlinear second-order cone pro- grams without strict complementarity, SIAM Journal on Optimization 20 (2009) 297–320.

[21] D.P. Bertsekas, Nonlinear Programming, Athena Scientific, Belmont, MA, 1995.

[22] J.M. Ortega, W.C. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables, SIAM, Philadelphia, 2000.

[23] M. Fukushima, Equivalent differentiable optimization problems and descent methods for asymmetric variational inequality problems, Mathematical Programming 53 (1) (1992) 99–110.

[24] M. Fukushima, Z.-Q. Luo, P. Tseng, Smoothing functions for second- order-cone complementarity problems, SIAM Journal on Optimization 12 (2002) 436–460.

[25] R. Golden, Mathematical Methods for Neural Network Analysis and Design, The MIT Press, Cambridge, MA, 1996.

[26] J. Sun, J.-S. Chen, C.-H. Ko, Neural networks for solving second-order cone constrained variational inequality problem, Computational Optimization and Applications, in press, doi:10.1007/s10589-010-9359-x.

Chun-Hsu Ko received the Ph.D. degree in Electrical and Control Engineering from National Chiao Tung University, Taiwan, ROC, in 2003. He worked at ITRI in Taiwan in 1994–1998. He is currently an Associate Professor in the Department of Electrical Engineering, I-Shou University, Taiwan. His research interests include neural networks, control, and robotics.

Jein-Shan Chen, an associate professor at National Taiwan Normal University, obtained his Ph.D. degree in mathematics under Prof. Paul Tseng from University of Washington in 2004. His research interest is mainly on continuous optimization. He has published over 50 papers including a few in top journals like Mathema- tical Programming, SIAM Journal on Optimization, etc.

Ching-Yu Yang obtained his Ph.D. degree in mathe- matics from National Taiwan Normal University, Tai- wan, ROC, in 2010. He is currently a Lecturer in the Department of Mathematics, National Taiwan Normal University. His research interest is optimization.