www.elsevier.com/locate/physd
Multistability and convergence in delayed neural networks
Chang-Yuan Cheng
a, Kuang-Hui Lin
b, Chih-Wen Shih
b,∗aDepartment of Applied Mathematics, National Pingtung University of Education, Pingtung, 900, Taiwan, ROC bDepartment of Applied Mathematics, National Chiao Tung University, Hsinchu, 300, Taiwan, ROC
Received 24 December 2005; received in revised form 25 August 2006; accepted 5 October 2006 Available online 27 November 2006
Communicated by A. Doelman
Abstract
We present the existence of 2nstable stationary solutions for a general n-dimensional delayed neural networks with several classes of activation functions. The theory is obtained through formulating parameter conditions motivated by a geometrical observation. Positively invariant regions for the flows generated by the system and basins of attraction for these stationary solutions are established. The theory is also extended to the existence of 2nlimit cycles for the n-dimensional delayed neural networks with time-periodic inputs. It is further confirmed that quasiconvergence is generic for the networks through justifying the strongly order preserving property as the self-feedback time lags are small for the neurons with negative self-connection weights. Our theory on existence of multiple equilibria is then incorporated into this quasiconvergence for the network. Four numerical simulations are presented to illustrate our theory.
c
2006 Elsevier B.V. All rights reserved.
Keywords:Multistability; Neural networks; Monotone dynamics; Convergence
1. Introduction
Existence of many equilibria is a necessary feature in the applications of neural networks to associative memory storage or pattern recognition [1–4]. The notion of “multistability” of a neural network describes coexistence of multiple stable patterns such as equilibria or periodic orbits. Recently, further application potentials of multistability have been found in decision making, digital selection or analogy amplification [5]. “Quasiconvergence” for a system refers to that every solution tends to the set of stationary solutions, while “convergence” means that every solution tends to a single stationary solution, as time tends to infinity.
In this presentation, we address multistability and quasicon-vergence for a general delayed neural network:
dxi(t) dt = −µixi(t) + n X j =1 αi jgj(xj(t)) + n X j =1 βi jgj(xj(t − τi j)) + Ii, (1.1)
∗Corresponding author. Tel.: +886 3 5722088; fax: +886 3 5724679.
E-mail address:[email protected](C.-W. Shih).
where i = 1, . . . , n; µi > 0; αi j, βi j are connection weights
from neuron j to neuron i ; gj(·) are activation functions;
0 ≤ τi j ≤ τ are time lags; Ii stands for an independent
bias current source. System(1.1)reduces to the classical and delayed Hopfield neural networks [3,6], asβi j =0 andαi j =0
for all i, j, respectively. It also represents the cellular neural networks (CNN) without delays [7] and with delays [8]. Indeed, a CNN system built in a multi-dimensional coupling fashion can always be rewritten in a one-dimensional coupling form, by renaming the indices [9]. Such an arrangement, however, suppresses the local connection representation.
In electronic implementation, time delays of neural network systems are unavoidable due to axonal conduction times, distances of interneurons and the finite switching speeds of amplifiers. The dynamics for differential equations with delays can be rather complicated. Although the stationary equations are identical for system (1.1) without delay (τi j = 0 for
all i, j) and with delay (τi j > 0), the stability for the
equilibrium points and dynamical behaviors of the systems can be very different. There have been papers [10–15] exploring the effects of delays in differential equations and neural network systems. For system (1.1), the theory of unique equilibrium and global convergence to the equilibrium has been studied
0167-2789/$ - see front matter c 2006 Elsevier B.V. All rights reserved.
extensively in [8,16–25]. These studies indicate a coincidence of dynamics between the systems with delays and without delays. This presentation moves the investigation in this direction by establishing the existence of multiple stationary solutions for system (1.1). More specifically, we construct 2n stable stationary solutions for a general n-dimensional delayed neural network with several classes of activation functions. The theory is obtained through formulating parameter conditions based on a geometrical setting. We first derive conditions for the existence of 3n equilibria for (1.1) with sigmoidal activation functions and saturated activation functions. Some regions containing these stationary solutions are shown to be positively invariant under the flows generated by (1.1) and the basins of attraction for these stationary solutions are also estimated. In fact, through a further subtle estimate, it can be justified that the basins of attraction are at least as large as the positively invariant sets. The theory is also extended to confirm the existence of 2n limit cycles for system (1.1)
with time-periodic inputs. We further discuss the property of strongly order preserving, hence quasiconvergent behaviors for
(1.1). The dynamics scenario for system(1.1)is thus composed of multiple equilibria and quasiconvergence (or convergence) almost everywhere. Our investigations also illustrate different criteria and distinct dynamical behaviors between (1.1) with smooth sigmoidal activation functions and(1.1)with saturated activation functions.
The existence of multiple equilibria and their attractive domains for (1.1) with the standard activation function have been studied in [26]. The result therein strongly relies on the piecewise linearity and saturations of the standard activation function as well as subsequent partition of the phase space. Our geometrical approach can be applied to(1.1)with general sigmoidal activation functions. In addition, larger positively invariant sets and basins of attraction are established. Moreover, the criteria in our theory are weaker than those in [26]. The approach in this presentation is an extension from [27] which mainly treats multistability for the Hopfield neural networks with smooth sigmoidal activation functions.
The monotone dynamic property for the classical neural networks (without delays) was first exploited by Hirsch [28]. Such a property was then extended to the two-neuron and the n-neuron delayed Hopfield neural networks in [12] and [13] respectively; see also [29] for a comprehensive overview. The investigations in [12,13] are mainly concerned with global attractivity of a single equilibrium and the effect of delays upon such a dynamical scenario. Our study aims at incorporating the existence of multiple equilibria into the monotone dynamics and the quasiconvergence for(1.1).
The remaining part of this presentation is organized as follows. In Section 2, we consider two classes of activation functions which are commonly employed in neural networks. We then derive conditions for the existence of 3n equilibria for the networks. In Section3, we show that, with additional condition, there are 2n regions in Rn which are positively invariant under the flow generated by system (1.1). Each of these regions contains one equilibrium out of those 3n equilibria. Subsequently, it is argued that these 2n equilibria
are asymptotically stable. Existence of multiple stable periodic orbits for system (1.1) with periodic inputs is demonstrated in Section 4. We discuss strongly order preserving property and quasiconvergence for system(1.1)in Section5. Finally, in Section6, we present four numerical simulations to illustrate the present theory and distinct dynamical behaviors for different activation functions.
2. Activation functions and multiple equilibria
Existence and stability of stationary patterns for neural networks certainly depend on the characteristics of activation functions. We shall consider the following two classes of activation functions gifor(1.1):
class A : gi ∈C2, ui < gi(ξ) < vi, gi0(ξ) > 0, (ξ − σi)gi00(ξ) < 0, for all ξ ∈ R, lim ξ →+∞gi(ξ) = vi, lim ξ →−∞gi(ξ) = ui; class B : gi ∈C, gi(ξ) = ui if − ∞< ξ < pi, gi(ξ) if pi ≤ξ ≤ qi, vi if qi < ξ < ∞,
where, ui,vi, pi, qi andσiare constants with ui < vi, pi < qi,
and gi(·), i = 1, . . . , n, are C1 increasing functions. Class A contains general bounded smooth sigmoidal functions, and class B consists of nondecreasing functions with saturations, including the piecewise linear functions with two corner points at pi, qi:
gi(ξ) = ui+
vi−ui
qi−pi(ξ − pi);
(2.1)
and, in particular, the standard activation function for the CNN:
gi(ξ) = gs(ξ) :=
1
2(|ξ + 1| − |ξ − 1|), i = 1, . . . , n. (2.2) Typical configurations of these functions are depicted inFigs. 1
and2(a). Notably, in practical implementation, the transition from the linear regime to the saturated regime in the standard activation function is smooth. Thus, the theory developed for the dynamics of(1.1) should also be valid for the activation functions which are smooth at ξ = ±1, as demonstrated in
Fig. 2(b). Our investigations have provided theoretical basis for all these activation functions. In Section5, we will see some distinct dynamics between (1.1)with activation functions of class A and(1.1)with the ones of class B.
Let us review some basic notion of delayed differential equations. We set τ = max1≤i, j≤nτi j. The initial condition
for(1.1)is xi(θ) = φi(θ), −τ ≤ θ ≤ 0, i = 1, . . . , n, with
φ = (φ1, . . . , φn) ∈ C([−τ, 0], Rn). We define the norm of
φ as k φ k= max1≤i ≤n{supθ∈[−τ, 0]|φi(θ)|}. Let ` > 0. For
x(·) = (x1(·), . . . , xn(·)) ∈ C([−τ, `], Rn), and t ∈ [0, `], we
define
Fig. 1. The configurations of (a) typical smooth sigmoidal activation functions in class A and (b) saturated activation functions in class B.
Fig. 2. The graphs for (a) the standard activation function gs(ξ) =12(|ξ + 1| − |ξ − 1|), (b) saturated activation functions with smooth corners.
Let us denote ˜F = ( ˜F1, . . . , ˜Fn), where ˜Fi is the right hand
side of system(1.1), ˜ Fi(xt) = −µixi(t) + n X j =1 αi jgj(xj(t)) + n X j =1 βi jgj(xj(t − τi j)) + Ii.
A function x(·) is called a solution of (1.1) on [−τ, `) if x(·) ∈ C([−τ, `), Rn), and xt defined as (2.3) lies in the
domain of ˜F and satisfies (1.1) for t ∈ [0, `). For a given φ ∈ C([−τ, 0], Rn), let us denote by x(t; φ) the solution of
(1.1)with x(θ; φ) = φ(θ), for θ ∈ [−τ, 0]. Notably, the stationary equation for(1.1)is
Fi(x) := −µixi + n X j =1 (αi j+βi j)gj(xj) + Ii =0, i =1, . . . , n. (2.4) Next, we shall consider the above activation functions and formulate sufficient conditions for the existence of multiple stationary solutions for (1.1). Our approach is based on a geometrical observation. The first condition for (1.1) with activation functions in classes A, B is, respectively,
(H1A): 0 = inf ξ∈Rg 0 i(ξ) < µi αi i +βi i < max ξ∈Rg 0 i(ξ)(= gi0(σi)), i = 1, . . . , n (H1B): (αi i +βi i) max ξ∈Rg 0 i(ξ) > µi, i = 1, . . . , n.
Condition (H1B) reduces to (αi i + βi i)qvii−p−uii > µi, if
piecewise linear activation functions (2.1) are adopted, and reduces to
αi i+βi i > µi, i = 1, . . . , n, (2.5)
if the standard activation function gs(·) in (2.2)is employed, with pi =ui = −1, qi =vi =1. We define, for i = 1, . . . , n,
ˆ fi(ξ) = −µiξ + (αi i +βi i)gi(ξ) + ki+, ˇ fi(ξ) = −µiξ + (αi i +βi i)gi(ξ) + ki−, (2.6) where k+i = Pn j =1, j6=iρj(|αi j| + |βi j|) + Ii, ki− = −Pn
j =1, j6=iρj(|αi j| + |βi j|) + Ii, andρj :=max{|uj|, |vj|}. It
follows that ˇfi(xi) ≤ Fi(x) ≤ ˆfi(xi), for all x = (x1, . . . , xn)
and i = 1, . . . , n. We introduce a family of single neuron equations, for i = 1, . . . , n dξ dt = fi(ξ) := −µiξ + (αi i+βi i)gi(ξ) + Ji, ξ ∈ R, k− i ≤ Ji ≤k + i .
Proposition 2.1. There exist two points ˜pi and ˜qi with ˜pi <
σi < ˜qi (resp. ˜pi ≥ pi and ˜qi ≤ qi) such that fi0( ˜pi) =
fi0( ˜qi) = 0, i = 1, . . . , n, under condition (H1A) (resp. (H1B)),
for activation functions of classA (resp. B).
Proof. We only prove for class A. For each i , since f0 i(ξ) = −µi+(αi i+βi i)g0 i(ξ), we have f 0 i(ξ) = 0 if and only if g 0 i(ξ) =
µi/(αi i +βi i). The graph of function gi0(ξ) is concave down
Fig. 3. (a) The graph of activation function giin class A, (b) configurations of
functions ˆfiand ˇfi.
Hence, since each gi0is continuous, if 0 = inf ξ∈Rg 0 i(ξ) < µi αi i +βi i < max ξ∈Rg 0 i(ξ) (= g 0 i(σi)), i = 1, . . . , n,
there exist two points ˜pi, ˜qi, with ˜pi < σi < ˜qi, such
that gi0( ˜pi) = gi0( ˜qi) = µi/(αi i +βi i). This completes the
proof.
For (1.1) with piecewise linear activation functions, fi
attains its local minimum at ˜pi = pi, and local maximum at
˜
qi =qi, under assumption(H1B). In particular, for the standard
activation function gs, ˜pi = −1, ˜qi = 1, i = 1, . . . , n. A
consequence ofProposition 2.1is that fi is strictly increasing
on(−∞, ˜pi], decreasing on [ ˜qi, ∞), under condition (H1∗).
Note that condition(H1∗), ∗ = A, B, implies αi i +βi i > 0 for
each i = 1, . . . , n, since µi is already assumed positive.
We consider the second parameter condition which is used to establish existence of multiple equilibria for(1.1):
(H2): fˆi( ˜pi) < 0, fˇi( ˜qi) > 0, i = 1, . . . , n.
The configuration that motivates(H2) is depicted inFigs. 3and
4. Under assumptions(H1∗) and (H2), ∗ = A, B, there exist
points ˆai, ˆbi, ˆci with ˆai < ˆbi < ˆci such that ˆfi(ˆai) = ˆfi( ˆbi) =
ˆ
fi(ˆci) = 0 as well as points ˇai, ˇbi, ˇci with ˇai < ˇbi < ˇci, such
that ˇfi(ˇai) = ˇfi( ˇbi) = ˇfi(ˇci) = 0.
Theorem 2.2. There exist 3n equilibria for system(1.1)with activation functions of class ∗, ∗ = A, B, under conditions (H1∗) and (H2).
Proof. We only prove the case of class A. The equilibria of system (1.1) are roots of (2.4). Conditions (H1A) and (H2)
induce a configuration depicted in Fig. 3. Accordingly, there are 3ndisjoint closed regions in Rn, namely,
w= {(x
1, . . . , xn) ∈ Rn|xi ∈Ωiwi},
w =(w1, . . . , wn), wi =“l”, “m”, “r”,
(2.7)
where Ωil = [ ˇai, ˆai], Ωim = [ ˆbi, ˇbi], Ωir = [ ˇci, ˆci]
are intervals. Herein, “l”, “m”, “r” mean respectively “left”, “middle” and “right”. Letwbe one of these regions. For any given ˜x =( ˜x1, . . . , ˜xn) ∈ w, we solve for xi in
hi(xi) := −µixi+(αi i +βi i)gi(xi) + n X j =1, j6=i (αi j+βi j)gj( ˜xj) + Ii =0, (2.8)
i = 1, . . . , n. Note that hi is a vertical shift and lies between
ˆ
fi and ˇfi, due to (2.6). Accordingly, one can always find
three solutions to (2.8), which lie in regions Ωil, Ωim, Ωir
respectively, for each i . We define a mapping Hw:w→ w
by Hw(˜x) = x = (x1, . . . , xn) where xi is the solution of(2.8)
lying in Ωiwi. The mapping Hwas defined is continuous, since
gi is continuous. It follows from Brouwer’s fixed point theorem
that there exists one fixed point ¯x =( ¯x1, . . . , ¯xn) of Hwinw,
which is also a zero of F in(2.4). Consequently, there exist 3n equilibria for system(1.1)and each of them lies in one of the 3nregionsw.
3. Stability of equilibria and basins of attraction
In this section, we first establish some positively invariant sets for system(1.1)and investigate stability of the equilibrium in each invariant set. As a result, we also obtain a basin of attraction for each of the asymptotically stable equilibria.
We consider the following 2nsubsets of C([−τ, 0], Rn). Let
w =(w1, . . . , wn) with wi =“l” or “r”, and set
˜
3w = {ϕ = (ϕ1, . . . , ϕn) | ϕi ∈ ˜Λli
ifwi =“l”, ϕi ∈ ˜Λri ifwi =“r”}, (3.1)
where ˜Λli = {ϕi ∈ C([−τ, 0], R) | ϕi(θ) < ˆbi for allθ ∈
[−τ, 0]}, ˜Λr
i = {ϕi ∈ C([−τ, 0], R) | ϕi(θ) > ˇbifor allθ ∈
[−τ, 0]}.
Theorem 3.1. Assume that(H1∗), (H2), ∗ = A, B, and βi i >
0, i = 1, . . . , n, then each ˜3w is positively invariant under the solution flow generated by system (1.1) with activation functions of class ∗.
Proof. We only prove the A case. Let ˜3wbe a subset defined in
(3.1). Consider any initial conditionφ = (φ1, . . . , φn) ∈ ˜3 w
; there exists a sufficiently small constant ε0 > 0 such that
φi(θ) ≥ ˇbi + ε0 for all θ ∈ [−τ, 0], if wi = “r”, and
φi(θ) ≤ ˆbi −ε0 for allθ ∈ [−τ, 0], if wi = “l”. We claim
that the solution x(t; φ) remains in ˜3w for all t ≥ 0. If this is not true, there exists a component of x(t; φ) which is the first one (or one of the first ones) decreasing across the value ˇbi+ε0
or increasing across the value ˆbi −ε0; i.e., there exists some
i ∈ {1, . . . , n} and t1 > 0 such that either xi(t1) = ˇbi +ε0,
(dxi/dt)(t1) ≤ 0, and xi(t) > ˇbi +ε0 for −τ ≤ t < t1 or
xi(t1) = ˆbi −ε0,(dxi/dt)(t1) ≥ 0 and xi(t) < ˆbi −ε0 for
−τ ≤ t < t1. For the first case, we derive from(1.1)that dxi dt (t1) = −µi( ˇbi+ε0) + αi igi( ˇbi+ε0) + βi igi(xi(t1−τi i)) + n X j =1, j6=i αi jgj(xj(t1)) + n X j =1, j6=i βi jgj(xj(t1−τi j)) + Ii ≤0. (3.2)
On the other hand,
−µi( ˇbi+ε0) + αi igi( ˇbi +ε0) + βi igi(xi(t1−τi i)) + n X j =1, j6=i αi jgj(xj(t1)) + n X j =1, j6=i βi jgj(xj(t1−τi j)) + Ii ≥ −µi( ˇbi+ε0) + (αi i +βi i)gi( ˇbi+ε0) − n X j =1, j6=i ρj(|αi j| + |βi j|) + Ii = ˇfi( ˇbi +ε0) > 0, (3.3) due to (H2), βi i > 0, |gj(·)| ≤ ρj, and gi(xi(t1−τi i)) ≥
gi( ˇbi +ε0), from the monotonicity of gi and the definition of
t1. This yields a contradiction to(3.2). Hence, xi(t) ≥ ˇbi +ε0
for all t > 0. Similar arguments can be employed to show that xi(t) ≤ ˆbi −ε0, for all t > 0 for the situation that
xi(t1) = ˆbi − ε0 and (dxi/dt)(t1) ≥ 0. Therefore, ˜3 w
is positively invariant under the flow generated by system(1.1). The proof is completed.
Next, we consider the following criterion concerning stability of the equilibria for the system with activation functions in class A. Let ηj, j = 1, . . . , n, be real numbers
satisfying
max{gj0(ξ)|ξ = ˇcj, ˆaj}< ηj < min{gj0(ξ)|ξ = ˜pi, ˜qi}.
Consider (H3): µi > n X j =1 ηj(|αi j| + |βi j|), i = 1, . . . , n.
For activation functions gj(·) in class A, we define dj and ¯dj
as
dj =min{ξ | gj0(ξ) = ηj}, dj =max{ξ | gj0(ξ) = ηj}. (3.4) Then dj > ˆaj, dj < ˇcj. For the activation functions gj in class
B, gi in(2.1), and gsin(2.2), we define, respectively,
dj = ˜pj, dj = ˜qj;dj = pj, dj =qj;dj = −1, dj =1. (3.5)
We consider the following 2n subsets of C([−τ, 0], Rn). Let w =(w1, . . . , wn) with wi =“l” or “r”, and set
3w= {ϕ = (ϕ 1, . . . , ϕn) | ϕi ∈Λil ifwi =“l”, ϕi ∈Λirifwi =“r”}, (3.6) where Λli = {ϕi ∈ C([−τ, 0], R) | ϕi(θ) ≤ di, ∀θ ∈ [−τ, 0]}, Λr i = {ϕi ∈ C([−τ, 0], R) | ϕi(θ) ≥ di, ∀θ ∈
[−τ, 0]}. In the following, we will derive that each of these 2n subsets3w of C([−τ, 0], Rn) lies in the basin of attraction for the respective equilibrium and justify that these 2n equilibria are exponentially stable.
Theorem 3.2. Under conditions(H1A), (H2), (H3), and βi i >
0, i = 1, . . . , n, there exist 2n exponentially stable equilibria for system (1.1) with activation functions of class A. The same assertion holds for activation functions of classB, under conditions(H1B), (H2).
Proof. We only prove the case of class A. Let3wbe a subset defined in (3.6) and ¯x be an equilibrium lying in 3w. For each i = 1, . . . , n, we consider the single-variable function Gi(ζ ) = µi −ζ − Pnj =1ηj|αi j| −Pnj =1ηj|βi j|eζ τi j. Then,
(H3) implies Gi(0) > 0, and there exists a constant λ > 0 such
that Gi(λ) > 0, for all i = 1, . . . , n, due to continuity of Gi.
Let x(t) = x(t; φ) be the solution to system(1.1)with initial condition φ ∈ 3w. With translation y(t) = x(t) − ¯x, system
(1.1)becomes dyi(t) dt = −µiyi(t) + n X j =1 αi j[gj(xj(t)) − gj(xj)] + n X j =1 βi j[gj(xj(t − τi j)) − gj(xj)], (3.7)
where y =(y1, . . . , yn). Now, consider functions zi(·) defined
by zi(t) = eλt|yi(t)|, i = 1, . . . , n. Let δ > 1 and let
K = max1≤i ≤n{supθ∈[−τ,0]|xi(θ) − ¯xi|} > 0. It follows that
zi(t) < K δ, for t ∈ [−τ, 0] and i = 1, . . . , n. We shall justify
that
zi(t) < K δ, for all t > 0, i = 1, . . . , n. (3.8)
Suppose (3.8)does not hold, then there is a k ∈ {1, . . . , n}
and a t1 > 0 for the first time such that zi(t) ≤ K δ, t ∈
[−τ, t1], i = 1, . . . , n, i 6= k, zk(t) ≤ K δ, t ∈ [−τ, t1), and zk(t1) = K δ, with ˙zk(t1) ≥ 0. Note that |yk(t)| and zk(t)
are differentiable at t = t1, since zk(t1) = K δ > 0 implies
yk(t1) 6= 0. From(3.7), we compute that
d dt|yk(t1)| ≤ −µk|yk(t1)| + n X j =1 |αk jg0j(ξj)yj(t1)| + n X j =1 |βk jg0j(ςj)yj(t1−τk j)|,
for some ξj between xj(t1) and ¯xj as well as ςj between
xj(t1−τk j) and ¯xj. Hence, dzk(t1) dt ≤ λe λt1|y k(t1)| + eλt1 " −µk|yk(t1)| + n X j =1 |αk jg0j(ξj)yj(t1)| + n X j =1 |βk jg0j(ςj)yj(t1−τk j)| # =λzk(t1) − µkzk(t1) + n X j =1 |αk j|g0j(ξj)zj(t1) + n X j =1 |βk j|g0j(ςj)eλτk jzj(t1−τk j) ≤ −(µk−λ)zk(t1) + n X j =1 |αk j|ηjzj(t1) + n X j =1 |βk j|ηjeλτk j[ sup θ∈[t1−τ,t1] zj(θ)].
Herein, the positive invariance property of3w can be verified using the same treatment as the proof of Theorem 3.1, under condition βi i > 0, i = 1, . . . , n, for activation functions in
class A. Due to Gk(λ) > 0, we obtain a contradiction that
0 ≤ dzk(t1) dt ≤ − ( µk−λ − n X j =1 ηj|αk j| − n X j =1 ηj|βk j|eλτk j ) Kδ < 0.
Hence assertion (3.8) holds and zi(t) ≤ K for all t > 0,
i = 1, . . . , n, by taking δ → 1+. We thus obtain |x i(t) −
¯
xi| ≤ e−λtmax1≤ j ≤n{supθ∈[−τ,0]|xj(θ) − ¯xj|}, for t > 0 and
i =1, . . . , n. Therefore, x(t) converges to ¯x exponentially. This completes the proof.
In the above theorem, we have imposed a restriction: βi i > 0, i = 1, . . . , n (positive self-feedback delays) for the
activation functions in class A. The situation is different for the activation functions in class B, thanks to the zero slopes of these functions in the saturated parts. In addition, for the piecewise linear functions gi in (2.1), since the slopes νi := (vi −
ui)/(qi − pi) in the middle parts are fixed, there cannot exist
parametersµi, αi j, βi j, andηi satisfying both(H3) and (H1B).
Indeed, a contradiction arises inµi > νi(Pnj =1|αi j| + |βi j|)
versusνi(αi i +βi i) > µi. Thus, the definition of3w for the
activation functions in B and the standard activation function gs are as indicated in(3.5)and every3w lies in the saturated parts corresponding to the activation functions.
Corollary 3.3. Each of these 2nsubsets3wofC([−τ, 0], Rn),
defined in(3.6), lies in the basin of attraction for the unique equilibrium in3w, under the assumptions of Theorem3.2. Corollary 3.4. Under conditionαi i+βi i−Pnj =1, j6=i(|αi j| +
|βi j|) − |Ii| > µi, i = 1, . . . , n, there exist 2n exponentially stable equilibria for(1.1)with activation function gsin(2.2). Proof. The condition yields(2.5), and(H2) with ˜pi = −1 and
˜
qi =1 for all i = 1, . . . , n. The assertion hence holds.
Remark 3.1. (i) There exists a globally attracting set for system(1.1), according to [30]. Therefore, every solution of system(1.1)is bounded in forward time.
(ii) System(1.1)withµi =1, i = 1, . . . , n, and the standard
activation function gs was investigated in [26]. It was proved therein that under condition
αi i − n X j =1, j6=i |αi j| − n X j =1 |βi j| − |Ii|> 1, i =1, . . . , n, (3.9) there exist exactly 2n exponentially stable isolated equilibria. It is obvious that our condition inCorollary 3.4
is weaker than condition(3.9). In addition, it was shown that the set {x | x = (x1, . . . , xn), xi < −1 or xi > 1}
is positively invariant. OurTheorem 3.1 has exploited a larger positively invariant set ˜3w. The computations in deriving the results in [26] heavily depend on the saturation of the activation functions. Restated, as xj(t − τi j) lies in
{ξ < −1} or {ξ > 1}, gs(xj(t − τi j)) is either −1 or 1, and thus the delays in(1.1)do not have any actual effect in these regions. The numerical simulations therein thus dealt with ordinary differential equations. As mentioned in Section2, the transition from the linear regime to the saturated regime in the standard activation function is smooth in a practical situation. Our theory is based on a geometrical observation and has been established with these practical considerations being taken into account. (iii) It can be further justified that the basins of attraction
for the equilibria are actually larger than 3w, through additional derivations and estimates. In fact, they are at least as large as the positively invariant sets ˜3w. The justification can be found in [31]. We have shown
(Theorem 3.2) that the solutions lying entirely in 3w converge exponentially to the respective equilibrium in 3w. However, the convergence for the solutions lying
entirely in ˜3wmay not be of exponential rate. 4. Periodic orbits for systems with periodic inputs
In this section, we study the periodic solutions of the delayed neural networks with periodic input:
dxi(t) dt = −µixi(t) + n X j =1 αi jgj(xj(t)) + n X j =1 βi jgj(xj(t − τi j)) + Ji(t), (4.1)
where i = 1, . . . , n, each Ji : R+ −→ R is a continuous
function of period T , i.e., Ji(t + T ) = Ji(t) for all t ≥ 0. There
have been investigations on the existence of a single periodic solution for system(4.1), cf. [18]. The results in this direction of study can be achieved by constructing a suitable Lyapunov functional or Poincar´e mapping. In this section, we establish existence of multiple stable periodic solutions via constructing suitable Poincar´e mapping.
Theorem 4.1. Under conditions(H1A), (H2), (H3), and βi i >
0, i = 1, . . . , n, there exist 2n exponentially stable T -period solutions for system(4.1)with activation functions of classA. The same conclusion holds for(4.1)with activation functions of classB, under conditions(H1B), (H2).
Proof. Recall the notations in Section 2: x(t; φ) = (x1(t; φ), . . . , xn(t; φ)), the solution of (4.1)with x(θ; φ) =
φ(θ), θ ∈ [−τ, 0], and xt(θ; φ) = x(t +θ; φ), θ ∈ [−τ, 0], t ≥
0. Consider ϕ, ψ ∈ 3w, for some w = (w1, . . . , wn), with
wi =“l” or “r”, defined in(3.6). Then x(t; ϕ), x(t; ψ) ∈ 3w
for all t ≥ 0, by positive invariance of3w. From(4.1)we have d dt[xi(t; ϕ) − xi(t; ψ)] = −µi[xi(t; ϕ) − xi(t; ψ)] + n X j =1 αi jgj(xj(t; ϕ)) − gj(xj(t; ψ)) + n X j =1 βi jgj(xj(t − τi j;ϕ)) − gj(xj(t − τi j;ψ)) ,
for t ≥ 0, i = 1, 2, . . . , n. Similar to the proof ofTheorem 3.2, we obtain |xi(t; ϕ) − xi(t; ψ)| ≤ e−λt max 1≤ j ≤n θ∈[−τ,0]sup |xj(θ; ϕ) −xj(θ; ψ)| ! ,
for t ≥ 0 and i = 1, . . . , n, where λ > 0 is a small constant. Therefore,
kxi(t + θ; ϕ) − xi(t + θ; ψ)k ≤ e−λ(t+θ)kϕ − ψk
≤e−λ(t−τ)kϕ − ψk,
forθ ∈ [−τ, 0] and t ≥ τ, and then
kxt(θ; ϕ) − xt(θ; ψ)k ≤ e−λ(t−τ)kϕ − ψk, for all t ≥ 0.
(4.2) We choose a positive integer m such that e−λ(mT −τ) =κ < 1. Define a Poincar´e mapping P : 3w → 3w by P(ϕ) = xT(·, ϕ). Then we can derive from(4.2)that
kPm(ϕ) − Pm(ψ)k ≤ κkϕ − ψk.
This inequality implies that Pmis a contraction mapping, hence there exists a unique fixed pointφ ∈ 3wsuch that Pm(φ) = φ. Note that
Pm(Pφ) = P(Pmφ) = P(φ).
Thus P(φ) ∈ 3wis also a fixed point of Pm, and so P(φ) = φ, i.e. xT(·; φ) = φ. Let x(t; φ) be the solution of(4.1)with initial
conditionφ at t = 0, then x(t + T ; φ) is also a solution of(4.1). Note that xt +T(·; φ) = xt(·; xT(·; φ)) = xt(·; φ), for all t ≥ 0;
therefore
x(t + T ; φ) = x(t; φ), for all t ≥ 0.
This shows that x(t; φ) is exactly a T -period solution of
(4.1) in3w, and all other solutions of (4.1)in 3w converge exponentially to it as t → +∞. Thus, there exist 2n exponentially stable T -period solutions for system(4.1).
We remark that the above result can be extended to the existence of 2n stable periodic solutions of the delayed neural network(4.1)with periodic coefficients and connection weights, i.e., the situation thatµi =µi(t), αi j =αi j(t), βi j =
βi j(t) are T -periodic. The extension can be confirmed
by modifying the assumptions to adapt to the considered circumstances and by similar arguments as the proof of
Theorem 4.1. The investigation can also be generalized to almost periodic coefficients and solutions.
5. Quasiconvergence
We shall discuss the monotone property for system (1.1)
with activation functions of class A in this section. The derivation can be adapted to activation functions of class B. Let us first recall the following definition.
Definition 5.1. Let E be the set of all equilibrium points for a system with phase space C. We say that φ ∈ C is a quasiconvergent point if its ω-limit set ω(φ) ⊂ E. The set of such points is denoted by Q. A point φ ∈ C is called a convergent point, ifω(φ) consists of a single point of E. Note that quasiconvergence yields convergence for continuous-time dynamical systems, if all equilibria are isolated. In order to apply the theory of monotone dynamical systems, we need the following notations and definitions. Consider the standard componentwise partial order “≤” and inequality “<” on Rn: x ≤ y ⇔ xi ≤ yi, for all i,
Then the partial order “≤”, called the standard order, and the inequality “<” on C = C([−τ, 0], Rn) are defined by
φ ≤ ψ ⇔ φ(θ) ≤ ψ(θ) for all θ ∈ [−τ, 0], φ < ψ ⇔ φ ≤ ψ and φ 6= ψ,
φ ψ ⇔ φ(θ) ψ(θ) for all θ ∈ [−τ, 0].
Definition 5.2. Let “<” be a partial order. (i) A semiflow Φ is said to be monotone provided Φt(φ) ≤ Φt(ψ) whenever
φ ≤ ψ and t ≥ 0. (ii) Φ is called strongly order preserving (SOP), if it is monotone and wheneverφ < ψ, there exist open subsets U, V of C with φ ∈ U and ψ ∈ V and t0> 0 such that
Φt0(U) ≤ Φt0(V ).
It has been shown in [32] that if the phase space can be approximated from below or above, then IntQ is dense in C for a SOP system, under a compactness assumption. The assumptions of this theorem can all be justified in our situation herein.
Notably, the one-dimensional delayed equation dx(t)
dt = −ax(t) + bg(x(t − τ)), a > 0, b < 0,
fails to be monotone under the standard ordering in C; so do the higher-dimensional cases [32]. We shall adopt a special order introduced in [33] to conclude the monotone behavior for system (1.1). Let M be an n × n essentially nonnegative matrix, which means that M +λI is entrywise nonnegative for all sufficiently largeλ. Define
KM = {ψ ∈ C|ψ ≥ 0 and e−t Mψ(t) ≥ e−s Mψ(s),
for −τ ≤ s ≤ t ≤ 0}. (5.1) Then KM is a cone in the space C, that is, under addition and
scalar multiplication by nonnegative scalars, KM is closed in C
and KM∩(−KM) = ∅. Moreover, KM is a normal cone, which
means that every order interval is a bounded set in C. According to [33], KM induces a partial order on C.
Definition 5.3. Ifφ, ψ ∈ C, we say φ ≤Mψ whenever ψ −φ ∈
KM. We writeφ <Mψ to indicate that φ ≤Mψ and φ 6= ψ.
Consider the functional differential equation, with F˜ ∈ C1(C, Rn)
dx(t)
dt = ˜F(xt). (5.2)
Theorem 5.1 ([13,33]). The semiflow Φ generated by(5.2)is SOP on C under order “≤M”, if the following conditions hold: (i) d ˜F(φ)ψ −Mψ(0) 0 for every φ ∈ C and every ψ ∈ KM
withψ 0,
(ii) If φ ∈ C, ψ ∈ KM and L is a (nonempty) proper subset of
{1, . . . , n} such that ψj 0 for j ∈ L andψk(0) = 0 for
k 6∈ L, then(d ˜F(φ)ψ)i > 0, for some i 6∈ L.
Herein, we set the n × n matrix M = diag(−µ1 −
ν1, . . . , −µn−νn), where νi > 0 will be chosen later. Indeed,
the matrix M is essentially nonnegative. An n × n matrix A = [ Ai j]is called irreducible if whenever the set {1, . . . , n}
is expressed as the union of two disjoint proper subsets S, S0, then for every i ∈ S there exists j , k ∈ S0 such that Ai j 6=0,
Aki 6=0. Letγi =supξ∈Rg0i(ξ).
Proposition 5.2. Assume that one of the matrices A and B is irreducible, where A = [αi j], B = [βi j],αi j ≥ 0,βi j ≥0 for
all i 6= j ,αi i+βi i > 0 for all i, and the time lags {τi j}satisfy
τi i ≤1/(µi+e|βi i|γi), (5.3)
for all i withβi i < 0. Then the semiflow Φ generated by the
solutions of system(1.1)isSOP under order “≤M”.
Proof. Recall the previous definition of ˜F defined in(1.1):
˜ Fi(φ) = −µiφi(0) + n X j =1 αi jgj(φj(0)) + n X j =1 βi jgj(φj(−τi j)) + Ii, i = 1, . . . , n.
For anyφ = (φ1, . . . , φn) ∈ C and ψ = (ψ1, . . . , ψn) ∈ KM
withψ 0, we have (d ˜F(φ)ψ)i−(Mψ(0))i =νiψi(0) + n X j =1 αi jg0j(φj(0))ψj(0) + n X j =1 βi jg0j(φj(−τi j))ψj(−τi j) (5.4) ≥ [(νie−τi i(µi+νi)+β i igi0(φi(−τi i))]ψi(−τi i) +αi igi0(φi(0)))ψi(0) + n X j =1, j6=i αi jg0j(φj(0))ψj(0) + n X j =1, j6=i βi jg0j(φj(−τi j))ψj(−τi j), (5.5)
since ψi(0) ≥ e−τi i(µi+νi)ψ(−τi i), from ψ ∈ KM, and
ψ(0) ≥ e−s Mψ(s), for all s ∈ [−τ, 0]. Here, we take ν i > 0
satisfying νi = e|βi i|γi. Ifβi i < 0, then αi i > 0, and the
assumptionτi i ≤ 1/(µi +e|βi i|γi) yields νiexp[−τi i(µi +
νi)] + βi igi0(φi(−τi i)) > 0. Thus (d ˜F(φ)ψ)i−(Mψ(0))i > 0,
from(5.5). Whenβi i ≥0,(d ˜F(φ)ψ)i−(Mψ(0))i > 0 follows
fromνi+αi iγi > 0 and(5.4). Next, we will prove that condition
(ii) inTheorem 5.1holds. For anyφ ∈ C and ψ ∈ KM, let L be
a (nonempty) proper subset of {1, . . . , n} such that ψj 0 for
j ∈ L andψk(0) = 0 for k 6∈ L. Then ψi(−τi i) = 0 for each
i 6∈ L, due toψi(−τi i) ≤ exp[τi i(µi+νi)]ψi(0). Since one of
matrices A and B is irreducible, there is some i 6∈ L such that
(d ˜F(φ)ψ)i = −µiψi(0) + n X j =1 αi jg0j(φj(0))ψj(0) + n X j =1 βi jg0j(φj(−τi j))ψj(−τi j)
= n X j =1, j6=i αi jg0j(φj(0))ψj(0) + n X j =1, j6=i βi jg0j(φj(−τi j))ψj(−τi j) = n X j ∈L αi jg0j(φj(0))ψj(0) + n X j ∈L βi jg0j(φj(−τi j))ψj(−τi j) > 0.
Hence, it follows from Theorem 5.1 that the semiflow Φ generated by the solutions of(1.1)is SOP under order “≤M”.
Notably, condition (HA1) yields αi i +βi i > 0 for all i.
Thus, under conditions(HA1) and (H2), and the assumptions
inProposition 5.2, there exist 3n equilibria for(1.1)and intQ is dense in C. In fact, the assumptions of irreducibility of A, B and non-inhibitory interactions, αi j, βi j ≥ 0 for all
i 6= j, can be removed via a decomposition approach in competitive–cooperative systems after imbedding the network into a larger system. Such a technique was adopted to study global convergence to an unique equilibrium in [34,35]. It was previously employed by Cosner [36] and Wu and Zhao [37] in the study of population dynamics. This decomposition approach fits in with our formulation for multiple equilibria pertinently and skillfully.
Theorem 5.3. Assume that(HA1) and (H2) hold and the delay
time {τi j}satisfy(5.3). Then system(1.1)has3nequilibria and
intQ is dense in C.
Proof. Define matrices A+ = [ai j+], A− = [ai j−], B+ = [bi j+] and B−= [b− i j]by ai j+= α i i, for j = i α+ i j +s, for j 6= i, ai j−=0, for j = i α− i j +s, for j 6= i, bi j+= β i i, for j = i β+ i j +s, for j 6= i, bi j−=0, for j = i β− i j +s, for j 6= i,
whereαi j+ = max{αi j, 0}, α−i j = max{−αi j, 0}, similarly for
β+ i j, β
−
i j; s > 0 will be suitably chosen. Since αi j =a + i j −a − i j andβi j =b+i j−b − i j, system(1.1)becomes dxi(t) dt = −µixi(t) + n X j =1 ai j+gj(xj(t)) − n X j =1 a−i jgj(xj(t)) + n X j =1 bi j+gj(xj(t − τi j)) − n X j =1 b−i jgj(xj(t − τi j)) + Ii, (5.6)
i =1, . . . , n. Define yi = −xi, and set ˜gi(ξ) = −gi(−ξ), i =
1, . . . , n. Then(5.6)is embedded into the following system: dxi(t) dt = −µixi(t) + n X j =1 ai j+gj(xj(t)) + n X j =1 a−i jg˜j(yj(t)) + n X j =1 b+i jgj(xj(t − τi j)) + n X j =1 b−i jg˜j(yj(t − τi j)) + Ii dyi(t) dt = −µiyi(t) + n X j =1 ai j−gj(xj(t)) + n X j =1 a+i jg˜j(yj(t)) + n X j =1 b−i jgj(xj(t − τi j)) + n X j =1 b+i jg˜j(yj(t − τi j)) − Ii, (5.7)
i =1, . . . , n. Note that each ˜gi also admits the characteristics
of gi. We define zk(t) and hk(ξ) by zi(t) = xi(t), zn+i(t) =
yi(t), and hi(ξ) = gi(ξ), hn+i(ξ) = ˜gi(ξ), for i = 1, . . . , n.
Then(5.7)can be written as dzi(t) dt = − ˜µizi(t) + 2n X j =1 ˜ ai jhj(zj(t)) + 2n X j =1 ˜ bi jhj(zj(t − ˜τi j)) + ˜Ii, (5.8)
i =1, . . . , 2n, where the 2n × 2n matrices ˜A and ˜B are defined by ˜ A = [ ˜ai j] = A+ A− A− A+ , B = [ ˜˜ b i j] = B+ B− B− B+ , and ˜µi, ˜Ii, ˜τi j are given by ˜µi = µi, ˜µn+i = µi; ˜Ii =
Ii, ˜In+i = −Ii, i = 1, . . . , n; ˜τi j = τ˜n+i, j = τ˜i,n+ j =
˜
τn+i,n+ j =τi j, i, j = 1, . . . , n.
Note that ˜A, ˜B as defined are both irreducible; in addition, ˜
ai j > 0, ˜bi j > 0, for all i 6= j. System (5.8)thus satisfies
the assumptions except that ˜ai i + ˜bi i > 0, for all i, in
Proposition 5.2. It can be justified that conditions(HA1) and (H2) for system(1.1)yield conditions analogous to(HA1) and
(H2) for system(5.8), by choosing sufficiently small s > 0.
Thus, ˜ai i + ˜bi i > 0, for all i, and the semiflow generated
by the solutions of system(5.8)is SOP. Therefore, there exist 32n equilibria and IntQ is dense in C([−τ, 0], R2n) for system
(5.8). On the other hand, one also observes that if xi(0) +
yi(0) = 0, then xi(t) + yi(t) = 0 for all t ≥ 0 for solutions
(x1(t), . . . , xn(t), y1(t), . . . , yn(t)) of system (5.7). Restated,
the dynamics of system(5.8)on the positively invariant regions {x1 = −y1, . . . , xn = −yn} are exactly the dynamics for
system(1.1). Thereafter, there exist 3n equilibria for(1.1)and IntQ is dense in C([−τ, 0], Rn) for system(1.1), if(HA1), (H2)
Fig. 5. Illustration for the dynamics inExample 6.1. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
6. Numerical illustrations
In this section, four two-dimensional examples of system
(1.1) are presented to illustrate our theory. In particular,
Example 6.2 demonstrates the multistability of system (1.1)
with the standard activation function(2.2). This example adopts parameters satisfying the criteria in our theory but not the one in [26]. Example 6.3 demonstrates Theorem 4.1. The parameters inExample 6.4satisfy conditions(H1∗), ∗ = A, B,
and(H2), but not (H3).
Example 6.1. Consider the following system with activation functions g1(ξ) = g2(ξ) = tanh(ξ), which belongs to class
A: dx1(t) dt = −x1(t) + 4g1(x1(t)) + g2(x2(t)) +3g1(x1(t − 10)) + g2(x2(t − 10)) dx2(t) dt = −3x2(t) + 2g1(x1(t)) + 7g2(x2(t)) +g1(x1(t − 10)) + 5g2(x2(t − 10)).
Direct computation gives ˆf1(x1) = −x1+7g(x1)+2, ˇf1(x1) =
−x1+7g(x1) − 2, ˆf2(x2) = −3x2+12g(x2) + 3, ˇf2(x2) =
−3x2 + 12g(x2) − 3. Herein, the parameters satisfy our
conditions inTheorem 3.2: Condition (H1A): 0< µ1/(α11+β11) = 1/7 < 1, 0< µ2/(α22+β22) = 3/12 < 1. Condition (H2): ˆ f1(p1) = −2.8524 < 0, fˇ1(q1) = 2.8524 > 0, ˆ f2(p2) = −3.4414 < 0, fˇ2(q2) = 3.4414 > 0. Condition(H3): µ1=1> 0.98 = (|α11| + |β11|)η1+(|α12| + |β12|)η2, µ2=3> 1.98 = (|α21| + |β21|)η1+(|α22| + |β22|)η2,
whereη1 = 0.1 and η2 = 0.14 are chosen in (H3); ˆa1 =
−4.9994, d1 = −1.8184, p1 = −1.6283, ˆb1 = −0.3491, q1 = 1.6283, d1 = 1.8184, ˆc1 = 9.0000, ˇa1 = −9.0000, ˇ b1 = 0.3491, ˇc1 = 4.9993, ˆa2 = −2.9793, d2 = −1.6392, p2 = −1.3170, ˆb2 = −0.3518, q2 = 1.3170, d2 = 1.6392, ˆ c2 = 4.9996, ˇa2 = −4.9996, ˇb2 = 0.3518, ˇc2 = 2.9793.
The dynamics of this system are illustrated in Fig. 5, where evolutions of 72 initial conditions have been tracked. The constant initial conditions are plotted in red color, and the time-dependent initial conditions are plotted in purple. There are four exponentially stable equilibria in the system, as confirmed by our theory. The simulation demonstrates convergence to these four equilibria from initial functionsφ lying in the respective basin for the equilibrium.
Example 6.2. Consider the following system with the standard activation function(2.2): dx1(t) dt = −x1(t) + 2g1(x1(t)) + g2(x2(t)) +3g1(x1(t − 5)) + g2(x2(t − 5)) dx2(t) dt = −x2(t) − g1(x1(t)) + 4g2(x2(t)) +2g1(x1(t − 5)) + 5g2(x2(t − 5)) + 1, where g1(ξ) = g2(ξ) = gs(ξ) = 12(|ξ + 1| − |ξ − 1|).
The parameters satisfy the criterion in Corollary 3.4: α11 +
β11 −(|α12| + |β12|) − |I1| = 3 > 1 = µ1, α22 +β22−
(|α21| + |β21|) − |I2| =5> 1 = µ2. Therefore, there exist 2n
Fig. 6. Illustration for the dynamics inExample 6.2( ˜pi= −1, ˜qi=1).
Fig. 7. Illustration for the dynamics inExample 6.2( ˜pi= −1, ˜qi=1).
satisfy the criterion(3.9)for the theory in [26]:α11− |α12| −
(|β11| + |β12|) − |I1| = −3 which is not greater thanµ1 =1.
The dynamics of the system are illustrated inFig. 6. We allow initial conditions from larger basins of attraction inFig. 7, to demonstrate the assertion inRemark 3.1(iii).
Example 6.3. Consider the following system with periodic inputs and the standard activation function(2.2):
dx1(t) dt = −x1(t) + 2g1(x1(t)) + g2(x2(t)) +3g1(x1(t − 2)) + g2(x2(t − 2)) + cos(t) dx2(t) dt = −x2(t) − g1(x1(t)) + 4g2(x2(t)) +2g1(x1(t − 2)) + 5g2(x2(t − 2)) + 1 + sin(t), where g1(ξ) = g2(ξ) = gs(ξ) = 12(|ξ + 1| − |ξ − 1|). The
existence of four limit cycles for the system is illustrated in
Fig. 8.
Example 6.4. Consider the following system with activation functions g1(ξ) = g2(ξ) = tanh(ξ), which belongs to class A,
dx1(t)
dt = −x1(t) + 7g1(x1(t)) + 0.5g2(x2(t)) −4g1(x1(t − τ11)) + 0.5g2(x2(t − τ12))
Fig. 8. Illustration for the dynamics inExample 6.3( ˜pi= −1, ˜qi=1).
Fig. 9. Illustration for the dynamics inExample 6.4with activation function gi(ξ) = tanh(ξ) and τ11=0.08, τ12=10, τ21=10, τ22=0.08.
dx2(t)
dt = −x2(t) + 0.5g1(x1(t)) + 7g2(x2(t)) +0.5g1(x1(t − τ21)) − 4g2(x2(t − τ22)).
Direct computation gives ˆf1(x1) = −x1+7g(x1)+2, ˇf1(x1) =
−x1+7g(x1) − 2, ˆf2(x2) = −3x2+12g(x2) + 3, ˇf2(x2) =
−3x2+12g(x2) − 3. Herein, the parameters satisfy condition
(H1A) : 0 < µ1/(α11+β11) = µ2/(α22+β22) = 1/3 < 1, and condition (H2) : ˆf1(p1) = −2.8524 < 0, ˇf1(q1) = 2.8524 > 0, ˆf2(p2) = −3.4414 < 0, ˇf2(q2) = 3.4414 > 0. In addition, ˆa1 = −1.8572, p1 = −1.1462, ˆb1 = −0.5902, q1 = 1.1462, ˆc1 = 3.9980, ˇa1 = −3.9980, ˇb1 = 0.5902, ˇ c1 =1.8572, ˆa2 = −1.8572, p2 = −1.1462, ˆb2 = −0.5902, q2 = 1.1462, ˆc2 = 3.9980, ˇa2 = −3.9980, ˇb2 = 0.5902, ˇ
c2 = 1.8572. Note that g0(ξ) is decreasing for ξ > 0 and
increasing for ξ < 0. Condition (H3) does not hold since
µ1 = 1 < (|α11| + |β11|)g0(ˆa1) + (|α12| + |β12|)g0(ˆa1) '
11 × 0.0929 + 1 × 0.0929 = 1.1148. We choose τ11 =
0.08, τ12 = 10, τ21 = 10, τ22 = 0.08 to satisfy(5.3):τ11 =
τ22 = 0.08 < 1/(1 + 4e) ' 0.08475. The dynamics of this
system are illustrated inFig. 9.
Fig. 10depicts the dynamics for the system with the same parameters but with time lagsτ11 = τ12 = τ21 = τ22 = 10,
which do not satisfy criterion(5.3). It appears that two of the four equilibria become unstable. The dynamics are apparently different if we replace the activation function tanh(ξ) by the
Fig. 10. Illustration for the dynamics inExample 6.4with activation function gi(ξ) = tanh(ξ) and τ11=τ12=τ21=τ22=10.
Fig. 11. Illustration for the dynamics inExample 6.4with the standard activation function gi(ξ) = gs(ξ) =12(|ξ + 1| − |ξ − 1|) and τ11=τ12=τ21=τ22=10.
standard activation function gs(ξ) = 12(|ξ +1|−|ξ −1|). There still exist four stable equilibria, as illustrated inFig. 11.
7. Conclusions
With a geometrical observation, parameter conditions(H1∗)
and (H2) assuring the existence of 3n stationary solutions
for a general n-dimensional neural network with delays, have been derived. Under the same conditions, 2n out of these 3n equilibria are stable if the activation functions of class B are employed for the system. Additional assumptions (H3) and
βi i > 0 are required to guarantee the same assertion of multiple
stable equilibria, if activation functions of class A are adopted.
Further analysis has been performed to establish existence of 2n limit cycles for the network with time-periodic inputs. The derived parameter conditions are concrete and can be examined easily. We have also applied the theory of monotone dynamics to confirm the strongly order preserving property for the networks. Subsequently, that generic points in the phase space are quasiconvergent as well as existence of 3n equilibria
comprise the phase structure for the system, under conditions (H1∗), (H2), and that delays τi i are small enough for those
neurons i withβi i < 0. We have provided several numerical
simulations to illustrate these theories for the two-neuron cases. It is interesting to see the distinct dynamics between system
ones in class B. When we take parameters satisfying conditions (H1∗), (H2), and τi inot satisfying condition(5.3), there exist 32
equilibria for system(1.1)with activation functions in classes A and B. There are still four stable equilibria if the activation function of class B is employed, as illustrated inExample 6.4, and confirmed by ourTheorem 3.2. However, if we adopt the activation function in class A, two of these four equilibria become unstable.
Acknowledgments
This work is partially supported by The National Science Council, The National Center of Theoretical Sciences, and the MOEATU program, of R.O.C. on Taiwan.
References
[1] L.O. Chua, L. Yang, Cellular neural networks: Theory, IEEE Trans. Circuits Syst. 35 (1988) 1257–1272.
[2] J. Foss, A. Longtin, B. Mensour, J. Milton, Multistability and delayed recurrent loops, Phys. Rev. Lett. 76 (1996) 708–711.
[3] J. Hopfield, Neurons with graded response have collective computational properties like those of two sate neurons, Proc. Natl. Acad. Sci. USA 81 (1984) 3088–3092.
[4] M. Morita, Associative memory with non-monotone dynamics, Neural Netw. 6 (1993) 115–126.
[5] R.L.T. Hahnloser, On the piecewise analysis of networks of linear threshold neurons, Neural Netw. 11 (1998) 691–697.
[6] C.M. Marcus, R.M. Westervelt, Stability of analog neural networks with delay, Phys. Rev. A 39 (1989) 347–359.
[7] L.O. Chua, L. Yang, Cellular neural networks: Applications, IEEE Trans. Circuits Syst. 35 (1988) 1273–1290.
[8] T. Roska, L.O. Chua, Cellular neural networks with nonlinear and delay-type template, Int. J. Circ. Theor. Appl. 20 (1992) 469–481.
[9] L.O. Chua, CNN: A Paradigm for Complexity, World Scientific, 1998. [10] P. Baldi, A.F. Atiya, How delays affect neural dynamics and learning,
IEEE Trans. Neural Netw. 5 (1994) 612–621.
[11] J. B´elair, S.A. Campbell, P. van den Driessche, Frustration, stability, and delay-induced oscillations in a neural network model, SIAM J. Appl. Math. 56 (1996) 245–255.
[12] L. Olien, J. B´elair, Bifurcations, stability, and monotonicity properties of a delayed neural network model, Physica D 102 (1997) 349–363. [13] P. van den Driessche, X. Zou, Global attractivity in delayed Hopfield
neural network models, SIAM J. Appl. Math. 58 (1998) 1878–1890. [14] L.P. Shayer, S.A. Campbell, Stability, bifurcation, and multistability in a
system of two coupled neurons with multiple time delays, SIAM J. Appl. Math. 61 (2000) 673–700.
[15] S.A. Campbell, R. Edwards, P. van den Driessche, Delayed coupling between two neural network loops, SIAM J. Appl. Math. 65 (2004) 316–335.
[16] J. Cao, Global stability analysis in delayed cellular neural networks, Phys. Rev. E 59 (1999) 5940–5944.
[17] J. Cao, New results concerning exponential stability and periodic solutions of delayed cellular neural networks, Phys. Lett. A 307 (2003) 136–147. [18] J. Cao, Q. Li, On the exponential stability and periodic solutions of
delayed cellular neural networks, J. Math. Anal. Appl. 252 (2000) 50–64. [19] P.P. Civalleri, M. Gilli, On stability of cellular neural networks with delay,
IEEE Trans. Circuits Syst. 40 (1993) 157–165.
[20] C. Feng, R. Plamondon, On the stability analysis of delayed neural networks systems, Neural Netw. 14 (2001) 1181–1188.
[21] X. Liao, G. Chen, E.N. Sanchez, Delay-dependent exponential stability analysis of delayed neural networks: An LMI approach, Neural Netw. 15 (2002) 855–866.
[22] M. Joy, Results concerning the absolute stability of delayed neural networks, Neural Netw. 13 (2000) 613–616.
[23] X. Liao, C. Li, An LMI approach to asymptotical stability of multi-delayed neural networks, Physica D 200 (2005) 139–155.
[24] S. Mohamad, K. Gopalsamy, Exponential stability of continuous-time and discrete-time cellular neural networks with delays, Appl. Math. Comput 135 (2003) 17–38.
[25] Q. Zhang, X. Wei, J. Xu, Global exponential convergence analysis of delayed neural networks with time-varying delays, Phys. Lett. A 318 (2003) 537–544.
[26] Z. Zeng, D.S. Huang, Z. Wang, Memory pattern analysis of cellular neural networks, Phys. Lett. A 342 (2005) 114–128.
[27] C.Y. Cheng, K.H. Lin, C.W. Shih, Multistability in recurrent neural networks, SIAM J. Appl. Math. 66 (4) (2006) 1301–1320.
[28] M. Hirsch, Convergent activation dynamics in continuous-time networks, Neural Netw. 2 (1989) 331–349.
[29] J.H. Wu, Introduction to Neural Dynamics and Signal Transmission Delay, Walter de Gruyter, Berlin, 2001.
[30] X. Liao, J. Wang, Global dissipativity of continuous-time recurrent neural networks with time delay, Phys. Rev. E 68 (2003) 016118.
[31] C.Y. Cheng, Multistability and Convergence in Delayed Neural Networks, Ph.D. Dissertation, National Chiao Tung University, Hsinchu, Taiwan, 2006.
[32] H.L. Smith, Monotone Dynamical Systems, in: Mathematical Surveys and Monographs, vol. 41, American Mathematical Society, 1995.
[33] H.L. Smith, H.R. Thieme, Strongly order preserving semiflows generated by functional differential equations, J. Differential Equations 93 (1991) 332–363.
[34] P. van den Driessche, J. Wu, X. Zou, Stabilization role of inhibitory self-connections in a delayed neural network, Physica D 150 (2001) 84–90. [35] T. Chu, Z. Zhang, Z. Wang, A decomposition approach to analysis of
competitive–cooperative neural networks with delay, Phys. Lett. A 312 (2003) 339–347.
[36] C. Cosner, Comparison principles for systems that embed in cooperative systems with applications to diffusive Lotka–Volterra models, Dyn. Contin. Discrete Impuls. Syst. 3 (1997) 283–303.
[37] J. Wu, X.Q. Zhao, Permanence and convergence in multi-species competition systems with delay, Proc. Amer. Math. Soc. 126 (1998) 1709–1714.
