to appear in Journal of Mathematical Analysis and Applications, 2011

**A merit function method for infinite-dimensional SOCCPs**

Yungyen Chiang ^{1}

Department of Applied Mathematics National Sun Yat-sen University

Kaohsiung 80424, Taiwan

Shaohua Pan ^{2}

School of Mathematical Sciences South China University of Technology

Guangzhou 510640, China

Jein-Shan Chen ^{3}
Department of Mathematics
National Taiwan Normal University

Taipei 11677, Taiwan

July 25, 2009

(revised December 20, 2010)

**Abstract. We introduce the Jordan product associated with the second-order cone IK**
into the real Hilbert space*H, and then deﬁne a one-parametric class of complementarity*
functions Φ* _{t}* on

*H × H with the parameter t ∈ [0, 2). We show that the squared norm*of Φ

_{t}*with t*

*∈ (0, 2) is a continuously F(r´echet)-diﬀerentiable merit function. By this,*the second-order cone complementarity problem (SOCCP) in

*H can be converted into*an unconstrained smooth minimization problem involving this class of merit functions, and furthermore, under the monotonicity assumption, every stationary point of this minimization problem is shown to be a solution of the SOCCP.

**Key words: Hilbert space, complementarity, second-order cone, merit functions.**

1The author’s work is partially supported by grants from the National Science Council of the Republic of China. E-mail : chiangyy@math.nsysu.edu.tw.

2The author’s work is supported by Guangdong Natural Science Foundation (No. 9251802902000001) and the Fundamental Research Funds for the Central Universities (SCUT). E-mail: shhpan@scut.edu.cn

3Member of Mathematics Division, National Center for Theoretical Sciences, Taipei Oﬃce.

The author’s work is partially supported by National Science Council of Taiwan. E-mail:

jschen@math.ntnu.edu.tw.

**1** **Introduction**

Let*H be a real Hilbert space endowed with an inner product ⟨·, ·⟩. The complementarity*
*problem CP(K, T ) in* *H is, for any given closed convex cone K ⊆ H and a continuously*
F(r´*echet)-diﬀerentiable mapping T :H → H, to ﬁnd a vector x ∈ H such that*

*x∈ K, T (x) ∈ K** ^{∗}* and

*⟨x, T (x)⟩ = 0*(1)

*where K*

*:=*

^{∗}*{x ∈ H | ⟨x, y⟩ ≥ 0 ∀y ∈ K} is the dual cone of K. A closed convex*

*cone K in*

*H is called self-dual if K coincides with its dual cone K*

*; for example, the non-negative orthant cone IR*

^{∗}

^{n}_{+}:=

*{(x*1

*, . . . x*

*)*

_{n}*∈ IR*

^{n}*| x*

*j*

*≥ 0, j = 1, 2, . . . , n} and the*second-order cone (also called Lorentz cone) IK

*:=*

^{n}*{(r, x*

*)*

^{′}*∈ IR × IR*

^{n}

^{−1}*| r ≥ ∥x*

^{′}*∥}.*

This paper is concerned with the complementarity problem associated with the inﬁnite-
dimensional second-order cone IK in*H which is closed, convex and self-dual (see Section*
*2 for its deﬁnition). The problem, denoted by CP(IK, T ), is to ﬁnd an x∈ IK such that*
*x∈ IK, T (x) ∈ IK and ⟨x, T (x)⟩ = 0.* (2)
This class of problems arises directly from the optimality conditions of certain types of
inﬁnite-dimensional optimization problems such as the one in [10], which is the refor-
mulation of a min-max optimization problem with linear constraints in a Hilbert space.

Recently, nonlinear symmetric cone optimization and complementarity problems in ﬁnite-dimensional spaces such as semideﬁnite cone optimization and complementarity problems, second-order cone (SOC) optimization and complementarity problems, and general symmetric cone optimization and complementarity problems, become an active research ﬁeld of mathematical programming. Taking SOC optimization and comple- mentarity problems for example, there have proposed many eﬀective solution methods, including the interior point methods [2, 18, 21, 22], the smoothing Newton methods [5, 9, 11], the semismooth Newton methods [16, 19], and the merit function method [6, 3]. However, to our best knowledge, there are few works about nonlinear symmetric cone optimization and complementarity problems in inﬁnite-dimensional spaces except [10], in which with the JB algebras of ﬁnite rank primal-dual interior-point methods are presented for some special type of inﬁnite-dimensional cone optimization problems.

*In this paper, we consider a merit function method for solving the problem CP(IK, T ).*

The method aims to seek a smooth merit function Ψ :*H × H → IR*+ satisfying

*Ψ(x, y) = 0* *⇐⇒ x ∈ IK, y ∈ IK, ⟨x, y⟩ = 0,* (3)
*and reformulates the problem CP(IK, T ) as a smooth minimization problem*

min*x**∈H**Ψ(x, T (x))* (4)

*in the sense that x*^{∗}*is a solution of CP(IK, T ) if and only if x** ^{∗}*solves (4) with zero optimal
value. We call such Ψ a merit function associated with IK. Like handling complemen-
tarity problems in ﬁnite-dimensional spaces, we seek a merit function associated with IK
with a complementarity function (C-function for short) associated with IK. Speciﬁcally,
a mapping Φ :

*H × H → H is called a C-function associated with IK if for any x, y ∈ H,*

*Φ(x, y) = 0* *⇐⇒ x ∈ IK, y ∈ IK and ⟨x, y⟩ = 0.*

Clearly, the squared norm of Φ induces a merit function associated with IK.

When*H is the Euclidean space IR** ^{n}*, the Fischer-Burmeister (FB) and natural residual
(NR) C-functions associated with the SOC IK

*[9] are respectively deﬁned as*

^{n}Φ_{FB}*(x, y) := (x*^{2}*+ y*^{2})^{1/2}*− (x + y)* *∀x, y ∈ IR** ^{n}* (5)
and

Φ_{NR}*(x, y) := x− (x − y)*+ *∀x, y ∈ IR*^{n}*,* (6)
*where x*^{2} *= x•x with “•” means the Jordan product in IR*^{n}*, x*^{1/2}*with x∈ IK** ^{n}* is a vector

*such that x*

^{1/2}*• x*

^{1/2}*= x, and (x)*

_{+}denotes the projection onto IK

*. The function Φ*

^{n}_{FB}was well-studied in [6, 20], and particularly its squared norm was shown to be a smooth merit function in [6]. Since the squared norm of Φ

_{NR}is not diﬀerentiable, it is often involved in the smoothing methods for the SOCCPs [5, 11]. The above two C-functions are subsumed in Kanzow and Kleinmichel C-function associated with IK

*:*

^{n}Φ_{t}*(x, y) :=*[

*(x− y)*^{2}*+ 2tx• y*]*1/2*

*− (x + y)* *∀x, y ∈ IR** ^{n}* (7)

*where t is an arbitrary but ﬁxed real number from [0, 2). This function was studied in [4]*

*and its squared norm with t∈ (0, 2) was shown to be continuously diﬀerentiable. Note*
*that, when n = 1, Φ*_{FB}, Φ_{NR} and Φ* _{t}* reduce to the FB NCP-function [8], the minimum
function [14], and the Kanzow and Kleinmichel NCP-function [12], respectively.

To deﬁne these C-functions in the Hilbert space*H, we introduce the Jordan product*
associated with the cone IK, and extend the Kanzow and Kleinmichel C-function deﬁned
in (7) to*H and show that it satisﬁes the property (3) for each t ∈ [0, 2). In Section 4, we*
*prove that the squared norm of this class of C-functions with t∈ (0, 2) are continuously*
F-diﬀerentiable in *H × H. Note that the corresponding results in [4, 6] were proved by*
the spectral factorization of vectors, but here we shall not formally use this concept. In
Section 5, under the monotonicity assumption, we establish that every stationary point
of the unconstrained minimization problem involving this class of merit functions is a
*solution of CP(IK, T ), which generalizes the results of [4, Prop.4.1] and [6, Prop.3].*

Throughout this paper, *∥ · ∥ denote the norm induced by the inner product ⟨·, ·⟩ in*
*H. For any given Banach spaces X and Y, let L(X , Y) denote the Banach space of all*

continuous linear mappings from *X into Y. We simply write L(X , X ) = L(X ) and de-*
note GL(*X ) by the set of all invertible mappings in L(X ). The norm of any l ∈ L(X , Y)*
is deﬁned by *∥l∥ := sup{∥l(x)∥ | x ∈ X and ∥x∥ = 1}. In addition, for any self-adjoint*
*linear operator l from* *X → X , we write l ≻ 0 (respectively, l ≽ 0) to mean that l is*
positive deﬁnite (respectively, positive semideﬁnite).

**2** **Lorentz cone and Jordan product**

This section is devoted to introducing the Lorentz cone IK mentioned above which is the
*unique self-dual cone in a family of pointed closed convex cones K in* *H. Every cone*
*in K is the image of IK under some mapping in GL(H). Associated with the self-dual*
closed convex cone, the Jordan product is introduced into the Hilbert space *H.*

*For every integer n > 1, the Lorentz cone IK** ^{n}* given in Section 1 can be written as
IK

*:=*

^{n}{

*x∈ IR*^{n}*| ⟨x, e⟩ ≥* 1

*√*2*∥x∥*

}

*with e = (1, 0)∈ IR × IR*^{n}^{−1}*.*

This motivates us to consider the following closed convex cone in the Hilbert space *H:*

*K(e, r) :=*

{

*x∈ H | ⟨x, e⟩ ≥ r∥x∥*}

*where e∈ H with ∥e∥ = 1 and r ∈ IR with 0 < r < 1. Observe that K(e, r) is pointed,*
*i.e., K(e, r)∩ (−K(e, r)) = {0}. Let ⟨e⟩** ^{⊥}* :={

*x∈ H | ⟨x, e⟩ = 0*}

*. Then any x∈ H can*
*be written as x = x*^{′}*+ λe with x*^{′}*∈ ⟨e⟩*^{⊥}*and λ* *∈ IR. By noting that*

*⟨x, e⟩ ≥ r∥x∥ ⇐⇒ λ ≥ r(∥x*^{′}*∥*^{2}*+ λ*^{2})^{1/2}*⇐⇒ λ ≥* *r*

*√*1*− r*^{2}*∥x*^{′}*∥,*

*the closed convex cone K(e, r) can be expressed as*
*K(e, r) =*

{

*x*^{′}*+ λe∈ H | x*^{′}*∈ ⟨e⟩*^{⊥}*and λ≥* *r*

*√*1*− r*^{2} *∥x*^{′}*∥*
}

*.*

**Proposition 2.1 For any unit vector e**∈ H and 0 < r < 1, the dual cone of K(e, r) is*K(e,√*

1*− r*^{2}*). Hence, the cone K*
(

*e,**√*^{1}
2

)

={

*x*^{′}*+ λe∈ H | λ ≥ ∥x*^{′}*∥*}

*is self-dual.*

**Proof. Let x = x**^{′}*+ λe∈ K(e,√*

1*− r*^{2}*) and y = y*^{′}*+ µe* *∈ K(e, r) be arbitrary. Since*
*λµ≥ ∥x*^{′}*∥ · ∥y*^{′}*∥, we have ⟨x, y⟩ ≥ ⟨x*^{′}*, y*^{′}*⟩ + ∥x*^{′}*∥ · ∥y*^{′}*∥ ≥ 0. This proves that*

*K(e,√*

1*− r*^{2})*⊂ K*^{∗}*(e, r).*

*Conversely, let x = x*^{′}*+ λe∈ K*^{∗}*(e, r) be arbitrary, and we will prove x∈ K(e,√*

1*− r*^{2}),
*i.e., λ≥ r*^{−1}*√*

1*− r*^{2} *∥x*^{′}*∥. This is trivial when x*^{′}*= 0. When x*^{′}*̸= 0, by considering the*
*element v =−r*^{−1}*√*

1*− r*^{2}*x** ^{′}*+

*∥x*

^{′}*∥ e of K(e, r), we have*0

*≤ ⟨x, v⟩ =*(

*λ− r*^{−1}*√*

1*− r*^{2}*∥x*^{′}*∥*)

*∥x*^{′}*∥,*
which implies the result. The proof is complete. *2*

*Note that the unit vector e* *∈ H is not unique. Every unit vector e determines a*
*Lorentz cone K(e,**√*^{1}

2*). In this work, we consider a ﬁxed unit vector e and write*
*IK = K*

(
*e,* 1

*√*2
)

= {

*x*^{′}*+ λe∈ H | λ ≥ ∥x*^{′}*∥*}
*.*

*Unless stated otherwise, we shall alternatively write any x* *∈ H as x = x*^{′}*+ λe with*
*x*^{′}*∈ ⟨e⟩*^{⊥}*and λ =* *⟨x, e⟩. This expression is needed for stating many results and sim-*
*plifying the computation in the subsequent analysis. In addition, for any x, y* *∈ H, we*
*shall write x≻*IK *y (respectively, x≽*IK *y) if x− y ∈ intIK (respectively, x − y ∈ IK).*

Next we show that the solution sets of complementarity problems associated with
*any K(e, r) are related to those associated with IK via the mappings in GL(H).*

**Lemma 2.1 For any given 0 < r, s < 1, let Λ*** _{(r,s)}*:

*H → H be the mapping deﬁned by*Λ

_{(r,s)}*(x*

^{′}*+ λe) :=*

*√*1*− s*^{2}

*√*1*− r*^{2} *x** ^{′}*+

*sλ*

*r* *e* *∀x*^{′}*+ λe∈ H.*

*Then, the following statements hold.*

**(a) Λ**_{(r,s)}*∈ GL(H) with Λ*^{−1}* _{(r,s)}*= Λ

_{(s,r)}*, and Λ*

_{(r,s)}*maps K(e, r) onto K(e, s).*

**(b) Let Λ*** _{r}* := Λ

_{(r,}

_{√}^{1}

2)*. If r*^{2} *+ s*^{2} *= 1, then* *⟨Λ**r**(x), Λ*_{s}*(y)⟩ =* _{2rs}^{1} *⟨x, y⟩ for all x, y ∈ H.*

**Proof. (a) It is clear that Λ*** _{(r,s)}* is linear and Λ

^{−1}*= Λ*

_{(r,s)}

_{(s,r)}*. For x*

^{′}*∈ ⟨e⟩*

^{⊥}*and λ∈ IR,*

*∥Λ**(r,s)**(x*^{′}*+ λe)∥*^{2} = 1*− s*^{2}

1*− r*^{2} *∥x*^{′}*∥*^{2}+ *s*^{2}

*r*^{2} *λ*^{2} *≤ max*

{ 1*− s*^{2}
1*− r*^{2}*,* *s*^{2}

*r*^{2}
}

*∥x*^{′}*+ λe∥*^{2}*.*
This proves the continuity of Λ* _{(r,s)}*. Also, Λ

_{(r,s)}*maps K(e, r) onto K(e, s) by noting that*

*x*^{′}*+ λe∈ Λ**(r,s)**(K(e, r))* *⇐⇒ Λ**(s,r)**(x*^{′}*+ λe)∈ K(e, r)*

*⇐⇒* *rλ*

*s* *≥* *r*

*√*1*− r*^{2} *·*

*√*1*− r*^{2}

*√*1*− s*^{2} *∥x*^{′}*∥*

*⇐⇒ λ ≥* *s*

*√*1*− s*^{2} *∥x*^{′}*∥.*

*(b) We write x = x*^{′}*+ λe and y = y*^{′}*+ µe. Then,*
Λ_{r}*(x*^{′}*+ λe) =* 1

√2(1*− r*^{2}) *x** ^{′}* +

*λ*

*√2r* *e =* 1

*√2s* *x** ^{′}*+

*λ*

*√2r* *e ;*

Λ_{s}*(y*^{′}*+ µe) =* 1

*√2r* *y** ^{′}*+

*µ*

√2(1*− r*^{2}) *e =* 1

*√2r* *y** ^{′}*+

*µ*

*√2s* *e .*
Now, the assertion follows immediately by a direct computation. *2*

From Lemma 2.1, we immediately obtain the following proposition.

**Proposition 2.2 Let 0 < r, s < 1 be such that r**^{2}*+ s*^{2} *= 1, and T :H → H be given.*

**(a) A point x***∈ H solves the problem CP(K(e, r), T ) if and only if Λ**r**(x) solves the*
*problem CP(IK, Λ*_{s}*◦ T ◦ Λ*^{−1}*r* *).*

**(b) If Φ :**H × H → H is a C-function associated with IK, then the mapping Φ*r**(x, y) :=*

Φ(Λ_{r}*(x), Λ*_{s}*(y)) is a C-function associated with K(e, r).*

Next we introduce the Jordan product associated with the Lorentz cone IK. For any
*x = x*^{′}*+ λe∈ H and y = y*^{′}*+ µe* *∈ H, we deﬁne the Jordan product of x and y by*

*x• y := (µx*^{′}*+ λy** ^{′}*) +

*⟨x, y⟩e,*(8)

*and write x*

^{2}

*= x*

*• x. Clearly, when H = IR*

^{n}*and e = (1, 0)*

*∈ IR × IR*

^{n}*, this deﬁnition is same as the one given by [7, Chapter II]. By the deﬁnition in (8) and a direct computation, it is easy to verify that the following properties hold.*

^{−1}**Property 2.1 (i) x**• y = y • x and x • e = x for all x, y ∈ H.

**(ii) (x + y)**• z = x • z + y • z for all x, y, z ∈ H.

**(iii)** *⟨x, y • z⟩ = ⟨y, x • z⟩ = ⟨z, x • y⟩ for all x, y, z ∈ H.*

**(iv) For any x = x**^{′}*+ λe∈ H, x*^{2} *= x• x = 2λx** ^{′}*+

*∥x∥*

^{2}

*e∈ IK and ⟨x*

^{2}

*, e⟩ = ∥x∥*

^{2}

*.*

**(v) If x = x**

^{′}*+ λe∈ IK, then there is a unique x*

^{1/2}*∈ IK such that (x*

*)*

^{1/2}^{2}

*= x, where*

*x** ^{1/2}* =

{ 0 *if x = 0;*

*x*^{′}*/(2τ ) + τ e othewise* with *τ =*

√

*λ +*√

*λ*^{2}*− ∥x*^{′}*∥*^{2}

2 *.* (9)

**(vi) Every x = x**^{′}*+ λe∈ H with λ*^{2}*− ∥x*^{′}*∥*^{2} *̸= 0 is invertible w.r.t. the Jordan product,*
*i.e., there is a unique point x*^{−1}*∈ H such that x • x*^{−1}*= e, where*

*x** ^{−1}* =

*−x*

^{′}*+ λe*

*λ*^{2}*− ∥x*^{′}*∥*^{2}*.* (10)

*Moreover, x∈ intIK if and only if x*^{−1}*∈ intIK.*

*Associated with every x* *∈ H, we deﬁne a linear mapping L**x* from *H to H by*

*L*_{x}*y := x• y for any y ∈ H.* (11)

*Clearly, L**x* *∈ L(H). Also, the mapping possesses the following favorable properties.*

**Lemma 2.2 For any x**∈ H, let L*x* *∈ L(H) be deﬁned as above. Then, we have*
* (a) x≻*IK 0

*⇐⇒ L*

*x*

*≻ 0 and x ≽*IK 0

*⇐⇒ L*

*x*

*≽ 0.*

**(b) If x = x**^{′}*+ λe with λ̸= 0 and |λ| ̸=∥x*^{′}*∥, then L**x**∈ GL(H) with the inverse given by*
*L*^{−1}_{x}*y = λ** ^{−1}*(

*y*^{′}*− ⟨x*^{−1}*, y⟩x** ^{′}*)

+*⟨x*^{−1}*, y⟩e for any y = y*^{′}*+ µe* *∈ H.* (12)
**Proof. (a) Fix any x = x**^{′}*+ λe∈ H. It suﬃces to prove the ﬁrst equivalence, and the*
second equivalence follow from the ﬁrst equivalence and the closedness of IK. Note that
*L*_{x}*≻ 0 if and only if ⟨h, L**x**h⟩ > 0 for any h = h*^{′}*+ ξe* *∈ H\{0}, whereas*

*⟨h, L**x**h⟩ > 0 ⇐⇒ λ∥h*^{′}*∥*^{2}*+ 2ξ⟨x*^{′}*, h*^{′}*⟩ + λξ*^{2} *> 0*

*⇐⇒ λ > 0 and 4⟨x*^{′}*, h*^{′}*⟩*^{2}*− 4λ*^{2}*∥h*^{′}*∥*^{2} *< 0*

*⇐⇒ λ > 0 and ∥x*^{′}*∥ < λ.*

*(b) To prove L*_{x}*∈ GL(H), it suﬃces to prove that L**x**y = 0 for some y = y*^{′}*+ µe* *∈ H*
*implies y = 0. Indeed, since L*_{x}*y = 0 implies* *∥x • y∥*^{2} = 0, which is equivalent to

*λy*^{′}*+ µy** ^{′}* = 0 and

*⟨x*

^{′}*, y*

^{′}*⟩ + λµ = 0.*

*Since λ* *̸= 0, from the ﬁrst equality we have y** ^{′}* =

*−λ*

^{−1}*µx*

*. Substituting it into the*

^{′}*second equality yields µ = 0, and so y*

*= 0. A direct computation veriﬁes (12).*

^{′}*2*

**3** **Kanzow-Kleinmichel merit function**

In this section, we will extend Kanzow-Kleinmichel C-function in (7) to the real Hilbert
space*H, and present some technical lemmas that will be used in the subsequent analysis.*

*Let t be an arbitrary real number in [0, 2). Deﬁne the mapping Φ** _{t}*:

*H × H → H by*Φ

_{t}*(x, y) :=*[

*(x− y)*^{2}*+ 2t(x• y)*]*1/2*

*− (x + y).* (13)

*Note that, for any t∈ [0, 2) and any x, y ∈ H,*

*(x− y)*^{2}*+ 2t(x• y) = (x + (t − 1)y)*^{2} *+ t(2− t)y*^{2} *∈ IK.* (14)

Hence, the function Φ_{t}*is well-deﬁned. It is easy to see that when t = 1 and t = 0, Φ** _{t}*
reduces to the FB and the NR C-function associated with IK, respectively.

To show that each Φ* _{t}*is a C-function associated with IK, we need the following result
which is an inﬁnitely dimensional version of [9, Prop.2.1]. The proof given in [9] was
based on the geometry of vectors in Euclidean spaces, that is, the notion of an angle
between vectors. We here give another proof without using this notion.

**Lemma 3.1 For any x, y**∈ H, the following statements are equivalent:

**(a) x**∈ IK, y ∈ IK and ⟨x, y⟩ = 0;

**(b) x**∈ IK, y ∈ IK and x • y = 0;

**(c) x + y**∈ IK and x • y = 0.

**(d) It holds that (i) x = 0, y***∈ IK; or (ii) x ∈ IK, y = 0; or (iii) x ∈ ∂IK, y ∈ ∂IK and*

*⟨x, y⟩ = 0, where ∂IK := {x*^{′}*+ λe∈ H | λ = ∥x*^{′}*∥} denotes the boundary of IK.*

**Proof. Clearly, (b)**⇒ (c) and (d) ⇒ (a). We need to prove (a) ⇒ (b) and (c) ⇒ (d).

(a) *⇒ (b). Write x = x*^{′}*+ λe and y = y*^{′}*+ µe. By (8) and* *⟨x, y⟩ = 0, we have*
*x• y = (µx*^{′}*+ λy*^{′}*). Since λ≥ ∥x*^{′}*∥ and µ ≥ ∥y*^{′}*∥ by x, y ∈ IK, it follows that*

*∥µx*^{′}*+ λy*^{′}*∥*^{2} *= µ*^{2}*∥x*^{′}*∥*^{2}*− 2λ*^{2}*µ*^{2}*+ λ*^{2}*∥y*^{′}*∥*^{2} *≤ 0,*

*and µx*^{′}*+ λy*^{′}*= 0 follows. Thus, we obtain x• y = 0, and hence (a) implies (b).*

(c) *⇒ (d). Since x • y = 0 implies ∥(µx*^{′}*+ λy** ^{′}*) +

*⟨x, y⟩e∥*

^{2}=

*∥µx*

^{′}*+ λy*

^{′}*∥*

^{2}+

*⟨x, y⟩*

^{2}

*= 0,*we have

*⟨x, y⟩ = 0 and µx*

^{′}*+ λy*

^{′}*= 0. If λ = 0, µ*

*̸= 0, then from µx*

^{′}*+ λy*

*= 0 and*

^{′}*⟨x, y⟩ = 0, we get x*^{′}*= 0, and then x = 0. Together with x + y* *∈ IK, we obtain y ∈ IK,*
*and so Case (i) holds. If λ* *̸= 0, µ = 0, a similar argument yields that Case (ii) holds.*

*If λ = µ = 0, then from x + y* *∈ IK it follows that ∥x*^{′}*+ y*^{′}*∥ = 0. This along with*

*⟨x, y⟩ = 0 and λ = 0, µ = 0 yields that x*^{′}*= 0 and y*^{′}*= 0, and consequently, x = y = 0.*

*Hence, Cases (i), (ii) and (iii) hold. Now, assume that λµ* *̸= 0. From µx*^{′}*+ λy** ^{′}* = 0
and

*⟨x, y⟩ = 0, we obtain λ*

^{2}=

*∥x*

^{′}*∥*

^{2}

*and µ*

^{2}=

*∥y*

^{′}*∥*

^{2}

*. This, together with x + y*

*∈ IK,*

*i.e. (λ + µ)*

^{2}

*≥ ∥x*

^{′}*+ y*

^{′}*∥*

^{2}

*, implies λµ*

*≥ ⟨x*

^{′}*, y*

^{′}*⟩ = −λµ, and hence λµ > 0. Since*

*λ + µ≥ ∥x*

^{′}*+ y*

^{′}*∥, we get λ > 0 and µ > 0. Thus, λ = ∥x*

^{′}*∥ and µ = ∥y*

^{′}*∥, which implies*

*that x, y*

*∈ IK. That is, Case (iii) follows.*

*2*

Let Ψ*t*:*H × H → IR*+ denote the squared norm of the function Φ*t*, that is,

Ψ_{t}*(x, y) :=∥Φ**t**(x, y)∥*^{2} *∀x, y ∈ H.* (15)

From the expression of Φ* _{t}* and Lemma 3.1, it follows that

Ψ_{t}*(x, y) = 0⇔ Φ**t**(x, y) = 0* *⇔ x + y ∈ IK and (x − y)*^{2}*+ 2t(x• y) = (x + y)*^{2}

*⇔ x + y ∈ IK and x • y = 0*

*⇔ x ∈ IK, y ∈ IK, ⟨x, y⟩ = 0.*

These equivalence immediately implies the following result.

**Proposition 3.1 The functions Φ***t* *and Ψ**t* *are respectively a C-function and a merit*
*function associated with IK.*

In what follows, we provide some necessary technical lemmas that will be used later.

**Lemma 3.2 For any given 0 < t < 2, x = x**^{′}*+ λe∈ H and y = y*^{′}*+ µe* *∈ H, we have*
*(x− y)*^{2}*+ 2t(x• y) ∈ ∂IK ⇐⇒ x*^{2}*+ y*^{2} *∈ ∂IK*

*⇐⇒ |λ| = ∥x*^{′}*∥, |µ| = ∥y*^{′}*∥, λµ = ⟨x*^{′}*, y*^{′}*⟩* (16)

=*⇒ λy*^{′}*= µx*^{′}

**Proof. Using***|λ| = ∥x*^{′}*∥, |µ| = ∥y*^{′}*∥ and λµ = ⟨x*^{′}*, y*^{′}*⟩, it is easy to verify ∥λy*^{′}*−µx*^{′}*∥*^{2} = 0.

So, the implication in (16) holds. Now we prove the second equivalence. Noting that
*x*^{2}*+ y*^{2} *= 2(λx*^{′}*+ µy** ^{′}*) + (

*∥x∥*

^{2}+

*∥y∥*

^{2}

*)e,*

2*∥λx*^{′}*+ µy*^{′}*∥ ≤ 2∥λx*^{′}*∥ + 2∥µy*^{′}*∥ ≤ ∥x∥*^{2}+*∥y∥*^{2}*,*

*we have x*^{2}*+ y*^{2} *∈ ∂IK if and only if ∥x∥*^{2}+*∥y∥*^{2} = 2*∥λx*^{′}*∥ + 2∥µy*^{′}*∥ = 2∥λx*^{′}*+ µy*^{′}*∥, i.e.,*
(*|λ| − ∥x*^{′}*∥)*^{2}+ (*|µ| − ∥y*^{′}*∥)*^{2} = 0 and *∥λx*^{′}*∥ + ∥µy*^{′}*∥ = ∥λx*^{′}*+ µy*^{′}*∥. Thus, we have*

*x*^{2}*+ y*^{2} *∈ ∂IK ⇐⇒ |λ| = ∥x*^{′}*∥, |µ| = ∥y*^{′}*∥, λµ⟨x*^{′}*, y*^{′}*⟩ = |λµ| · ∥x*^{′}*∥ · ∥y*^{′}*∥.*

We may argue that, when *|λ| = ∥x*^{′}*∥ and |µ| = ∥y*^{′}*∥, there holds that*
*λµ⟨x*^{′}*, y*^{′}*⟩ = |λµ| · ∥x*^{′}*∥ · ∥y*^{′}*∥ ⇐⇒ λµ = ⟨x*^{′}*, y*^{′}*⟩.*

*Indeed, if the equality on the right hand side holds, then λµ⟨x*^{′}*, y*^{′}*⟩ = λ*^{2}*µ*^{2} = *|λµ| ·*

*∥x*^{′}*∥ · ∥y*^{′}*∥, which implies the equality of the left hand side. Assume that the equality of*
*the left hand side holds. If λµ = 0 or* *∥x*^{′}*∥ · ∥y*^{′}*∥ = 0, then x = 0 or y = 0, and thus*
*λµ = 0 =⟨x*^{′}*, y*^{′}*⟩; and if λµ ̸= 0 and ∥x*^{′}*∥·∥y*^{′}*∥ ̸= 0, using λµ⟨x*^{′}*, y*^{′}*⟩ = |λµ|·∥x*^{′}*∥·∥y*^{′}*∥ > 0*
then yields that *|⟨x*^{′}*, y*^{′}*⟩| = ∥x*^{′}*∥ · ∥y*^{′}*∥ = |λµ| and ⟨x*^{′}*, y*^{′}*⟩ = λµ. This proves that the*
equality of the right hand side holds, and the second equivalence in (16) follows.

To establish the ﬁrst equivalence in (16), it suﬃces to prove that

*(x− y)*^{2}*+ 2t(x• y) ∈ ∂IK ⇐⇒ |λ| = ∥x*^{′}*∥, |µ| = ∥y*^{′}*∥, λµ = ⟨x*^{′}*, y*^{′}*⟩.* (17)

*Recall that (x− y)*^{2}*+ 2t(x• y) = (x + (t − 1)y)*^{2}+ (√

*t(2− t) y)*^{2}*. By the result above,*
*(x− y)*^{2}*+ 2t(x• y) ∈ ∂IK ⇐⇒ |λ + (t − 1)µ| = ∥x*^{′}*+ (t− 1)y*^{′}*∥, |µ| = ∥y*^{′}*∥,*

*µ(λ + (t− 1)µ) = ⟨x*^{′}*+ (t− 1)y*^{′}*, y*^{′}*⟩.*

Taking into account that *|µ| = ∥y*^{′}*∥ implies the following equivalences*

*|λ + (t − 1)µ| = ∥x*^{′}*+ (t− 1)y*^{′}*∥ ⇐⇒ λ*^{2} *+ 2(t− 1)λµ = ∥x*^{′}*∥*^{2}*+ 2(t− 1)⟨x*^{′}*, y*^{′}*⟩,*
*µ(λ + (t− 1)µ) = ⟨x*^{′}*+ (t− 1)y*^{′}*, y*^{′}*⟩ ⇐⇒ λµ = ⟨x*^{′}*, y*^{′}*⟩,*

we immediately obtain (17). Thus, the proof is complete. *2*

The following lemma is essentially proved in [6, Lemma 3]. We give a simpler proof.

**Lemma 3.3 For j = 1, 2, let x**_{j}*= x*^{′}_{j}*+ λ*_{j}*e∈ H. If λ*1*x*^{′}_{1}*+ λ*_{2}*x*^{′}_{2} *̸= 0, then for j = 1, 2,*
(

*λ** _{j}* + (

*−1)*

^{j}⟨ *λ*_{1}*x*^{′}_{1}*+ λ*_{2}*x*^{′}_{2}

*∥λ*1*x*^{′}_{1}*+ λ*_{2}*x*^{′}_{2}*∥, x*^{′}_{j}

⟩)2

*≤*

*
x*^{′}* ^{j}* + (

*−1)*

^{j}*λ*

_{j}*λ*

_{1}

*x*

^{′}_{1}

*+ λ*

_{2}

*x*

^{′}_{2}

*∥λ*1*x*^{′}_{1}*+ λ*_{2}*x*^{′}_{2}*∥*
^{2}

*≤ ∥x*1*∥*^{2}+*∥x*2*∥*^{2}+ 2(*−1)*^{j}*∥λ*1*x*^{′}_{1}*+ λ*_{2}*x*^{′}_{2}*∥.*

* Proof. It suﬃces to prove the inequalities for j = 1. The ﬁrst inequality holds trivially*
since

*|⟨v, w⟩| ≤ ∥v∥ · ∥w∥ for all v, w ∈ H. The second inequality is proved as follows.*

*
x*^{′}^{1}*− λ*1

*λ*_{1}*x*^{′}_{1}*+ λ*_{2}*x*^{′}_{2}

*∥λ*1*x*^{′}_{1}*+ λ*_{2}*x*^{′}_{2}*∥*

^{2} = *∥x** ^{′}*1

*∥*

^{2}

*−*2

*∥λ*1*x*^{′}_{1}*+ λ*_{2}*x*^{′}_{2}*∥⟨λ*1*x*^{′}_{1}*, λ*1*x*^{′}_{1}*+ λ*2*x*^{′}_{2}*⟩ + λ*^{2}1

= *∥x*1*∥*^{2} *− 2∥λ*1*x*^{′}_{1}*+ λ*_{2}*x*^{′}_{2}*∥ +*2*⟨λ*2*x*^{′}_{2}*, λ*_{1}*x*^{′}_{1} *+ λ*_{2}*x*^{′}_{2}*⟩*

*∥λ*1*x*^{′}_{1}*+ λ*_{2}*x*^{′}_{2}*∥*

*≤ ∥x*1*∥*^{2} *− 2∥λ*1*x*^{′}_{1}*+ λ*_{2}*x*^{′}_{2}*∥ + 2|λ*2*|∥x*^{′}_{2}*∥*

*≤ ∥x*1*∥*^{2} *− 2∥λ*1*x*^{′}_{1}*+ λ*_{2}*x*^{′}_{2}*∥ + ∥x*2*∥*^{2}*,*

where the last inequality is using*∥x*2*∥*^{2} *= λ*^{2}_{2}+*∥x** ^{′}*2

*∥*

^{2}. Thus, the proof is complete.

*2*

To end the contents of this section, we recall the concept of F(r´echet)-diﬀerentiability
and present some continuously F-diﬀerentiable mappings for later use. For given Banach
spaces *X and Y, a mapping f from a nonempty open subset X of X into Y is said to*
*be F-diﬀerentiable at x∈ X if there exists l**x* *∈ L(X , Y) such that*

*h→0*lim

*f (x + h)− f(x) − l**x**h*

*∥h∥* *= 0,*

*and l*_{x}*is called the F-diﬀerential of f at x, written by f*^{′}*(x). When f is F-diﬀerentiable*
*at every point of X, we say that f is F-diﬀerentiable on X. If f is F-diﬀerentiable*

*on a neighborhood U* *⊂ X of a point x*0 *∈ X, and if, as a mapping from U into*
the Banach space *L(X , Y), the mapping x 7−→ f*^{′}*(x) is continuous at x*_{0}*, then f is*
*said to be continuously F-diﬀerentiable at x*_{0}*. The mapping f is called continuously*
*F-diﬀerentiable on X if it is continuously F-diﬀerentiable at every point of X. Note*
*that if f* *∈ L(X , Y), then f is continuously F-diﬀerentiable on X with f*^{′}*(x) = f for*
*every x* *∈ X , i.e., f*^{′}*(x)v = f (v) for all v∈ X . By the deﬁnition, it is easy to verify the*
continuous F-diﬀerentiability of the mappings given below.

**Example 3.1 (i) f (x) =**⟨x, e⟩ for any x ∈ H with f^{′}*(x)v =⟨v, e⟩ for all v ∈ H.*

**(ii) f (x) = x**− ⟨x, e⟩e for any x ∈ H with f^{′}*(x)v = v− ⟨v, e⟩e for all v ∈ H.*

**(iii) f (x) = x**^{2} *= x• x for any x ∈ H with f*^{′}*(x)v = 2x• v for all v ∈ H.*

**(iv) f (x) =**∥x∥^{2} *for any x∈ H with f*^{′}*(x)v = 2⟨x, v⟩ for all x, v ∈ H.*

**(v) f (x) =**∥x∥ = ⟨x, x⟩^{1/2}*for any x∈ H. Such f is continuously F-diﬀerentiable only*
*on* *H \ {0} with f*^{′}*(x)v =* _{∥x∥}^{1} *⟨x, v⟩ for all v ∈ H.*

**4** **Smoothness of merit function**

This section is devoted to establishing the continuous F-diﬀerentiability (smoothness) of
Ψ*t*. For this purpose, we ﬁrst investigate the F-diﬀerentiability of two special mappings
deﬁned as in the following two lemmas, respectively.

**Lemma 4.1 Let σ(x) := x**^{1/2}*for any x∈ IK. Then, the following statements hold.*

*(a) σ is continuously F-diﬀerentiable on intIK, and for all v∈ H,*

*σ*^{′}*(x)v =*

√*λ*^{2} *− ∥x*^{′}*∥*^{2}

*2τ* *⟨x*^{−1/2}*, v⟩x** ^{−1/2}*+

*v− ⟨v, e⟩e*

*2τ*

*where τ is given as in (9).*

*(b) For every x∈ intIK, 2σ*^{′}*(x)v = L*^{−1}_{σ(x)}*v for all v* *∈ H.*

*(c) For every x* *∈ intIK, the F-diﬀerential σ*^{′}*(x) is a self-adjoint operator in* *L(H),*
*i.e.,* *⟨σ*^{′}*(x)v, w⟩ = ⟨v, σ*^{′}*(x)w⟩ for all v, w ∈ H.*

**Proof. (a) Recall that σ(x) =**_{2τ}^{x}^{′}*+ τ e for x = x*^{′}*+ λe∈ IK \ {0}. Since τ as a mapping*
*of x* *∈ IK \ {0} is F-diﬀerentiable on intIK, the function σ is F-diﬀerentiable on intIK.*

*The diﬀerential of σ is computed as follows. Taking into account 2τ*^{2} *= λ +*√

*λ*^{2}*− ∥x*^{′}*∥*^{2}*,*
*by Example 3.1 it is not hard to calculate that for all v* *∈ H,*

*4τ τ*^{′}*(x)v =* *⟨v, e⟩ +* *λ⟨v, e⟩ − ⟨v, x*√ ^{′}*⟩*

*λ*^{2}*− ∥x*^{′}*∥*^{2} = *⟨v, 2τ*√ ^{2}*e− x*^{′}*⟩*
*λ*^{2}*− ∥x*^{′}*∥*^{2}*,*
and consequently,

*τ*^{′}*(x)v =* 1
2√

*λ*^{2}*− ∥x*^{′}*∥*^{2}

⟨

*τ e−* *x*^{′}*2τ, v*

⟩

= 1 2

⟨*x*^{−1/2}*, v*⟩
*.*

*Together with the expression σ(x) =* _{2τ}^{x}^{′}*+ τ e, we obtain that*
*σ*^{′}*(x)v =* *−τ*^{′}*(x)v*

*2τ*^{2} *x** ^{′}*+ 1

*2τ* *(v− ⟨v, e⟩e) + (τ*^{′}*(x)v) e*

=

⟨*x*^{−1/2}*, v*⟩
*2τ*

(*−x*^{′}*2τ* *+ τ e*

) + 1

*2τ* *(v− ⟨v, e⟩e)* (18)

=

⟨*x*^{−1/2}*, v*⟩
*2τ* *·*√

*λ*^{2}*− ∥x*^{′}*∥*^{2}*x** ^{−1/2}*+ 1

*2τ* *(v− ⟨v, e⟩e) .*

*We next prove that the F-diﬀerential σ*^{′}*is continuous at any given point a = a*^{′}*+ αe* *∈*
*intIK. For any x = x*^{′}*+ λe∈ intIK, we write*

*τ (x) = τ =*

√

*λ +*√

*λ*^{2}*− ∥x*^{′}*∥*^{2}

2 and *p(x) =*

√*λ*^{2}*− ∥x*^{′}*∥*^{2}
*2τ (x)* *.*
*Then, from the last equality in (18), it follows that for all v* *∈ H,*

*∥σ*^{′}*(x)v− σ*^{′}*(a)v∥*

*≤*
*p(x)⟨x*^{−1/2}*, v⟩x*^{−1/2}*− p(a)⟨a*^{−1/2}*, v⟩a** ^{−1/2}*
+
1

*2τ (x)−* 1
*2τ (a)*

* · ∥v − ⟨v, e⟩e∥*

*≤ |p(x) − p(a)| ·⟨x*^{−1/2}*, v⟩ ·
x*^{−1/2}*+ p(a)·⟨x*^{−1/2}*− a*^{−1/2}*, v⟩ ·
x*^{−1/2}*+p(a)·⟨a*^{−1/2}*, v⟩ ·
x*^{−1/2}*− a** ^{−1/2}*
+

1

*2τ (x)* *−* 1
*2τ (a)*

* · ∥v∥*

*≤ |p(x) − p(a)| · ∥x*^{−1/2}*∥*^{2}*· ∥v∥ +*
1

*2τ (x)* *−* 1
*2τ (a)*

* · ∥v∥*

*+p(a)·*
*x*^{−1/2}*− a** ^{−1/2}*
(

*x*

*+*

^{1/2}*a*

^{−1/2}*) · ∥v∥.*

This implies that

*∥σ*^{′}*(x)− σ*^{′}*(a)∥ ≤ |p(x) − p(a)| ·*
*x*^{−1/2}^{2}+
1

*2τ (x)* *−* 1
*2τ (a)*

*+p(a)·*
*x*^{−1/2}*− a*^{−1/2}*
(∥x*^{1/2}*∥ + ∥a*^{−1/2}*∥*)

*,*

and consequently *∥σ*^{′}*(x)− σ*^{′}*(a)∥ → 0 as x → a.*

*(b) From the second equality in (18) and equation (12), we obtain for any v = v*^{′}*+θe∈ H,*
*2σ*^{′}*(x)v =* 1

*τ⟨σ(x)*^{−1}*, v⟩ ·*
(*−x*^{′}

*2τ* *+ τ e*
)

+*v*^{′}*τ*

= 1

*τ*
(

*v*^{′}*− ⟨σ(x)*^{−1}*, v⟩ ·* *x*^{′}*2τ*

)

+*⟨σ(x)*^{−1}*, v⟩e = L*^{−1}_{σ(x)}*v.*

*(c) For any given v, w* *∈ H, we write σ*^{′}*(x)v = v*_{1} *and σ*^{′}*(x)w = w*_{1}. Then, by part (b),
*we have v = 2σ(x)• v*1 *and w = 2σ(x)• w*1, and consequently

*⟨σ*^{′}*(x)v, w⟩ = 2⟨v*1*, σ(x)• w*1*⟩ = 2⟨σ(x) • v*1*, w*_{1}*⟩ = ⟨v, σ*^{′}*(x)w⟩.*

*This shows that σ*^{′}*(x) is a self-adjoint operator. The proof is completed.* *2*

**Lemma 4.2 For any x, y***∈ H and r ∈ IR, let ψ**r**(x, y) := 2⟨(x*^{2}*+ y*^{2})^{1/2}*, x + ry⟩. Then,*
**(a) ψ***r* *is F-diﬀerentiable at every point (a, b)∈ H × H with a*^{2}*+ b*^{2} *∈ ∂IK.*

**(b) For any given x = x**^{′}*+ λe∈ H and y = y*^{′}*+ µe* *∈ H with x*^{2}*+ y*^{2} *∈ ∂IK \ {0},*
*ψ*^{′}_{r}*(x, y)(v, w) =* *2(λ + rµ)*

√*λ*^{2}*+ µ*^{2} *· (⟨v, x⟩ + ⟨w, y⟩) + 2*⟨(

*x*^{2}*+ y*^{2})*1/2*

*, v + rw*

⟩

*for all v, w∈ H. Furthermore, ∥ψ*^{′}*r**(x, y)∥ ≤ 4(1 + |r|)*√

*∥x∥*^{2}+*∥y∥*^{2}*.*

**Proof. (a) For any (x, y)***̸= (0, 0), it can be seen that ψ**r* *is F-diﬀerentiable at (0, 0)*
since

*|ψ**r**(x, y)− ψ**r**(0, 0)| = 2*⟨*(x*^{2} *+ y*^{2})^{1/2}*, x + ry⟩ ≤* 2√

*∥x∥*^{2}+*∥y∥*^{2}*· ∥x + ry∥.*

*Next, we consider the case where (a, b)* *̸= (0, 0). Write a = a*^{′}*+ αe and b = b*^{′}*+ βe.*

*Since a*^{2} *+ b*^{2} *∈ ∂IK, we have 2∥αa*^{′}*+ βb*^{′}*∥ = ∥a∥*^{2} +*∥b∥*^{2} *> 0. So, there exist a convex*
*and bounded open neighborhood U of (a, b) in* *H × H and a constant ρ > 0 such that*

*∥λx*^{′}*+ βy*^{′}*∥ ≥ ρ for any (x, y) ∈ U with x = x*^{′}*+ λe and y = y*^{′}*+ µe. Notice that*

*(x*^{2}*+ y*^{2})* ^{1/2}* =

*λx*

^{′}*+ µy*

^{′}*τ (x, y)* *+ τ (x, y)e*
where

*τ (x, y) =*

√

*∥x∥*^{2}+*∥y∥*^{2}+√

(*∥x∥*^{2}+*∥y∥*^{2})^{2}*− 4∥λx*^{′}*+ µy*^{′}*∥*^{2}

2 *.*

Write

*τ*_{j}*= τ*_{j}*(x, y) :=* *∥x∥*^{2} +*∥y∥*^{2}+ 2(*−1)*^{j}*∥λx*^{′}*+ µy*^{′}*∥ for j = 1, 2.*

It is not diﬃcult to verify that
*τ (x, y) =*

*√τ*_{1}+*√*
*τ*_{2}

2 and 1

*τ (x, y)* =

*√τ*_{2}*− √τ*1

2*∥λx*^{′}*+ µy*^{′}*∥.* (19)
Consequently,

*ψ*_{r}*(x, y)* = 2

⟨*λx*^{′}*+ µy*^{′}

*τ (x, y)* *, x*^{′}*+ ry*^{′}

⟩

*+ 2τ (x, y)(λ + rµ)*

= (*√*

*τ*_{2}*−√*
*τ*_{1})

⟨ *λx*^{′}*+ µy*^{′}

*∥λx*^{′}*+ µy*^{′}*∥, x*^{′}*+ ry*^{′}

⟩
+ (*√*

*τ*_{1}+*√*

*τ*_{2}*) (λ + rµ)*
*:= φ*_{1}*(x, y) + φ*_{2}*(x, y)*

where

*φ*_{j}*(x, y) :=*

√

*τ*_{j}*(x, y)*
(

*λ + rµ + (−1)*^{j}

⟨ *λx*^{′}*+ µy*^{′}

*∥λx*^{′}*+ µy*^{′}*∥, x*^{′}*+ ry*^{′}

⟩)

*for j = 1, 2.*

*Since λx*^{′}*+ µy*^{′}*̸= 0 for any (x, y) ∈ U, the mappings*

*(x, y)7−→ ∥λx*^{′}*+ µy*^{′}*∥ and (x, y) 7−→ ∥λx*^{′}*+ µy*^{′}*∥*^{−1}*are continuously F-diﬀerentiable on U , and then*√

*τ*2*(x, y) is continuously F-diﬀerentiable*
*on U since τ*_{2}*(x, y) > 0 for (x, y)̸= (0, 0). Hence, φ*2is continuously Fr´echet diﬀerentiable
*on U . To prove that φ*_{1} *is F-diﬀerentiable at (a, b), we let*

*f (x, y) := λ + rµ,* *g(x, y) := λx*^{′}*+ µy*^{′}*,* *p(x, y) :=* *g(x, y)*

*∥g(x, y)∥,*
*h(x, y) := x*^{′}*+ ry*^{′}*,* *φ*_{3}*(x, y) := f (x, y)− ⟨p(x, y), h(x, y)⟩*

*for any (x, y)∈ U with x = x*^{′}*+ λe and y = y*^{′}*+ µe. Then,*
*τ*_{1}*(x, y) =* *∥x∥*^{2}+*∥y∥*^{2}*− 2∥g(x, y)∥ and φ*1*(x, y) =* √

*τ*_{1}*(x, y) φ*_{3}*(x, y).* (20)
*By Example 3.1, it is not hard to calculate that for any (v, w)∈ H × H,*

*f*^{′}*(x, y)(v, w) =* *⟨v, e⟩ + r⟨w, e⟩;*

*g*^{′}*(x, y)(v, w) = λv +⟨v, e⟩(x*^{′}*− λe) + µw + ⟨w, e⟩(y*^{′}*− µe);*

*p*^{′}*(x, y)(v, w) =* *g*^{′}*(x, y)(v, w)*

*∥g(x, y)∥* *−⟨g*^{′}*(x, y)(v, w), g(x, y)⟩*

*∥g(x, y)∥*^{3} *g(x, y),*
*h*^{′}*(x, y)(v, w) = v− ⟨v, e⟩e + rw − r⟨w, e⟩e.*

Note that *∥g(x, y)∥ ≥ ρ for all (x, y) ∈ U. By the boundedness of U, there is a constant*
*c > 0 such that* *∥x∥ + ∥y∥ ≤ c for all (x, y) ∈ U. Thus, for any (x, y) ∈ U and any*
*(v, w)∈ H × H, from the last four equalities it follows that*

*∥f*^{′}*(x, y)(v, w)∥ ≤ (|r| + 1)(∥v∥ + ∥w∥), ∥h*^{′}*(x, y)(v, w)∥ ≤ (|r| + 1)(∥v∥ + ∥w∥),*

*∥g*^{′}*(x, y)(v, w)∥ ≤ 2c(∥v∥ + ∥w∥), ∥p*^{′}*(x, y)(v, w)∥ ≤* *4c(∥v∥ + ∥w∥)*

*ρ* *.*

Consequently,

*∥τ*1^{′}*(x, y)(v, w)∥ =*

*
2⟨x,v⟩ + 2⟨y,w⟩ −* 2*⟨g*^{′}*(x, y)(v, w), g(x, y)⟩*

*∥g(x, y)∥*

*
,*

*≤ 2c(∥v∥ + ∥w∥) + 2∥g*^{′}*(x, y)(v, w)∥ ≤ 6c(∥v∥ + ∥w∥),*

*|φ** ^{′}*3

*(x, y)(v, w)| ≤ ∥f*

^{′}*(x, y)(v, w)∥ + ∥p*

^{′}*(x, y)(v, w)∥ · ∥h(x, y)∥ + ∥h*

^{′}*(x, y)(v, w)∥*

*≤ M*1(*∥v∥ + ∥w∥)*

*where M*1 = 2(*|r| + 1) + 4ρ*^{−1}*c*^{2}(*|r| + 1). By the mean-value theorem, for any given*
*(x, y)∈ U, there exists (¯x, ¯y) ∈ U on the line segment joining (a, b) to (x, y) such that*

*|φ*3*(x, y)− φ*3*(a, b)| = |φ** ^{′}*3(¯

*x, ¯y)(x− a, y − b)| ≤ M*1(

*∥x − a∥ + ∥y − b∥).*

*We claim that φ*_{3}*(a, b) = 0. To see this, from Lemma 3.2,* *|α| = ∥a*^{′}*∥, |β| = ∥b*^{′}*∥ and*
*αβ =⟨a*^{′}*, b*^{′}*⟩, which implies that ∥αa*^{′}*+ βb*^{′}*∥ = α*^{2}*+ β*^{2}, and

*φ*3*(a, b) = α + rβ−* 1

*α*^{2}*+ β*^{2}*(α∥a*^{′}*∥*^{2}*+ (rα + β)⟨a*^{′}*, b*^{′}*⟩ + rβ∥b*^{′}*∥*^{2})

*= α + rβ−* 1

*α*^{2}*+ β*^{2}*(α*^{3}*+ rα*^{2}*β + αβ*^{2} *+ rβ*^{3}*) = 0.*

This claim implies that

*|φ*3*(x, y)| ≤ M*1(*∥x − a∥ + ∥y − b∥) for any (x, y) ∈ U.* (21)
*In addition, noting that τ*_{1}*(a, b) = 0 and applying the Mean Value Theorem to τ*_{1},

√*τ*_{1}*(x, y)≤ M*2*·*√

*∥x − a∥ + ∥y − b∥ for any (x, y) ∈ U,* (22)
*where M*_{2} =*√*

*6c. Now from equations (20)–(22) it follows that, for any (x, y)∈ U,*

*|φ*1*(x, y)− φ*2*(a, b)| = |φ*1*(x, y)| ≤ M*1*M*_{2}(*∥x − a∥ + ∥y − b∥)*^{3/2}*,*

*which says that φ*_{1} *is F-diﬀerentiable at (a, b) with φ*^{′}_{1}*(a, b) being the zero mapping in*
*L(H × H, IR). So, ψ**r* *is F-diﬀerentiable at (a, b) with ψ*_{r}^{′}*(a, b) = φ*^{′}_{2}*(a, b).*

*(b) From part (a), we know that ψ*^{′}_{r}*(x, y) = φ*^{′}_{2}*(x, y). To compute φ*^{′}_{2}*(x, y), we write*
*φ*_{4}*(x, y) := f (x, y) +⟨p(x, y), h(x, y)⟩ = λ + rµ +*

⟨ *λx*^{′}*+ µy*^{′}

*∥λx*^{′}*+ µy*^{′}*∥, x*^{′}*+ ry*^{′}

⟩
*.*