**Programs/Complementarity Problems**

Paul Tseng

University of Washington, Seattle

SIAM Conf. Optim May, 2005

**Abstract**

This is a talk given at SIAM Conf. Optim, Stockholm, May 2005.

**Talk Outline**

• I. Second-Order Cone (SOC) Program and Complementarity Problem

• Unconstrained Diff. Min. Reformulation

• Numerical Experience

• II. SOCP from Dist. Geometry Optim

• Simulation Results

**Convex SOCP**

min g(x) s.t. Ax = b

x ∈ K

A ∈ <^{m×n}, b ∈ <^{m}

g : <^{n} → <, convex, twice cont. diff.

K = K^{n}^{1} × · · · × K^{n}^{p}
K^{n}^{i} ^{def}= n

x_{i} = h

x_{1i}
x_{2i}

i ∈ < × <^{n}^{i}^{−1} : kx_{2i}k_{2} ≤ x_{1i}o

**Special cases?** LP, SOCP,...

**SOC** K

^{n}

**Suff. Optim. Conditions**

x ∈ K, y ∈ K, x^{T}y = 0,
Ax = b, y = ∇g(x) − A^{T}ζ_{d}

⇐⇒

x ∈ K, y ∈ K, x^{T}y = 0,
x = F (ζ), y = G(ζ)
with F (ζ) = d + (I − A^{T}(AA^{T})^{−1}A)ζ

G(ζ) = ∇g(F (ζ)) − A^{T}(AA^{T})^{−1}Aζ (Ad = b)

**SOCCP**

Find ζ ∈ <^{n} satisfying

x ∈ K, y ∈ K, x^{T}y = 0,
x = F (ζ), y = G(ζ)
F, G : <^{n} → <^{n} smooth

∇F (ζ), −∇G(ζ) column-monotone ∀ζ ∈ <^{n}, i.e.,

∇F (ζ)u − ∇G(ζ)v = 0 ⇒ u^{T}v ≥ 0

**Special cases?** convex SOCP, monotone NCP,...

**How to solve SOCCP?**

For LP, simplex methods and interior-point methods.

For SOCP, interior-point methods.

For convex SOCP and column-monotone SOCCP?

Interior-point methods not amenable to warm start. Non-interior methods?

**Nonsmooth Eq. Reformulation**

x_{i} · y_{i} ^{def}= h

x_{1i}
x_{2i}

i · h

y_{1i}
y_{2i}

i

= h

x^{T}_{i} y_{i}
x_{1i}y_{2i}+y_{1i}x_{2i}

i

(Jordan product assoc. with K^{ni})

φ_{FB}(x, y) ^{def}= h

(x^{2}_{i} + y_{i}^{2})^{1/2} − x_{i} − y_{i}i^{p}

i=1

Fact (Fukushima,Luo,T ’02):

φ_{FB}(x, y) = 0 ⇐⇒ x ∈ K, y ∈ K, x^{T}y = 0

Thus, SOCCP is equivalent to

φ_{FB}(F (ζ), G(ζ)) = 0

φ_{FB} is strongly semismooth (Sun,Sun ’03)

**Unconstr. Smooth Min. Reformulation**

min f_{FB}(ζ) ^{def}= kφ_{FB}(F (ζ), G(ζ))k^{2}

2

F, G smooth and ∇F (ζ), −∇G(ζ) column-monotone ∀ζ ∈ <^{n} (e.g., LP, SOCP,
convex SOCP, monotone NCP)

For monotone NCP (K = <^{n}_{+}),

f_{FB} is smooth, and ∇f_{FB}(ζ) = 0 ⇐⇒ ζ is a soln

(Geiger,Kanzow ’96)

The same holds for SOCCP. (J.-S. Chen,T ’04)

Advantage? Any method for unconstrained diff. min. (e.g., CG, BFGS,
L-BFGS) can be used to find ∇f_{FB}(ζ) = 0.

**Numerical Experience on Convex SOCP**

x = F (ζ) = d + (I − P )ζ y = G(ζ) = ∇g(F (ζ)) − P ζ

with P = A^{T}(AA^{T})^{−1}A, Ad = b. ^{(Solve} min kAx − bk to find d)

• Implement in Matlab CG-PR, BFGS, L-BFGS (memory=5) to minimize
f_{FB}(ζ), using Armijo stepsize rule, with ζ^{init} = 0. Stop when

max{f_{FB}(ζ), |x^{T}y|} ≤ accur.

• Let ψ_{FB}(x, y) ^{def}= kφ_{FB}(x, y)k^{2}

2. Then
f_{FB}(ζ) = ψ_{FB}(x, y)

∇f_{FB}(ζ) = (I − P )∇_{x}ψ_{FB}(x, y) − P ∇_{y}ψ_{FB}(x, y)

Compute P ζ using Cholesky factorization of AA^{T} or using preconditioned
CG. Compute ψ_{FB}(x, y) and ∇ψ_{FB}(x, y) within Fortran Mex files.

### DIMACS Challenge SOCPs

• Problem names and statistics:

nb (m = 123, n = 2383, K = (K^{3})^{793} × <^{4}_{+})

nb-L2 (m = 123, n = 4195, K = K^{1677} × (K^{3})^{838} × <^{4}_{+})

nb-L2-bessel (m = 123, n = 2641, K = K^{123} × (K^{3})^{838} × <^{4}_{+})

Compare iters/cpu(sec)/accuracy with Sedumi 1.05 (Sturm ’01), which implements a predictor-corrector interior-point method.

Problem SeDuMi (pars.eps=1e-5) L-BFGS-Chol (accur=1e-5)

Name iter/cpu iter/cpu

nb 19/7.6 1042/16.5

nb-L2 11/11.1 330/9.2

nb-L2-bessel 11/5.3 108/1.7

Table 1: (cpu times are in sec on an HP DL360 workstation, running Matlab 6.1)

### Regularized Sum-of-Norms Problems

min_{w≥0} PM

i=1 kA_{i}w − b_{i}k_{2} + h(w),

A_{i} ∼ U[−1, 1]^{m}^{i}^{×`}, b_{i} ∼ U[−5, 5]^{m}^{i}, m_{i} ∼ U{2, 3, ..., r} (r ≥ 2).

h(w) = 1^{T}w + ^{1}_{3}kwk^{3}_{3} (cubic reg.)
Reformulate as a convex SOCP:

minimize PM

i=1 z_{i} + h(w)

subject to A_{i}w + s_{i} = b_{i}, (z_{i}, s_{i}) ∈ K^{m}^{i}^{+1}, _{i=1,...,M}, w ∈ <^{`}_{+}.

Problem BFGS-Chol CG-PR-Chol L-BFGS-Chol

`, M, r (m, n) iter/cpu iter/cpu iter/cpu 500,10,10 (56,566) 352/24.6 1703/6.6 497/2.4 500,50,10 (283,833) 546/85.1 3173/69.0 700/12.4 500,10,50 (246,756) 272/36.3 1290/23.0 371/5.6

Table 2: (cpu times are in sec on an HP DL360 workstation, running Matlab 6.5.1, with accur=1e-3)

**Smoothing Newton Step**

φ^{µ}

FB(x, y) ^{def}= (x^{2} + y^{2} + µ^{2}e)^{1/2} − x − y

with e = (1, 0, .., 0

| {z }

n_{1}

, ..., 1, 0, .., 0

| {z }

n_{p}

)^{T}, µ > 0 (Fukushima,Luo,T ’02)

Given ζ, choose µ > 0 and solve

∇φ^{µ}

FB(F (ζ), G(ζ))^{T}∆ζ = −φ_{FB}(F (ζ), G(ζ))
Use ∆ζ to accelerate convergence.

This requires more work per iteration. Use it judiciously.

**Observations**

For our unconstrained smooth merit function approach:

Advantage:

• Less work/iteration, simpler matrix computation than interior-point methods.

• Applicable to convex SOCP and column-monotone SOCCP.

• Useful for warm start?

Drawback:

• Many more iters. than interior-point methods.

• Lower solution accuracy.

**SOCP from Dist. Geometry Optim**

**(ongoing work..)**

n pts in <^{d} (d = 2, 3).

Know x_{m+1}, ..., x_{n} and Eucl. dist. estimate for pairs of ‘neighboring’ pts
d_{ij} > 0 ∀(i, j) ∈ A ⊆ {1, ..., n} × {1, ..., n}.

Estimate x_{1}, ..., x_{m}.

Problem (nonconvex):

x_{1}min,...,x_{m}

X

(i,j)∈A

kx_{i} − x_{j}k^{2}_{2} − d^{2}_{ij}

Convex relaxation:

x_{1}min,...,x_{m}

X

(i,j)∈A

max{0, kx_{i} − x_{j}k^{2}_{2} − d^{2}_{ij}}

This is an unconstrained (nonsmooth) convex program, can be reformulated as an SOCP. Alternatives?

Smooth approx.:

max{0, t} ≈ µh t µ

(µ > 0) h smooth convex, lim

t→−∞h(t) = lim

t→∞h(t) − t = 0.

We use h(t) = ((t^{2} + 4)^{1/2} + t)/2 (CHKS).

**Smooth Approximation of Convex Relaxation**

x_{1}min,...,x_{m} f_{µ}(x_{1}, .., x_{m}) ^{def}= X

(i,j)∈A

µh kx_{i} − x_{j}k^{2} − d^{2}_{ij}
µ

!

Solve the smooth approximation using Inexact Block Coordinate Descent:

• If k∇_{x}_{i}f_{µ}k = Ω(µ), then update x_{i} by moving it along the Newton direction

−[∇^{2}_{x}

ix_{i}f_{µ}]^{−1}∇_{x}_{i}f_{µ}, with Armijo stepsize rule, and re-iterate.

• Decrease µ when k∇_{x}_{i}f_{µ}k = O(µ) ∀i.

µ^{init} = 1e − 3. µ^{end} = 2e − 6. Decrease µ by a factor of 5. Code in Matlab.

**Simulation Results**

Uniformly generate x˜_{1}, ..., ˜x_{n} in [−.5, .5]^{2}, m = 0.9n
two pts are nhbrs if dist< .06.

Set d_{ij} = k˜x_{i} − ˜x_{j}k (Biswas, Ye ’03)

SeDuMi Inexact BCD

n SOCP dim cpu/Err cpu/Err

1000 21472 × 33908 330/.48 373/.48 2000 84440 × 130060 12548/.57 2090/.52

Table 3: (cpu times are in secs on a Linux PC cluster, running Matlab 6.1.) Err = Pm

i=1 kx_{i} − ˜x_{i}k^{2}_{2}.

True soln (m = 900, n = 1000)

SOCP soln found by SeDuMi SOCP soln found by Inexact BCD

**Observations**

For our smoothing-Inexact BCD approach:

• Better cpu time than using SeDuMi.

Add barrier term to find analytic center soln.

• Computation easily distributes.

• Code in Fortran (instead of Matlab) to improve time?

**Lastly...**

### Thanks, Christian, for lending the use of your laptop!

6. .

^