Smoothing methods for Second-Order Cone Programs/Complementarity Problems

Programs/Complementarity Problems

Paul Tseng

University of Washington, Seattle

SIAM Conf. Optim May, 2005

Abstract

This is a talk given at SIAM Conf. Optim, Stockholm, May 2005.

Talk Outline

• I. Second-Order Cone (SOC) Program and Complementarity Problem

• Unconstrained Diff. Min. Reformulation

• Numerical Experience

• II. SOCP from Dist. Geometry Optim

• Simulation Results

Convex SOCP

min g(x) s.t. Ax = b

x ∈ K

A ∈ <^m×n, b ∈ <^m

g : <ⁿ → <, convex, twice cont. diff.

K = Kⁿ¹ × · · · × K^np K^{ni def}= n

x_i = h

x_1i x_2i

i ∈ < × <ⁿⁱ⁻¹ : kx_2ik₂ ≤ x_1io

Special cases? LP, SOCP,...

SOC K

Suff. Optim. Conditions

x ∈ K, y ∈ K, x^Ty = 0, Ax = b, y = ∇g(x) − A^Tζ_d

⇐⇒

x ∈ K, y ∈ K, x^Ty = 0, x = F (ζ), y = G(ζ) with F (ζ) = d + (I − A^T(AA^T)⁻¹A)ζ

G(ζ) = ∇g(F (ζ)) − A^T(AA^T)⁻¹Aζ (Ad = b)

SOCCP

Find ζ ∈ <ⁿ satisfying

x ∈ K, y ∈ K, x^Ty = 0, x = F (ζ), y = G(ζ) F, G : <ⁿ → <ⁿ smooth

∇F (ζ), −∇G(ζ) column-monotone ∀ζ ∈ <ⁿ, i.e.,

∇F (ζ)u − ∇G(ζ)v = 0 ⇒ u^Tv ≥ 0

Special cases? convex SOCP, monotone NCP,...

How to solve SOCCP?

For LP, simplex methods and interior-point methods.

For SOCP, interior-point methods.

For convex SOCP and column-monotone SOCCP?

Interior-point methods not amenable to warm start. Non-interior methods?

Nonsmooth Eq. Reformulation

x_i · y_i ^def= h

x_1i x_2i

i · h

y_1i y_2i

i

= h

x^T_i y_i x_1iy_2i+y_1ix_2i

i

(Jordan product assoc. with Kⁿⁱ)

φ_FB(x, y) ^def= h

(x²_i + y_i²)^1/2 − x_i − y_ii^p

i=1

Fact (Fukushima,Luo,T ’02):

φ_FB(x, y) = 0 ⇐⇒ x ∈ K, y ∈ K, x^Ty = 0

Thus, SOCCP is equivalent to

φ_FB(F (ζ), G(ζ)) = 0

φ_FB is strongly semismooth (Sun,Sun ’03)

Unconstr. Smooth Min. Reformulation

min f_FB(ζ) ^def= kφ_FB(F (ζ), G(ζ))k²

2

F, G smooth and ∇F (ζ), −∇G(ζ) column-monotone ∀ζ ∈ <ⁿ (e.g., LP, SOCP, convex SOCP, monotone NCP)

For monotone NCP (K = <ⁿ₊),

f_FB is smooth, and ∇f_FB(ζ) = 0 ⇐⇒ ζ is a soln

(Geiger,Kanzow ’96)

The same holds for SOCCP. (J.-S. Chen,T ’04)

Advantage? Any method for unconstrained diff. min. (e.g., CG, BFGS, L-BFGS) can be used to find ∇f_FB(ζ) = 0.

Numerical Experience on Convex SOCP

x = F (ζ) = d + (I − P )ζ y = G(ζ) = ∇g(F (ζ)) − P ζ

with P = A^T(AA^T)⁻¹A, Ad = b. ^(Solve min kAx − bk to find d)

• Implement in Matlab CG-PR, BFGS, L-BFGS (memory=5) to minimize f_FB(ζ), using Armijo stepsize rule, with ζ^init = 0. Stop when

max{f_FB(ζ), |x^Ty|} ≤ accur.

• Let ψ_FB(x, y) ^def= kφ_FB(x, y)k²

2. Then f_FB(ζ) = ψ_FB(x, y)

∇f_FB(ζ) = (I − P )∇_xψ_FB(x, y) − P ∇_yψ_FB(x, y)

Compute P ζ using Cholesky factorization of AA^T or using preconditioned CG. Compute ψ_FB(x, y) and ∇ψ_FB(x, y) within Fortran Mex files.

DIMACS Challenge SOCPs

• Problem names and statistics:

nb (m = 123, n = 2383, K = (K³)⁷⁹³ × <⁴₊)

nb-L2 (m = 123, n = 4195, K = K¹⁶⁷⁷ × (K³)⁸³⁸ × <⁴₊)

nb-L2-bessel (m = 123, n = 2641, K = K¹²³ × (K³)⁸³⁸ × <⁴₊)

Compare iters/cpu(sec)/accuracy with Sedumi 1.05 (Sturm ’01), which implements a predictor-corrector interior-point method.

Problem SeDuMi (pars.eps=1e-5) L-BFGS-Chol (accur=1e-5)

Name iter/cpu iter/cpu

nb 19/7.6 1042/16.5

nb-L2 11/11.1 330/9.2

nb-L2-bessel 11/5.3 108/1.7

Table 1: (cpu times are in sec on an HP DL360 workstation, running Matlab 6.1)

Regularized Sum-of-Norms Problems

min_w≥0 PM

i=1 kA_iw − b_ik₂ + h(w),

A_i ∼ U[−1, 1]^mi×`, b_i ∼ U[−5, 5]^mi, m_i ∼ U{2, 3, ..., r} (r ≥ 2).

h(w) = 1^Tw + ¹₃kwk³₃ (cubic reg.) Reformulate as a convex SOCP:

minimize PM

i=1 z_i + h(w)

subject to A_iw + s_i = b_i, (z_i, s_i) ∈ K^mi+1, _i=1,...,M, w ∈ <^`₊.

Problem BFGS-Chol CG-PR-Chol L-BFGS-Chol

`, M, r (m, n) iter/cpu iter/cpu iter/cpu 500,10,10 (56,566) 352/24.6 1703/6.6 497/2.4 500,50,10 (283,833) 546/85.1 3173/69.0 700/12.4 500,10,50 (246,756) 272/36.3 1290/23.0 371/5.6

Table 2: (cpu times are in sec on an HP DL360 workstation, running Matlab 6.5.1, with accur=1e-3)

Smoothing Newton Step

φ^µ

FB(x, y) ^def= (x² + y² + µ²e)^1/2 − x − y

with e = (1, 0, .., 0

| {z }

n₁

, ..., 1, 0, .., 0

| {z }

n_p

)^T, µ > 0 (Fukushima,Luo,T ’02)

Given ζ, choose µ > 0 and solve

∇φ^µ

FB(F (ζ), G(ζ))^T∆ζ = −φ_FB(F (ζ), G(ζ)) Use ∆ζ to accelerate convergence.

This requires more work per iteration. Use it judiciously.

Observations

For our unconstrained smooth merit function approach:

Advantage:

• Less work/iteration, simpler matrix computation than interior-point methods.

• Applicable to convex SOCP and column-monotone SOCCP.

• Useful for warm start?

Drawback:

• Many more iters. than interior-point methods.

• Lower solution accuracy.

SOCP from Dist. Geometry Optim

n pts in <^d (d = 2, 3).

Know x_m+1, ..., x_n and Eucl. dist. estimate for pairs of ‘neighboring’ pts d_ij > 0 ∀(i, j) ∈ A ⊆ {1, ..., n} × {1, ..., n}.

Estimate x₁, ..., x_m.

Problem (nonconvex):

x₁min,...,x_m

X

(i,j)∈A

kx_i − x_jk²₂ − d²_ij 

Convex relaxation:

x₁min,...,x_m

X

(i,j)∈A

max{0, kx_i − x_jk²₂ − d²_ij}

This is an unconstrained (nonsmooth) convex program, can be reformulated as an SOCP. Alternatives?

Smooth approx.:

max{0, t} ≈ µh  t µ



(µ > 0) h smooth convex, lim

t→−∞h(t) = lim

t→∞h(t) − t = 0.

We use h(t) = ((t² + 4)^1/2 + t)/2 (CHKS).

Smooth Approximation of Convex Relaxation

x₁min,...,x_m f_µ(x₁, .., x_m) ^def= X

(i,j)∈A

µh kx_i − x_jk² − d²_ij µ

!

Solve the smooth approximation using Inexact Block Coordinate Descent:

• If k∇_xif_µk = Ω(µ), then update x_i by moving it along the Newton direction

−[∇²_x

ix_if_µ]⁻¹∇_xif_µ, with Armijo stepsize rule, and re-iterate.

• Decrease µ when k∇_xif_µk = O(µ) ∀i.

µ^init = 1e − 3. µ^end = 2e − 6. Decrease µ by a factor of 5. Code in Matlab.

Simulation Results

Uniformly generate x˜₁, ..., ˜x_n in [−.5, .5]², m = 0.9n two pts are nhbrs if dist< .06.

Set d_ij = k˜x_i − ˜x_jk (Biswas, Ye ’03)

SeDuMi Inexact BCD

n SOCP dim cpu/Err cpu/Err

1000 21472 × 33908 330/.48 373/.48 2000 84440 × 130060 12548/.57 2090/.52

Table 3: (cpu times are in secs on a Linux PC cluster, running Matlab 6.1.) Err = Pm

i=1 kx_i − ˜x_ik²₂.

True soln (m = 900, n = 1000)

SOCP soln found by SeDuMi SOCP soln found by Inexact BCD

Observations

For our smoothing-Inexact BCD approach:

• Better cpu time than using SeDuMi.

Add barrier term to find analytic center soln.

• Computation easily distributes.

• Code in Fortran (instead of Matlab) to improve time?

Lastly...

Thanks, Christian, for lending the use of your laptop!

6. .

^