Convex optimization problems

(1)

Convex optimization problems

! optimization problem in standard form

! convex optimization problems

! linear optimization

! quadratic optimization

! geometric programming

! quasiconvex optimization

! generalized inequality constraints

! semidefinite programming

! vector optimization

IOE 611: Nonlinear Programming, Winter 2006 4. Convex optimization problems Page 4–1

Optimization problem in standard form

minimize f0(x)

subject to f_i(x)≤ 0, i = 1, . . . , m h_i(x) = 0, i = 1, . . . , p

! x ∈ Rⁿ is the optimization variable

! f0 : Rⁿ → R is the objective or cost function

! f_i : Rⁿ → R, i = 1, . . . , m, are the inequality constraint functions

! hi : Rⁿ → R are the equality constraint functions optimal value:

p^! = inf{f⁰(x)| fⁱ(x) ≤ 0, i = 1, . . . , m, hⁱ(x) = 0, i = 1, . . . , p}

! p^! = ∞ if problem is infeasible (no x satisfies the constraints)

! p^! = −∞ if problem is unbounded below

(2)

Optimal and locally optimal points

! x is feasible if x ∈ dom f0 and it satisfies the constraints

! a feasible x is optimal if f0(x) = p^!; Xopt is the set of optimal points

! x is locally optimal if there is an R > 0 such that x is optimal for

minimize (over z) f0(z)

subject to fi(z) ≤ 0, i = 1, . . . , m, hi(z) = 0, i = 1, . . . , p

&z − x&2 ≤ R examples (with n = 1, m = p = 0)

! f₀(x) = 1/x, dom f₀ = R₊₊: p^! = 0, no optimal point

! f0(x) = − log x, dom f⁰ = R++: p^! = −∞

! f₀(x) = x log x, dom f₀ = R₊₊: p^! = −1/e, x = 1/e is optimal

! f0(x) = x³ − 3x, p^! = −∞, local optimum at x = 1

Implicit constraints

The standard form optimization problem has an implicit constraint

x ∈ D =

!m i=0

dom f_i ∩

!p i=1

dom h_i,

! we call D the domain of the problem

! the constraints f_i(x)≤ 0, hi(x) = 0 are the explicit constraints

! a problem is unconstrained if it has no explicit constraints (m = p = 0)

example:

minimize f₀(x) =−"_k

i=1log(b_i − a^T_i x) is an unconstrained problem with implicit constraints a^T_i x < bi, i = 1, . . . , k

(3)

Feasibility problem

find x

can be considered a special case of the general problem with f₀(x) = 0:

minimize 0

! p^! = 0 if constraints are feasible; any feasible x is optimal

! p^! = ∞ if constraints are infeasible

Convex optimization problem

standard form convex optimization problem minimize f₀(x)

subject to f_i(x)≤ 0, i = 1, . . . , m a^T_i x = b_i, i = 1, . . . , p

! f₀, f₁, . . . , f_m are convex; equality constraints are affine

! problem is quasiconvex if f0 is quasiconvex (and f1, . . . , fm

convex) often written as

minimize f0(x)

subject to fi(x)≤ 0, i = 1, . . . , m Ax = b

important property: feasible set of a convex optimization problem is convex

(4)

example

minimize f₀(x) = x₁² + x₂²

subject to f1(x) = x1/(1 + x₂²) ≤ 0 h1(x) = (x1 + x2)² = 0

! f0 is convex; feasible set {(x1, x2)| x1 = −x2 ≤ 0} is convex

! not a convex problem (according to our definition): f1 is not convex, h1 is not affine

! equivalent (but not identical) to the convex problem minimize x₁² + x₂²

subject to x₁ ≤ 0 x₁ + x₂ = 0

Local and global optima

Any locally optimal point of a convex problem is (globally) optimal proof: suppose x is locally optimal and y is optimal with

f0(y ) < f0(x).

“x locally optimal” means there is an R > 0 such that z feasible, &z − x&2 ≤ R =⇒ f₀(z) ≥ f0(x).

Consider z = θy + (1− θ)x with θ = R/(2&y − x&²)

! &y − x&2 > R, so 0 < θ < 1/2

! z is a convex combination of two feasible points, hence also feasible

! &z − x&2 = R/2 and

f₀(z)≤ θf0(x) + (1− θ)f0(y ) < f₀(x),

which contradicts our assumption that x is locally optimal

(5)

Optimality criterion for differentiable f

₀

x is optimal if and only if it is feasible and

∇f⁰(x)^T(y − x) ≥ 0 for all feasible y

Optimality criterion for differentiable f

₀

x is optimal if and only if it is feasible and

∇f⁰(x)^T(y− x) ≥ 0 for all feasible y

PSfrag replacements

−∇f₀(x)

X x

if nonzero, ∇f⁰(x) defines a supporting hyperplane to feasible set X at x

Convex optimization problems 4–9

if nonzero, ∇f0(x) defines a supporting hyperplane to feasible set X at x

Optimality criterion: special cases

! unconstrained problem: x is optimal if and only if x ∈ dom f0, ∇f0(x) = 0

! equality constrained problem

minimize f₀(x) subject to Ax = b x is optimal if and only if there exists a ν such that

x ∈ dom f0, Ax = b, ∇f0(x) + A^Tν = 0

! minimization over nonnegative orthant

minimize f0(x) subject to x + 0 x is optimal if and only if

x ∈ dom f⁰, x + 0,

# ∇f⁰(x)_i ≥ 0 xi = 0

∇f⁰(x)i = 0 xi > 0

(6)

Equivalent convex problems

two problems are (informally) equivalent if the solution of one is readily obtained from the solution of the other, and vice-versa.

Some common transformations that preserve convexity:

! eliminating equality constraints minimize f0(x)

is equivalent to

minimize (over z) f₀(Fz + x₀)

subject to f_i(Fz + x0) ≤ 0, i = 1, . . . , m where F and x0 are such that

Ax = b ⇐⇒ x = Fz + x0 for some z

! introducing equality constraints minimize f0(A0x + b0)

subject to fi(Aix + bi) ≤ 0, i = 1, . . . , m is equivalent to

minimize (over x, yi) f0(y0)

subject to fi(yi) ≤ 0, i = 1, . . . , m

yi = Aix + bi, i = 0, 1, . . . , m

! introducing slack variables for linear inequalities minimize f0(x)

subject to a^T_i x ≤ bi, i = 1, . . . , m is equivalent to

minimize (over x, s) f0(x)

subject to a^T_i x + si = bi, i = 1, . . . , m si ≥ 0, i = 1, . . . m

(7)

! epigraph form: standard form convex problem is equivalent to

minimize (over x, t) t

subject to f₀(x)− t ≤ 0

f_i(x)≤ 0, i = 1, . . . , m Ax = b

! minimizing over some variables minimize f0(x1, x2)

subject to fi(x1) ≤ 0, i = 1, . . . , m is equivalent to

minimize f˜0(x1)

subject to fi(x1) ≤ 0, i = 1, . . . , m where ˜f0(x1) = infx₂ f0(x1, x2)

Linear program (LP)

minimize c^Tx + d subject to Gx - h

Ax = b

! convex problem with affine objective and constraint functions

! feasible set is a polyhedron

Linear program (LP)

minimize c^Tx + d subject to Gx ! h Ax = b

• convex problem with affine objective and constraint functions

• feasible set is a polyhedron

PSfrag replacements P x^!

−c

Convex optimization problemsIOE 611: Nonlinear Programming, Winter 2006 4. Convex optimization problems Page 4–14 4–17

(8)

Examples

diet problem: choose quantities x1, . . . , xn of n foods

! one unit of food j costs cj, contains amount aij of nutrient i

! healthy diet requires nutrient i in quantity at least b_i to find cheapest healthy diet,

minimize c^Tx

subject to Ax + b, x + 0

piecewise-linear minimization

minimize maxi=1,...,m(a^T_i x + bi) equivalent to an LP

minimize t

subject to a^T_i x + b_i ≤ t, i = 1, . . . , m

Chebyshev center of a polyhedron

Chebyshev center of

P = {x | a^Ti x ≤ bi, i = 1, . . . , m} is center of largest inscribed ball

B = {xc + u | &u&2 ≤ r}

Chebyshev center of a polyhedron Chebyshev center of

P = {x | a^Ti x≤ bⁱ, i = 1, . . . , m} is center of largest inscribed ball

B = {x^c+ u| "u"² ≤ r}PSfrag replacements

xcheb

• a^Ti x≤ bi for all x∈ B if and only if

sup{a^Ti (xc+ u)| "u"2 ≤ r} = a^Ti xc+ r"ai"2 ≤ bi

• hence, xc, r can be determined by solving the LP maximize r

subject to a^T_i xc+ r"aⁱ"² ≤ bⁱ, i = 1, . . . , m

! a^T_i x ≤ bⁱ for all x ∈ B if and only if

sup{a^Ti (xc + u)| &u&2 ≤ r} = a^Ti xc + r&ai&2 ≤ bi

! hence, xc, r can be determined by solving the LP maximizex_c,r r

subject to a^T_i xc + r&ai&2 ≤ bi, i = 1, . . . , m

(9)

Quadratic program (QP)

minimize (1/2)x^TPx + q^Tx + r subject to Gx - h

Ax = b

! P ∈ Sⁿ+, so objective is convex quadratic

! minimize a convex quadratic function over a polyhedron Quadratic program (QP)

minimize (1/2)x^TP x + q^Tx + r subject to Gx! h

Ax = b

• P ∈ Sⁿ+, so objective is convex quadratic

• minimize a convex quadratic function over a polyhedron

PSfrag replacements

P x^!

−∇f₀(x^!)

Examples

least-squares

minimize &Ax − b&²2

! analytical solution x^! = A^†b (A^† is pseudo-inverse)

! can add linear constraints, e.g., l - x - u linear program with random cost

minimize ¯c^Tx + γx^TΣx = E c^Tx + γ var(c^Tx) subject to Gx - h, Ax = b

! c is random vector with mean ¯c and covariance Σ

! hence, c^Tx is random variable with mean ¯c^Tx and variance x^TΣx

! γ > 0 is risk aversion parameter; controls the trade-off between expected cost and variance (risk)

(10)

Quadratically constrained quadratic program (QCQP)

minimize (1/2)x^TP0x + q₀^Tx + r0

subject to (1/2)x^TP_ix + q_i^Tx + r_i ≤ 0, i = 1, . . . , m Ax = b

! P_i ∈ Sⁿ+; objective and constraints are convex quadratic

! if P1, . . . , Pm ∈ Sⁿ++, feasible region is intersection of m ellipsoids and an affine set

Second-order cone programming

minimize f^Tx

subject to &Aⁱx + bi&² ≤ ci^Tx + di, i = 1, . . . , m Fx = g

(A_i ∈ Rⁿⁱ^×n, F ∈ R^p^×n)

! inequalities are called second-order cone (SOC) constraints:

(Aix + bi, c_i^Tx + di) ∈ second-order cone in Rⁿⁱ⁺¹

! for n_i = 0, reduces to an LP; if c_i = 0, reduces to a QCQP

! more general than QCQP and LP

(11)

Robust linear programming

the parameters in optimization problems are often uncertain, e.g., in an LP

minimize c^Tx

subject to a^T_i x ≤ bi, i = 1, . . . , m, there can be uncertainty in c, a_i, b_i

two common approaches to handling uncertainty (in a_i, for simplicity)

! deterministic model: constraints must hold for all a_i ∈ Ei

minimize c^Tx

subject to a^T_i x ≤ bi for all ai ∈ Ei, i = 1, . . . , m,

! stochastic model: ai is random variable; constraints must hold with probability η

minimize c^Tx

subject to prob(a_i^Tx ≤ bi) ≥ η, i = 1, . . . , m

Deterministic approach via SOCP

! choose an ellipsoid as Ei:

Ei = {¯ai + P_iu | &u&² ≤ 1} (¯a_i ∈ Rⁿ, P_i ∈ Rⁿ^×n) center is ¯ai, semi-axes determined by singular values/vectors of Pi

! robust LP

minimize c^Tx

subject to a^T_i x ≤ bⁱ ∀aⁱ ∈ Eⁱ, i = 1, . . . , m is equivalent to the SOCP

minimize c^Tx

subject to ¯a_i^Tx +&P_i^Tx&2 ≤ bi, i = 1, . . . , m (follows from sup"u"2≤1(¯a_i + P_iu)^Tx = ¯a^T_i x +&P_i^Tx&2)

(12)

Stochastic approach via SOCP

! assume a_i is Gaussian with mean ¯a_i, covariance Σ_i (a_i ∼ N (¯ai, Σ_i))

! a^T_i x is Gaussian r.v. with mean ¯a_i^Tx, variance x^TΣix; hence

prob(a^T_i x ≤ bi) = Φ

%b_i − ¯a^T_i x

&Σ^1/2_i x&2

&

where Φ(x) = (1/√

2π)'_x

−∞e^−t²^/2dt is CDF of N (0, 1)

! robust LP

minimize c^Tx

subject to prob(a_i^Tx ≤ bi) ≥ η, i = 1, . . . , m, with η ≥ 1/2, is equivalent to the SOCP

minimize c^Tx

subject to ¯a^T_i x + Φ⁻¹(η)&Σ^1/2_i x&² ≤ bⁱ, i = 1, . . . , m

Geometric programming

monomial function

f (x) = cx₁â¹x₂â² · · · xnâⁿ, dom f = Rⁿ₊₊

with c > 0; exponent αi can be any real number posynomial function: sum of monomials

f (x) =

$K k=1

c_kx₁â^1kx₂â^2k · · · xnâ^nk, dom f = Rⁿ₊₊

geometric program (GP) minimize f₀(x)

subject to f_i(x)≤ 1, i = 1, . . . , m hi(x) = 1, i = 1, . . . , p with fi posynomial, hi monomial

(13)

Geometric program in convex form

change variables to yi = log xi, and take logarithm of cost, constraints

! monomial f (x) = cx₁^a¹· · · xn^aⁿ transforms to

log f (e^y¹, . . . , e^yⁿ) = a^Ty + b (b = log c)

! posynomial f (x) = "_K

k=1c_kx₁â^1kx₂â^2k · · · xⁿâ^nk transforms to log f (e^y¹, . . . , e^yⁿ) = log

% _K

$

k=1

e^a^T^k^{y +b}^k

&

(b_k = log c_k)

! geometric program transforms to convex problem minimize log("K

k=1exp(a^T_0ky + b_0k)) subject to log("K

k=1exp(a^T_iky + b_ik))

≤ 0, i = 1, . . . , m Gy + d = 0

Design of cantilever beam Design of cantilever beam

PSfrag replacements

F segment 4 segment 3 segment 2 segment 1

• N segments with unit lengths, rectangular cross-sections of size w

ⁱ

× h

ⁱ

• given vertical force F applied at the right end design problem

minimize total weight

subject to upper & lower bounds on w

i

, h

i

upper bound & lower bounds on aspect ratios h

_i

/w

_i

upper bound on stress in each segment

upper bound on vertical deflection at the end of the beam variables: w

i

, h

i

for i = 1, . . . , N

! N segments with unit lengths, rectangular cross-sections of size wi × hi

! given vertical force F applied at the right end design problem

minimize total weight

subject to upper & lower bounds on w_i, h_i

upper bound & lower bounds on aspect ratios h_i/w_i upper bound on stress in each segment

upper bound on vertical deflection at the end of the beam variables: w_i, h_i for i = 1, . . . , N

(14)

Objective and constraint functions

! total weight w₁h₁ +· · · + wNh_N is posynomial

! aspect ratio hi/wi and inverse aspect ratio wi/hi are monomials

! maximum stress in segment i is given by 6iF /(w_ih²_i), a monomial

! the vertical deflection yi and slope vi of central axis at the right end of segment i are defined recursively as

vi = 12(i − 1/2) F

Ewih³_i + vi+1

y_i = 6(i − 1/3) F

Ew_ih_i³ + v_i+1+ y_i+1

for i = N, N − 1, . . . , 1, with vN+1 = y_N+1 = 0 (E is Young’s modulus)

vi and yi are posynomial functions of w , h

Formulation as a GP

minimize w1h1 +· · · + wNh_N

subject to w_max⁻¹ wi ≤ 1, wminw_i⁻¹ ≤ 1, i = 1, . . . , N h⁻¹_maxh_i ≤ 1, hminh⁻¹_i ≤ 1, i = 1, . . . , N

S_max⁻¹ w_i⁻¹h_i ≤ 1, Sminw_ih⁻¹_i ≤ 1, i = 1, . . . , N 6iF σ_max⁻¹ w_i⁻¹h⁻²_i ≤ 1, i = 1, . . . , N

y_max⁻¹ y1 ≤ 1 note

! we write wmin ≤ wi ≤ wmax and hmin ≤ hi ≤ hmax

w_min/w_i ≤ 1, wi/w_max ≤ 1, hmin/h_i ≤ 1, hi/h_max ≤ 1

! we write S_min ≤ hi/wi ≤ Smax as

Sminwi/hi ≤ 1, hi/(wiSmax) ≤ 1

! The number of monomials appearing in y₁ grows approximately as N².

(15)

Minimizing spectral radius of nonnegative matrix

Perron-Frobenius eigenvalue λ_pf(A)

! Consider (elementwise) positive A∈ Rⁿ^×n that is irreducible:

(I + A)ⁿ⁻¹ > 0.

! P-F Theorem: there is a real, positive eigenvalue of A, λ_pf, equal to spectral radius maxi |λⁱ(A)|

! determines asymptotic growth (decay) rate of A^k: A^k ∼ λ^k_pf as k → ∞

! alternative characterization:

λpf(A) = inf{λ | Av - λv for some v 2 0}

minimizing spectral radius of matrix of posynomials

! minimize λ_pf(A(x)), where the elements A(x)ij are posynomials of x

! equivalent geometric program:

minimize λ subject to "_n

j=1A(x)ijvj/(λvi)≤ 1, i = 1, . . . , n variables λ, v , x

Quasiconvex optimization

minimize f0(x)

with f0 : Rⁿ → R quasiconvex, f1, . . . , fm convex

can have locally optimal points that are not (globally) optimal Quasiconvex optimization

minimize f0(x)

subject to fi(x) ≤ 0, i = 1, . . . , m Ax = b

with f0 : Rⁿ → R quasiconvex, f1, . . . , fm convex

can have locally optimal points that are not (globally) optimal

PSfrag replacements

(x, f0(x))

(16)

convex representation of sublevel sets of f

₀

if f0 is quasiconvex, there exists a family of functions φt such that:

! φt(x) is convex in x for fixed t

! t-sublevel set of f0 is 0-sublevel set of φt, i.e., f₀(x)≤ t ⇐⇒ φt(x)≤ 0 example

f₀(x) = p(x) q(x)

with p convex, q concave, and p(x) ≥ 0, q(x) > 0 on dom f0

can take φt(x) = p(x)− tq(x):

! for t ≥ 0, φ^t convex in x

! p(x)/q(x)≤ t if and only if φt(x) ≤ 0

quasiconvex optimization via convex feasibility problems

φt(x)≤ 0, fi(x) ≤ 0, i = 1, . . . , m, Ax = b (1)

! for fixed t, a convex feasibility problem in x

! if feasible, we can conclude that t ≥ p^!; if infeasible, t ≤ p^!

Bisection method for quasiconvex optimization given l ≤ p^!, u ≥ p^!, tolerance * > 0.

repeat

1. t := (l + u)/2.

2. Solve the convex feasibility problem (1).

3. if (1) is feasible, u := t; else l := t.

until u − l ≤ *.

requires exactly 3log2((u− l)/*)4 iterations (where u, l are initial values)

(17)

(Generalized) linear-fractional program

minimize f0(x) subject to Gx - h

Ax = b linear-fractional program

f0(x) = c^Tx + d

e^Tx + f , dom f₀(x) ={x | e^Tx + f > 0}

! a quasiconvex optimization problem; can be solved by bisection

! also equivalent to the LP (variables y , z) minimize c^Ty + dz subject to Gy - hz

Ay = bz e^Ty + fz = 1 z ≥ 0

Generalized linear-fractional program

f0(x) = max

i=1,...,r

c_i^Tx + d_i

e_i^Tx + f_i , dom f₀(x) ={x | ei^Tx+fi > 0, i = 1, . . . , r} a quasiconvex optimization problem; can be solved by bisection

example: Von Neumann model of a growing economy maximize (over x, x⁺) mini=1,...,nx_i⁺/xi

subject to x⁺ + 0, Bx⁺ - Ax

! x, x⁺ ∈ Rⁿ: activity levels of n sectors, in current and next period

! (Ax)i, (Bx⁺)i: produced, resp. consumed, amounts of good i

! x_i⁺/x_i: growth rate of sector i

allocate activity to maximize growth rate of slowest growing sector

(18)

Generalized inequality constraints

convex problem with generalized inequality constraints minimize f₀(x)

subject to f_i(x) -K_i 0, i = 1, . . . , m Ax = b

! f0 : Rⁿ → R convex; fⁱ : Rⁿ → R^kⁱ Ki-convex w.r.t. proper cone Ki

! same properties as standard convex problem (convex feasible set, local optimum is global, etc.)

conic form problem: special case with affine objective and constraints

minimize c^Tx

subject to Fx + g -K 0 Ax = b

extends linear programming (K = R^m₊) to nonpolyhedral cones

Semidefinite program (SDP)

minimize c^Tx

subject to x₁F₁ + x₂F₂+· · · + xnF_n + G - 0 Ax = b

with F_i, G ∈ S^k

! inequality constraint is called linear matrix inequality (LMI)

! includes problems with multiple LMI constraints: for example, x1Fˆ1+· · · + xⁿFˆn + ˆG - 0, x1F˜1+· · · + xⁿF˜n + ˜G - 0 is equivalent to single LMI

x1

* ˆF₁ 0 0 F˜₁

+ +x2

* ˆF₂ 0 0 F˜₂

+

+· · ·+xⁿ* ˆF_n 0 0 F˜_n

+

+* ˆG 0 0 G˜

+ - 0

(19)

LP and SOCP as SDP

LP and equivalent SDP LP: minimize c^Tx

subject to Ax - b

SDP: minimize c^Tx

subject to diag(Ax − b) - 0 (note different interpretation of generalized inequality -)

SOCP and equivalent SDP SOCP: minimize f^Tx

subject to &Aix + bi&2 ≤ c_i^Tx + di, i = 1, . . . , m SDP: minimize f^Tx

subject to

* (c_i^Tx + di)I Aix + bi

(Aix + bi)^T c_i^Tx + di

+

+ 0, i = 1, . . . , m

Eigenvalue minimization

minimize λmax(A(x))

where A(x) = A0+ x1A1+· · · + xnAn (with given Ai ∈ S^k)

equivalent SDP

minimize t

subject to A(x) - tI

! variables x ∈ Rⁿ, t ∈ R

! follows from

λmax(A) ≤ t ⇐⇒ A- tI

(20)

Matrix norm minimization

minimize &A(x)&2 = ,

λ_max(A(x)^TA(x))-1/2

where A(x) = A0+ x1A1+· · · + xnAn (with given Ai ∈ S^p^×q) equivalent SDP

minimize t subject to

* tI A(x)

A(x)^T tI +

+ 0

! variables x ∈ Rⁿ, t ∈ R

! constraint follows from

&A&2 ≤ t ⇐⇒ A^TA - t²I , t ≥ 0

⇐⇒

* tI A A^T tI

+ + 0

Convexity of vector-valued functions

f : Rⁿ → R^m is K -convex if dom f is convex and f (θx + (1− θ)y) -K θf (x) + (1− θ)f (y) for x, y ∈ dom f , 0 ≤ θ ≤ 1

example f : S^m → S^m, f (X ) = X² is S^m₊-convex

proof: for fixed z ∈ R^m, z^TX²z = &Xz&²2 is convex in X , i.e., z^T(θX + (1− θ)Y )²z ≤ θz^TX²z + (1− θ)z^TY²z for X , Y ∈ S^m, 0 ≤ θ ≤ 1

therefore (θX + (1− θ)Y )² - θX²+ (1− θ)Y²

(21)

Vector optimization

general vector optimization problem minimize (w.r.t. K ) f₀(x)

subject to f_i(x)≤ 0, i = 1, . . . , m h_i(x) ≤ 0, i = 1, . . . , p vector objective f0 : Rⁿ → R^q, minimized w.r.t. proper cone K ∈ R^q

convex vector optimization problem minimize (w.r.t. K ) f₀(x)

subject to f_i(x)≤ 0, i = 1, . . . , m Ax = b

with f0 K -convex, f1, . . . , fm convex

Optimal and Pareto optimal points

set of achievable objective values

O = {f⁰(x)| x feasible}

! feasible x is optimal if f0(x) is a minimum value of O

! feasible x is Pareto optimal if f0(x) is a minimal value of O

Optimal and Pareto optimal points

O = {f⁰(x) | x feasible}

• feasible x is optimal if f⁰(x) is a minimum value of O

• feasible x is Pareto optimal if f⁰(x) is a minimal value of O

PSfrag replacements

O

f0(x^!)

x^! is optimal

PSfrag replacements

O f0(x^po)

x^po is Pareto optimal

Optimal and Pareto optimal points

O = {f⁰(x)| x feasible}

• feasible x is optimal if f0(x) is a minimum value of O

• feasible x is Pareto optimal if f0(x) is a minimal value of O

PSfrag replacements

O

f0(x^!)

x^! is optimal

PSfrag replacements

O f0(x^po)

x^po is Pareto optimal

(22)

Multicriterion optimization

vector optimization problem with K = R^q₊ f0(x) = (F1(x), . . . , Fq(x))

! q different objectives Fi; roughly speaking we want all Fi’s to be small

! feasible x^! is optimal if

y feasible =⇒ f₀(x^!) - f0(y ) if there exists an optimal point, the objectives are noncompeting

! feasible x^po is Pareto optimal if

y feasible, f0(y ) - f0(x^po) =⇒ f0(x^po) = f0(y ) if there are multiple Pareto optimal values, there is a trade-off between the objectives

Regularized least-squares

multicriterion problem with two objectives

F1(x) = &Ax − b&²2, F2(x) =&x&²2

! example with A∈ R¹⁰⁰^×10

! shaded region is O

! heavy line is formed by Pareto optimal points

Regularized least-squares

multicriterion problem with two objectives

F1(x) =!Ax − b!²2, F2(x) =!x!²2

• example with A ∈ R^100×10

• shaded region is O

• heavy line is formed by Pareto optimal points

PSfrag replacements

F1(x) = !Ax − b!²₂ F2(x)=!x!2 2

0 5 10 15

(23)

Risk return trade-off in portfolio optimization

minimize (w.r.t. R²₊) (−¯p^Tx, x^TΣx) subject to 1^Tx = 1, x + 0

! x ∈ Rⁿ is investment portfolio; x_i is fraction invested in asset i

! p ∈ Rⁿ is vector of relative asset price changes; modeled as a random variable with mean ¯p, covariance Σ

! ¯p^Tx = E r is expected return; x^TΣx = var r is return variance example

Risk return trade-off in portfolio optimization

minimize (w.r.t. R²₊) (−¯p^Tx, x^TΣx) subject to 1^Tx = 1, x" 0

• x ∈ Rⁿ is investment portfolio; xiis fraction invested in asset i

• p ∈ Rⁿ is vector of relative asset price changes; modeled as a random variable with mean ¯p, covariance Σ

• ¯p^Tx = E r is expected return; x^TΣx = var r is return variance example

PSfrag replacements

meanreturn

standard deviation of return

0% 10% 20%

0%

5%

10%

15%

PSfrag replacements

standard deviation of return

allocationx

x(1) x(2) x(3)

x(4)

0% 10% 20%

0 0.5 1

Scalarization

to find Pareto optimal points: choose λ 2K^∗ 0 and solve scalar problem

minimize λ^Tf₀(x)

subject to f_i(x)≤ 0, i = 1, . . . , m hi(x) = 0, i = 1, . . . , p

if x is optimal for scalar problem, then it is Pareto-optimal for vector optimization problem

Scalarization

to find Pareto optimal points: choose λ!^K^∗0 and solve scalar problem minimize λ^Tf0(x)

subject to fi(x)≤ 0, i = 1, . . . , m hi(x) = 0, i = 1, . . . , p

if x is optimal for scalar problem, then it is Pareto-optimal for vector optimization problem

PSfrag replacements O

f0(x1)

λ1

f0(x2) λ² f0(x3)

for convex vector optimization problems, can find (almost) all Pareto optimal points by varying λ!^K^∗0

for convex vector optimization problems, can find (almost) all Pareto optimal points by varying λ 2K^∗ 0

(24)

examples

! for multicriterion problem, find Pareto optimal points by minimizing positive weighted sum

λ^Tf0(x) = λ1F1(x) +· · · + λqFq(x)

! regularized least-squares of page 111 (with λ = (1, γ)) minimize &Ax − b&²2 + γ&x&²2

for fixed γ > 0, a least-squares problem

! risk-return trade-off of page 112 (with λ = (1, γ)) minimize −¯p^Tx + γx^TΣx subject to 1^Tx = 1, x + 0 for fixed γ > 0, a QP