Machine Learning Techniques (ᘤᢈ)

(1)

Machine Learning Techniques ( 機器學習技巧)

Lecture 4: Soft-Margin SVM

Hsuan-Tien Lin (林軒田) htlin@csie.ntu.edu.tw

Department of Computer Science

& Information Engineering

National Taiwan University ( 國立台灣大學資訊工程系)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 0/22

(2)

Soft-Margin SVM

Agenda

Lecture 4: Soft-Margin SVM

Soft-Margin SVM: Primal

Soft-Margin SVM: Dual

Soft-Margin SVM: Solution

Soft-Margin SVM: Selection

(3)

Soft-Margin SVM Soft-Margin SVM: Primal

Cons of Hard-Margin SVM

recall: SVM can still overfit :-(

Φ

₁

•

part of reasons: Φ

•

other part:

separable

Φ

₄

if always insisting on

separable

(=⇒

shatter),

have power to

overfit to noise

(4)

Give Up on Some Examples

want:

give up

on some noisy examples

pocket

min b,w N

X

n=1

qy

_n

6= sign(w

^T

z

_n

+ b) y

hard-margin SVM

min b,w

1 2 w

^T

w

s.t. y _n (w ^T x _n + b) ≥ 1 for all n

combination: min

b,w

1 2 w ^T w

+

C

·

N

X

n=1

r

y _n 6= sign(w ^T z _n + b) z

s.t. y

_n

(w

^T x _n

+b)≥ 1 for

correct

n

y

_n

(w

^T x _n

+b)≥

−∞

for

incorrect

n

C: trade-off of large margin

&

noise tolerance

(5)

Soft-Margin SVM (1/2)

min b,w

1 2 w

^T

w + C ·

N

X

n=1

qy

n

6= sign(w

^T

z

n

+ b) y

s.t. y _n (w ^T x _n + b) ≥ 1 − ∞ · qy

_n

6= sign(w

^T

z

_n

+ b) y

• J·K

: non-linear—not QP anymore :-((dual? kernel?)

•

cannot distinguish

small error (slightly away from fat boundary)

or

large error (a...w...a...y... from fat boundary)

•

record ‘margin violation’ by

ξ n

—linear constraints

•

penalize with

margin violation

instead of

error count

—quadratic objective

soft-margin SVM: min

b,w,ξ

1 2 w

^T

w + C ·

N

X

n=1

ξ

n

s.t. y n (w ^T x n + b) ≥ 1 − ξ

n

and ξ

n

≥ 0 for all n

(6)

Soft-Margin SVM (2/2)

•

record ‘margin violation’ by

ξ n

•

penalize with

margin violation

b,w,ξ min 1

2 w

^T

w + C ·

N

X

n=1

ξ

n

s.t. y _n (w ^T x _n + b) ≥ 1 − ξ

n

and ξ

n

≥ 0 for all n

Hi Hi

violation

•

parameter

C: trade-off of large margin

&

margin violation

• large C: want less margin violation

• small C: want large margin

• QP

of

d ˜

+1 + N variables, 2N constraints next: remove dependence on

d ˜

by

soft-margin SVM primal⇒

dual?

(7)

Fun Time

(8)

Soft-Margin SVM Soft-Margin SVM: Dual

Lagrange Dual

primal: min

b,w,ξ

1

2

w ^T w + C

·

N

X

n=1

ξ

_n

s.t.

y _n (w ^T x _n + b) ≥ 1 − ξ n

and

ξ n ≥ 0

for all n Lagrange function with Lagrange multipliers

α n

and

β n

L(b, w, ξ, α, β) = 1

2 w ^T w + C ·

N

X

n=1

ξ n

+

N

X

n=1

α

n

· 1 − ξ

n

− y

n

(w

^T

x

_n

+ b) +

N

X

n=1

β

n

· (−ξ

n

)

want: Lagrange dual max

α

n

≥0, β

n

≥0

b,w,ξ

min L(b, w, ξ,

α, β)

(9)

Simplify ξ _n and β _n

max

αn≥0,βn≥0

min

b,w,ξ

1 2 w

^T

w + C ·

N

X

n=1

ξ

_n

+

N

X

n=1

α

_n

· 1 − ξ

_n

− y

n

(w

^T

x

n

+ b) +

N

X

n=1

β

_n

· (− ξ

_n

)

!

• _∂ξ ^∂L

_n =0 = C

−α n −β n

•

no loss of optimality if solving with implicit constraint

β _n

=C−

α _n

and explicit constraint 0≤

α n

≤

C: β n

removed

ξ can also be removed :-), like how we removed b

max

0≤α

n

≤C,

βn

=C−α

n

b,w,ξ min 1 2 w ^T w +

N

X

n=1

α

n

(1 − y

n

(w

^T

z

_n

+ b))

XX XX

XX XX XX X +

N

P

n=1

(C − α

n

− β

n

) · ξ

n

!

(10)

Other Simplifications

max

0≤α

n

≤C,

β_n

=C−α

n

min b,w

1 2 w ^T w +

N

X

n=1

α

n

(1 − y

n

(w

^T

z

_n

+ b))

!

familiar? :-)

•

inner problem

same as hard-margin SVM

• ^∂L _∂b

=0: no loss of optimality if solving with constraint

N

P

n=1

α n y n = 0

• _∂w ^∂L

_i =0: no loss of optimality if solving with constraint

w =

N

P

n=1

α n y n z _n

standard dual can be derived

using the same steps as Lecture 18

(11)

Standard Soft-Margin SVM Dual

min

α

1 2

N

X

n=1 N

X

m=1

α n α m

y

n

y

m z ^T _n z m

−

N

X

n=1

α n

subject to

N

X

n=1

y

_n α n

=0;

0≤

α _n ≤ C

, for n = 1, 2, . . . , N;

implicitly

w =

N

X

n=1

α n

y

n z n

;

β n

=C−

α n

, for n = 1, 2, . . . , N

—only difference to hard-margin:

upper bound

on

α n

another (convex)

QP,

with

N variables

&

2N + 1

constraints

(12)

Fun Time

(13)

Soft-Margin SVM Soft-Margin SVM: Solution

Kernel Soft-Margin SVM

Kernel Soft-Margin SVM Algorithm

1 q n,m

=y

n

y

m K

(x

_n

, x

m

);

c

=−1

N

; (P,

r)

for

equ./lower-bound/upper-bound constraints

2

α← QP(

Q, c, P, r)

3

b←?

4

return

SVs

and theirα

n

as well as b such that for new

x,

gSVM(x) = sign

P

SV indices n

α

_n y _n K

(x

_n

, x) + b

• almost

the same as hard-margin

•

more flexible than hard-margin

—primal/dual always solvable

remaining question:

step 3

?

(14)

Solving for b

hard-margin SVM

complementary slackness:

α n

(1− y

ⁿ

(w

^T x _n

+b)) = 0

•

SV (α

m

> 0)

⇒ b = y

m

− w

^T x _m

•

unbounded (α

m

< C)

⇒

ξ m

=0

soft-margin SVM

α n

(1−

ξ n

− y

ⁿ

(w

^T x _n

+b)) = 0 (C−

α _n

)ξ

_n

=0

•

SV (α

m

> 0)

⇒ b = y

m

− y

m ξ _m

− w

^T x _m

•

unbounded (α

_m

< C)

⇒

ξ m

=0

solve unique b with

unbounded SV (x _m , y _m ):

b =

y m

−

N

X

n=1

α n

y

n

K (x

_n

,

x _m

)

—range of b otherwise

(15)

Soft-Margin Gaussian SVM in Action

C = 1 C = 10 C = 100

•

large C =⇒ less

noise tolerance

=⇒

‘overfit’?

• warning: SVM can still overfit :-(

soft-margin Gaussian SVM:

need

careful selection of (γ, C)

(16)

Physical Meaning of α _n

α n

(1

−ξ n − y n (w ^T x _n + b)) =

0 (C−

α n

)ξ

n

=0

•

non SV (0 =

α n

):

ξ n

=0,

‘away from’/on

fat boundary

•

unbounded SV (0 <

α n

< C):

ξ n

=0, on

fat boundary, locates b

•

4 bounded SV (

α n

=C):

ξ _n

=violation amount,

‘violate’/on

fat boundary

α n

can be used for

data analysis

(17)

Fun Time

(18)

Soft-Margin SVM Soft-Margin SVM: Selection

Practical Need: Model Selection

replacemen

•

complicated even for

(C, γ) of Gaussian SVM

•

more combinations if including other kernels or parameters

how to select?

validation :-)

(19)

Selection by Cross Validation

replacemen

0.3500 0.3250 0.3250

0.2000 0.2250 0.2750

0.1750 0.2250 0.2000

•

E

_cv

(C, γ): ‘non-smooth’

function of (C, γ)

—difficult to optimize

•

proper models can be chosen by

V -fold cross validation

on

a few grid values of (C, γ)

E

_cv

: very popular criteria for soft-margin SVM

(20)

Leave-One-Out CV Error for SVM

recall: E

_loocv

= E

_cv

with N folds claim: E

_loocv

≤

^#SV _N

•

for

(x _N , y _N ): if optimal α _N = 0

(non-SV)

=⇒

(α 1 , α 2 , . . . , α N−1 ) still optimal

when

leaving out (x _N , y N )

key:

what if there’s better

α

n

?

•

SVM:

g ⁻

=g when

leaving out non-SV

e

_non-SV

= err(g

⁻

,

non-SV)

= err(g,

non-SV) =

0 e

_SV

≤ 1

x¹−x²−1=0 0.707

motivation from hard-margin SVM:

only

SVs needed

scaled #SV bounds leave-one-out CV error

(21)

Selection by # SV

replacemen

38 37 37

27 21 17

21 18 19

•

nSV(C, γ): ‘non-smooth’

function of (C, γ)

—difficult to optimize

• just an upper bound!

•

dangerous models can be ruled out by

nSV

on

a few grid values of (C, γ)

nSV: often used as a

safety check

if computing E

_cv

not allowed

(22)

Fun Time

(23)

Machine Learning Techniques (ᘤᢈ)

Machine Learning Techniques ( 機器學習技巧)

Lecture 4: Soft-Margin SVM

Department of Computer Science

& Information Engineering

National Taiwan University ( 國立台灣大學資訊工程系)

Agenda

Lecture 4: Soft-Margin SVM

Soft-Margin SVM: Primal

Soft-Margin SVM: Dual

Soft-Margin SVM: Solution

Soft-Margin SVM: Selection

Cons of Hard-Margin SVM

recall: SVM can still overfit :-(

1

•

•

separable

4

separable

shatter),

overfit to noise

Give Up on Some Examples

give up

pocket

min b,w N

X

n=1

qy

6= sign(w

z

+ b) y

hard-margin SVM

min b,w

1 2 w

w

s.t. y n (w T x n + b) ≥ 1 for all n

b,w

1

2 w T w

C

N

n=1

r

y n 6= sign(w T z n + b) z

n

T x n

correct

n

T x n

−∞

incorrect

C: trade-off of large margin

noise tolerance

Soft-Margin SVM (1/2)

min b,w

1

2 w

w + C ·

N

X

n=1

qy

6= sign(w

z

+ b) y

s.t. y n (w T x n + b) ≥ 1 − ∞ · qy

6= sign(w

z

+ b) y

• J·K

•

small error (slightly away from fat boundary)

large error (a...w...a...y... from fat boundary)

•

ξ n

•

margin violation

error count

soft-margin SVM: min

Machine Learning Techniques (ᘤᢈ)

₁

₄

s.t. y _n (w ^T x _n + b) ≥ 1 for all n

2 w ^T w

y _n 6= sign(w ^T z _n + b) z

_n

^T x _n

_n

^T x _n

s.t. y _n (w ^T x _n + b) ≥ 1 − ∞ · qy

s.t. y n (w ^T x n + b) ≥ 1 − ξ

s.t. y _n (w ^T x _n + b) ≥ 1 − ξ

w ^T w + C

_n

y _n (w ^T x _n + b) ≥ 1 − ξ n

2 w ^T w + C ·

Simplify ξ _n and β _n