Machine Learning Techniques (ᘤᢈ)

Machine Learning Techniques ( 機器學習技法)

Lecture 1: Linear Support Vector Machine

htlin@csie.ntu.edu.tw

Department of Computer Science

& Information Engineering

National Taiwan University

( 國立台灣大學資訊工程系)

Linear Support Vector Machine Course Introduction

Course History

NTU Version

•

•

English and blackboard teaching

Coursera Version

•

• Mandarin teaching

• slides teaching

goal:

try

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 1/28

Linear Support Vector Machine Course Introduction

Course History

NTU Version

•

•

English and blackboard teaching

Coursera Version

•

(previous course) + 8 weeks of ‘techniques’ (this course)

• Mandarin teaching

• slides teaching

goal:

try

Linear Support Vector Machine Course Introduction

Course History

NTU Version

•

•

English and blackboard teaching

Coursera Version

•

(previous course) + 8 weeks of ‘techniques’ (this course)

• Mandarin teaching

• slides teaching

goal:

try

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 1/28

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

:-)

•

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

:-)

•

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 2/28

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

:-)

•

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

:-)

•

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 2/28

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

:-)

•

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

:-)

•

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 2/28

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

:-)

•

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

:-)

•

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 2/28

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

:-)

•

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Linear Support Vector Machine Course Introduction

Fun Time

Which of the following description of this course is true?

1

2

3

4

Reference Answer: 4

1

2

3

4

let’s get started!

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 3/28

Linear Support Vector Machine Course Introduction

Fun Time

Which of the following description of this course is true?

1

2

3

4

Reference Answer: 4

1

2

3

4

let’s get started!

Linear Support Vector Machine Course Introduction

Roadmap

1

Lecture 1: Linear Support Vector Machine

Course Introduction

Large-Margin Separating Hyperplane Standard Large-Margin Problem Support Vector Machine

Reasons behind Large-Margin Hyperplane

2 Combining Predictive Features: Aggregation Models

3 Distilling Implicit Features: Extraction Models

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 4/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Linear Classification Revisited

PLA/pocket

h(x) = sign(s)

s x

x

x x

1 2

d

h ( ) x

plausible err = 0/1

(small flipping noise)

specially

(linear separable)

linear (hyperplane) classifiers:

h(x) = sign(w

^T x)

Linear Support Vector Machine Large-Margin Separating Hyperplane

Which Line Is Best?

•

•

out

E _in (w)

| {z }

0

+

Ω( H)

| {z }

d

=d +1

You?

rightmost one, possibly :-)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 6/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Which Line Is Best?

•

•

out

E _in (w)

| {z }

0

+

Ω( H)

| {z }

d

=d +1

You?

rightmost one, possibly :-)

Linear Support Vector Machine Large-Margin Separating Hyperplane

Which Line Is Best?

•

•

E

_out

E _in (w)

| {z }

0

+

Ω( H)

| {z }

d

=d +1

You?

rightmost one, possibly :-)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 6/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Which Line Is Best?

•

•

E

_out

E _in (w)

| {z }

0

+

Ω( H)

| {z }

d

=d +1

You?

rightmost one, possibly :-)

Linear Support Vector Machine Large-Margin Separating Hyperplane

Why Rightmost Hyperplane?

informal argument

if (Gaussian-like) noise on future

x

ⁿ

⇐⇒

x _n further from hyperplane

⇐⇒

tolerate more noise

⇐⇒

more robust to overfitting

⇐⇒

distance to closest x _n

⇐⇒

amount of noise tolerance

⇐⇒

robustness of hyperplane

more robust

because of

larger distance to closest x n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 7/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Why Rightmost Hyperplane?

informal argument

if (Gaussian-like) noise on future

x

ⁿ

⇐⇒

x _n further from hyperplane

⇐⇒

tolerate more noise

⇐⇒

more robust to overfitting

⇐⇒

distance to closest x _n

⇐⇒

amount of noise tolerance

⇐⇒

robustness of hyperplane

more robust

because of

larger distance to closest x n

Linear Support Vector Machine Large-Margin Separating Hyperplane

Why Rightmost Hyperplane?

informal argument

if (Gaussian-like) noise on future

x

ⁿ

⇐⇒

x _n further from hyperplane

⇐⇒

tolerate more noise

⇐⇒

more robust to overfitting

⇐⇒

distance to closest x _n

⇐⇒

amount of noise tolerance

⇐⇒

robustness of hyperplane

more robust

because of

larger distance to closest x n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 7/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Why Rightmost Hyperplane?

informal argument

if (Gaussian-like) noise on future

x

ⁿ

⇐⇒

x _n further from hyperplane

⇐⇒

tolerate more noise

⇐⇒

more robust to overfitting

⇐⇒

distance to closest x _n

⇐⇒

amount of noise tolerance

⇐⇒

robustness of hyperplane

more robust

because of

larger distance to closest x n

Linear Support Vector Machine Large-Margin Separating Hyperplane

Why Rightmost Hyperplane?

informal argument

if (Gaussian-like) noise on future

x

ⁿ

⇐⇒

x _n further from hyperplane

⇐⇒

tolerate more noise

⇐⇒

more robust to overfitting

⇐⇒

distance to closest x _n

⇐⇒

amount of noise tolerance

⇐⇒

robustness of hyperplane

rightmost one:

more robust

larger distance to closest x n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 7/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Why Rightmost Hyperplane?

informal argument

if (Gaussian-like) noise on future

x

ⁿ

⇐⇒

x _n further from hyperplane

⇐⇒

tolerate more noise

⇐⇒

more robust to overfitting

⇐⇒

distance to closest x _n

⇐⇒

amount of noise tolerance

⇐⇒

robustness of hyperplane

rightmost one:

more robust

larger distance to closest x n

Linear Support Vector Machine Large-Margin Separating Hyperplane

Why Rightmost Hyperplane?

informal argument

if (Gaussian-like) noise on future

x

ⁿ

⇐⇒

x _n further from hyperplane

⇐⇒

tolerate more noise

⇐⇒

more robust to overfitting

⇐⇒

distance to closest x _n

⇐⇒

amount of noise tolerance

⇐⇒

robustness of hyperplane

more robust

because of

larger distance to closest x _n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 7/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Fat Hyperplane

• robust

fat

—far from both sides of examples

• robustness

fatness: distance to closest x _n

fattest

Linear Support Vector Machine Large-Margin Separating Hyperplane

Fat Hyperplane

• robust

fat

—far from both sides of examples

• robustness

fatness: distance to closest x _n

goal: find

fattest

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 8/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Fat Hyperplane

• robust

fat

—far from both sides of examples

• robustness

fatness: distance to closest x _n

fattest

Linear Support Vector Machine Large-Margin Separating Hyperplane

Large-Margin Separating Hyperplane

max

w fatness(w)

subject to

w classifies every (x _n

n

fatness(w) =

n=1,...,N

_n

max

w margin(w)

y _n w ^T x _n > 0

margin(w) =

n=1,...,N

_n

•

margin

• correctness: y _n

^T x _n

goal: find

largest-margin separating

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 9/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Large-Margin Separating Hyperplane

max

w margin(w)

subject to

w classifies every (x _n

n

margin(w) =

n=1,...,N

_n

max

w margin(w)

y _n w ^T x _n > 0

margin(w) =

n=1,...,N

_n

•

margin

• correctness: y _n

^T x _n

goal: find

largest-margin

separating

Linear Support Vector Machine Large-Margin Separating Hyperplane

Large-Margin Separating Hyperplane

max

w margin(w)

y _n w ^T x _n > 0

margin(w) =

n=1,...,N

_n

•

margin

• correctness: y _n

^T x _n

goal: find

largest-margin separating

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 9/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Large-Margin Separating Hyperplane

max

w margin(w)

y _n w ^T x _n > 0

margin(w) =

n=1,...,N

_n

•

margin

• correctness: y _n

^T x _n

goal: find

largest-margin

separating

Linear Support Vector Machine Large-Margin Separating Hyperplane

Fun Time

Consider two examples (v, +1) and (−v, −1) where v ∈ R

²

₀

largest-margin separating

v = (3, 2).

1

₁

2

₂

3

₁

₁

₂

₂

4

₂

₁

₁

₂

Reference Answer: 3

Here the

largest-margin separating

v and

v

^d

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 10/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Fun Time

Consider two examples (v, +1) and (−v, −1) where v ∈ R

²

₀

largest-margin separating

v = (3, 2).

1

₁

2

₂

3

₁

₁

₂

₂

4

₂

₁

₁

₂

Reference Answer: 3

Here the

largest-margin separating

v and

v

^d

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane: Preliminary

max w margin(w)

subject to every y n w ^T x _n > 0 margin(w) = min

n=1,...,N distance(x _n , w)

‘shorten’ x and w

distance

w ₀

(w ₁ , . . . , w _d )

b

w ₀







| w

|











 w ₁

.. . w _d







;

   XX x ₀ = X X 1







| x

|











 x ₁

.. . x _d







for this part: h(x) = sign(w

^T x

b)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 11/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane: Preliminary

max w margin(w)

subject to every y n w ^T x _n > 0 margin(w) = min

n=1,...,N distance(x _n , w)

‘shorten’ x and w

distance

w ₀

(w ₁ , . . . , w _d )

b

w ₀







| w

|











 w ₁

.. . w _d







;

   XX x ₀ = X X 1







| x

|











 x ₁

.. . x _d







for this part: h(x) = sign(w

^T x

b)

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane: Preliminary

max w margin(w)

subject to every y n w ^T x _n > 0 margin(w) = min

n=1,...,N distance(x _n , w)

‘shorten’ x and w

distance

w ₀

(w ₁ , . . . , w _d )

b

w ₀







| w

|











 w ₁

.. . w _d







;

   XX x ₀ = X X 1







| x

|











 x ₁

.. . x _d







for this part: h(x) = sign(w

^T x

b)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 11/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane: Preliminary

max w margin(w)

subject to every y n w ^T x _n > 0 margin(w) = min

n=1,...,N distance(x _n , w)

‘shorten’ x and w

distance

w ₀

(w ₁ , . . . , w _d )

b

w ₀







| w

|











 w ₁

.. . w _d







;

   XX x ₀ = X X 1







| x

|











 x ₁

.. . x _d







for this part: h(x) = sign(w

^T x

b)

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

b

consider

x ⁰

x ⁰⁰

1 w ^T x ⁰

−

b

,

w ^T x ⁰⁰

−

b

2 w









w ^T

⁰⁰

x ⁰

| {z } vector on hyperplane









=0

3

x ⁰

⊥ hyperplane

dist(x, h)

x^′ x^′′ w x

distance(x,

b, w) =

w ^T

w

(x−

x ⁰

=

1

w

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

b

consider

x ⁰

x ⁰⁰

1 w ^T x ⁰

−

b

,

w ^T x ⁰⁰

−

b

2 w









w ^T

⁰⁰

x ⁰

| {z } vector on hyperplane









=

0

3

x ⁰

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

w ^T

w

(x−

x ⁰

=

1

w

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

b

consider

x ⁰

x ⁰⁰

1 w ^T x ⁰

−

b

,

w ^T x ⁰⁰

−

b

2 w









w ^T

⁰⁰

x ⁰

| {z } vector on hyperplane









=

0

3

x ⁰

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

w ^T

w

(x−

x ⁰

=

1

w

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

b

consider

x ⁰

x ⁰⁰

1 w ^T x ⁰

b, w ^T x ⁰⁰

b

2 w









w ^T

⁰⁰

x ⁰

| {z } vector on hyperplane









=

0

3

x ⁰

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

w ^T

w

(x−

x ⁰

=

1

w

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

b

consider

x ⁰

x ⁰⁰

1 w ^T x ⁰

b, w ^T x ⁰⁰

b

2 w









w ^T

⁰⁰

x ⁰

| {z } vector on hyperplane









=

0

3

x ⁰

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

w ^T

w

(x−

x ⁰

=

1

w

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

b

consider

x ⁰

x ⁰⁰

1 w ^T x ⁰

b, w ^T x ⁰⁰

b

2 w









w ^T

⁰⁰

x ⁰

| {z } vector on hyperplane









=0

3

x ⁰

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

w ^T

w

(x−

x ⁰

=

1

w

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

b

consider

x ⁰

x ⁰⁰

1 w ^T x ⁰

b, w ^T x ⁰⁰

b

2 w









w ^T

⁰⁰

x ⁰

| {z } vector on hyperplane









=0

3

x ⁰

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

w ^T

w

(x−

x ⁰

=

1

w

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

b

consider

x ⁰

x ⁰⁰

1 w ^T x ⁰

b, w ^T x ⁰⁰

b

2 w









w ^T

⁰⁰

x ⁰

| {z } vector on hyperplane









=0

3

x ⁰

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

w ^T

w

(x−

x ⁰

=

1

w

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

b

consider

x ⁰

x ⁰⁰

1 w ^T x ⁰

b, w ^T x ⁰⁰

b

2 w









w ^T

⁰⁰

x ⁰

| {z } vector on hyperplane









=0

3

x ⁰

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

w ^T

k

w

x ⁰

=

1

w

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

b

consider

x ⁰

x ⁰⁰

1 w ^T x ⁰

b, w ^T x ⁰⁰

b

2 w









w ^T

⁰⁰

x ⁰

| {z } vector on hyperplane









=0

3

x ⁰

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

w ^T

k

w

x ⁰

=

1

w

w ^T x

+

b

|

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

b

consider

x ⁰

x ⁰⁰

1 w ^T x ⁰

b, w ^T x ⁰⁰

b

2 w









w ^T

⁰⁰

x ⁰

| {z } vector on hyperplane









=0

3

x ⁰

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

w ^T

k

w

x ⁰

=

1

k

w

w ^T x + b

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Separating Hyperplane

distance(x,

b, w) =

k

w

w ^T x + b

• separating

y n (w ^T x _n + b) > 0

•

separating

_n

b, w) =

k

w

y _n

^T x _n

b)

max

b,w

w)

subject to every

y n (w ^T x _n + b) > 0

w) =

n=1,...,N

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Separating Hyperplane

distance(x,

b, w) =

k

w

w ^T x + b

• separating

y n (w ^T x n + b) > 0

•

separating

_n

b, w) =

k

w

y _n

^T x _n

b)

max

b,w

w)

subject to every

y n (w ^T x _n + b) > 0

w) =

n=1,...,N

_n

b, w)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 13/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Separating Hyperplane

distance(x,

b, w) =

k

w

w ^T x + b

• separating

y n (w ^T x n + b) > 0

•

separating

distance(x

_n

b, w) =

k

w

y _n

^T x _n

b)

max

b,w

w)

subject to every

y n (w ^T x _n + b) > 0

w) =

n=1,...,N

_n

b, w)

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Separating Hyperplane

distance(x,

b, w) =

k

w

w ^T x + b

• separating

y n (w ^T x n + b) > 0

•

separating

distance(x

_n

b, w) =

k

w

y _n

^T x _n

b)

max

b,w

w)

subject to every

y n (w ^T x _n + b) > 0

w) =

n=1,...,N 1

kwk y _n

^T x _n

b)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 13/28

Linear Support Vector Machine Standard Large-Margin Problem

Margin of Special Separating Hyperplane

max

b,w

w)

subject to every y

_n

^T x _n

b)

w) =

n=1,...,N 1

kwk

_n

^T x _n

b)

• w ^T x + b

^T x + 3b

• special

w)

min

n=1,...,N y _n (w ^T x _n + b) = 1

margin(b,

w) = _kwk ¹

max

b,w 1 kwk

subject to every y

n

^T x _n

min

n=1,...,N y _n (w ^T x _n + b) = 1

Linear Support Vector Machine Standard Large-Margin Problem

Margin of Special Separating Hyperplane

max

b,w

w)

subject to every y

_n

^T x _n

b)

w) =

n=1,...,N 1

kwk

_n

^T x _n

b)

• w ^T x + b

^T x + 3b

• special

w)

min

n=1,...,N y _n (w ^T x _n + b) = 1

margin(b,

w) = _kwk ¹

max

b,w 1 kwk

subject to every y

n

^T x _n

min

n=1,...,N y _n (w ^T x _n + b) = 1

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 14/28

Linear Support Vector Machine Standard Large-Margin Problem

Margin of Special Separating Hyperplane

max

b,w

w)

subject to every y

_n

^T x _n

b)

w) =

n=1,...,N 1

kwk

_n

^T x _n

b)

• w ^T x + b

^T x + 3b

• special

w)

n=1,...,N min y n (w ^T x _n + b) = 1

margin(b,

w) = _kwk ¹

max

b,w 1 kwk

subject to every y

n

^T x _n

min

n=1,...,N y _n (w ^T x _n + b) = 1

Linear Support Vector Machine Standard Large-Margin Problem

Margin of Special Separating Hyperplane

max

b,w

w)

subject to every y

_n

^T x _n

b)

w) =

n=1,...,N 1

kwk

_n

^T x _n

b)

• w ^T x + b

^T x + 3b

• special

w)

n=1,...,N min y n (w ^T x _n + b) = 1

b, w) = _kwk ¹

max

b,w 1 kwk

subject to every y

n

^T x _n

min

n=1,...,N y _n (w ^T x _n + b) = 1

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 14/28

Linear Support Vector Machine Standard Large-Margin Problem

Margin of Special Separating Hyperplane

max

b,w

w)

subject to every y

_n

^T x _n

b)

w) =

n=1,...,N 1

kwk

_n

^T x _n

b)

• w ^T x + b

^T x + 3b

• special

w)

n=1,...,N min y n (w ^T x _n + b) = 1

b, w) = _kwk ¹

max

b,w 1 kwk

subject to every y

n

^T x _n

min

n=1,...,N y _n (w ^T x _n + b) = 1

Linear Support Vector Machine Standard Large-Margin Problem

Margin of Special Separating Hyperplane

max

b,w

w)

subject to every y

_n

^T x _n

b)

w) =

n=1,...,N 1

kwk

_n

^T x _n

b)

• w ^T x + b

^T x + 3b

• special

w)

n=1,...,N min y n (w ^T x _n + b) = 1

b, w) = _kwk ¹

max

b,w 1 kwk

subject to

every y n (w ^T x _n + b) > 0 min

n=1,...,N y _n (w ^T x _n + b) = 1

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 14/28

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

^T x _n

b)

original constraint:

min _n=1,...,N y n (w ^T x n + b) = 1

want: optimal (b,

w) here (inside)

w)

_n

^T x _n

b)

1.126

for all n

—can scale (b,

w)

_1.126 ^b

_1.126 ^w

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

b,w

1 2 w ^T w

subject to y

_n

^T x _n

b)

for all n

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

^T x _n

b)

min _n=1,...,N y n (w ^T x n + b) = 1

want: optimal (b,

w) here (inside)

w)

_n

^T x _n

b)

1.126

for all n

—can scale (b,

w)

_1.126 ^b

_1.126 ^w

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

b,w

1 2 w ^T w

subject to y

_n

^T x _n

b)

for all n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 15/28

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

^T x _n

b)

min _n=1,...,N y n (w ^T x n + b) = 1

w) here (inside)

if optimal (b,

w)

_n

^T x _n

b)

1.126

for all n

—can scale (b,

w)

_1.126 ^b

_1.126 ^w

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

b,w

1 2 w ^T w

subject to y

_n

^T x _n

b)

for all n

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

^T x _n

b)

min _n=1,...,N y n (w ^T x n + b) = 1

w) here (inside)

if optimal (b,

w)

e.g. y

_n

^T x _n

b)

1.126

for all n

—can scale (b,

w)

_1.126 ^b

_1.126 ^w

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

b,w

1 2 w ^T w

subject to y

_n

^T x _n

b)

for all n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 15/28

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

^T x _n

b)

min _n=1,...,N y n (w ^T x n + b) = 1

w) here (inside)

if optimal (b,

w)

_n

^T x _n

b)

1.126

for all n

—can scale (b,

w)

_1.126 ^b

_1.126 ^w

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

b,w

1 2 w ^T w

subject to y

_n

^T x _n

b)

for all n

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

^T x _n

b)

min _n=1,...,N y n (w ^T x n + b) = 1

w) here (inside)

if optimal (b,

w)

_n

^T x _n

b)

—can scale (b,

w)

_1.126 ^b

_1.126 ^w

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

b,w

1 2 w ^T w

subject to y

_n

^T x _n

b)

for all n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 15/28

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

^T x _n

b)

min _n=1,...,N y n (w ^T x n + b) = 1

w) here (inside)

if optimal (b,

w)

_n

^T x _n

b)

—can scale (b,

w)

_1.126 ^b

_1.126 ^w

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

b,w

1 2 w ^T w

subject to y

_n

^T x _n

b)

for all n

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

^T x _n

b)

min _n=1,...,N y n (w ^T x n + b) = 1

w) here (inside)

if optimal (b,

w)

_n

^T x _n

b)

—can scale (b,

w)

_1.126 ^b

_1.126 ^w

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

b,w

1 2 w ^T w

subject to y

_n

^T x _n

b)

for all n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 15/28

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

min

n=1,...,N y _n (w ^T x _n + b) = 1

necessary constraints: y

_n

^T x _n

b)

min _n=1,...,N y n (w ^T x n + b) = 1

w) here (inside)

if optimal (b,

w)

_n

^T x _n

b)

—can scale (b,

w)

_1.126 ^b

_1.126 ^w

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

max

b,w

min

b,w 1 2 w ^T w

1 kwk

subject to y

_n

^T x _n

b)

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

min

n=1,...,N y _n (w ^T x _n + b) = 1

necessary constraints: y

_n

^T x _n

b)

min _n=1,...,N y n (w ^T x n + b) = 1

w) here (inside)

if optimal (b,

w)

_n

^T x _n

b)

—can scale (b,

w)

_1.126 ^b

_1.126 ^w

(contradiction!)

w, add ¹ ₂

max

b,w

min

b,w 1 2 w ^T w

1 kwk

subject to y

_n

^T x _n

b)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 15/28

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

^T x _n

b)

min _n=1,...,N y n (w ^T x n + b) = 1

w) here (inside)

if optimal (b,

w)

_n

^T x _n

b)

—can scale (b,

w)

_1.126 ^b

_1.126 ^w

(contradiction!)

w, add ¹ ₂

min

b,w

1 2

w ^T w

subject to y

_n

^T x _n

b)