Machine Learning Techniques (ᘤᢈ)

Machine Learning Techniques ( 機器學習技法)

Lecture 1: Linear Support Vector Machine

Hsuan-Tien Lin (林軒田)

htlin@csie.ntu.edu.tw

Department of Computer Science

& Information Engineering

National Taiwan University

( 國立台灣大學資訊工程系)

Linear Support Vector Machine Course Introduction

Course History

NTU Version

•

15-17 weeks (2+ hours)

•

highly-praised with

English and blackboard teaching

Coursera Version

•

8 weeks of ‘foundations’ (previous course) + 8 weeks of ‘techniques’ (this course)

• Mandarin teaching

to reach more audience in need

• slides teaching

improved with Coursera’s quiz and homework mechanisms

goal:

try

making Coursera version even better than NTU version

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 1/28

Linear Support Vector Machine Course Introduction

Course History

NTU Version

•

15-17 weeks (2+ hours)

•

highly-praised with

English and blackboard teaching

Coursera Version

•

8 weeks of ‘foundations’

(previous course) + 8 weeks of ‘techniques’ (this course)

• Mandarin teaching

to reach more audience in need

• slides teaching

improved with Coursera’s quiz and homework mechanisms

goal:

try

making Coursera version even better than NTU version

Linear Support Vector Machine Course Introduction

Course History

NTU Version

•

15-17 weeks (2+ hours)

•

highly-praised with

English and blackboard teaching

Coursera Version

•

8 weeks of ‘foundations’

(previous course) + 8 weeks of ‘techniques’ (this course)

• Mandarin teaching

to reach more audience in need

• slides teaching

improved with Coursera’s quiz and homework mechanisms

goal:

try

making Coursera version even better than NTU version

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 1/28

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

mixture of philosophical illustrations, key theory, core algorithms, usage in practice, and hopefully jokes

:-)

•

three major techniques surrounding

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

mixture of philosophical illustrations, key theory, core algorithms, usage in practice, and hopefully jokes

:-)

•

three major techniques surrounding

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 2/28

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

mixture of philosophical illustrations, key theory, core algorithms, usage in practice, and hopefully jokes

:-)

•

three major techniques surrounding

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

mixture of philosophical illustrations, key theory, core algorithms, usage in practice, and hopefully jokes

:-)

•

three major techniques surrounding

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 2/28

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

mixture of philosophical illustrations, key theory, core algorithms, usage in practice, and hopefully jokes

:-)

•

three major techniques surrounding

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

mixture of philosophical illustrations, key theory, core algorithms, usage in practice, and hopefully jokes

:-)

•

three major techniques surrounding

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 2/28

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

mixture of philosophical illustrations, key theory, core algorithms, usage in practice, and hopefully jokes

:-)

•

three major techniques surrounding

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

mixture of philosophical illustrations, key theory, core algorithms, usage in practice, and hopefully jokes

:-)

•

three major techniques surrounding

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 2/28

Linear Support Vector Machine Course Introduction

Course Design

from Foundations to Techniques

•

mixture of philosophical illustrations, key theory, core algorithms, usage in practice, and hopefully jokes

:-)

•

three major techniques surrounding

feature transforms:

• Embedding Numerous Features: how to exploit and regularize numerous features?

—inspires Support Vector Machine (SVM) model

• Combining Predictive Features: how to construct and blend predictive features?

—inspires Adaptive Boosting (AdaBoost) model

• Distilling Implicit Features: how to identify and learn implicit features?

—inspires Deep Learning model

allows students to

use ML professionally

Linear Support Vector Machine Course Introduction

Fun Time

Which of the following description of this course is true?

1

the course will be taught in Taiwanese

2

the course will tell me the techniques that create the android Lieutenant Commander Data in Star Trek

3

the course will be 16 weeks long

4

the course will focus on three major techniques

Reference Answer: 4

1

no, my Taiwanese is unfortunately not good enough for teaching (yet)

2

no, although what we teach may serve as building blocks

3

no, unless you have also joined the previous course

4

yes,

let’s get started!

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 3/28

Linear Support Vector Machine Course Introduction

Fun Time

Which of the following description of this course is true?

1

the course will be taught in Taiwanese

2

the course will tell me the techniques that create the android Lieutenant Commander Data in Star Trek

3

the course will be 16 weeks long

4

the course will focus on three major techniques

Reference Answer: 4

1

no, my Taiwanese is unfortunately not good enough for teaching (yet)

2

no, although what we teach may serve as building blocks

3

no, unless you have also joined the previous course

4

yes,

let’s get started!

Linear Support Vector Machine Course Introduction

Roadmap

1

Embedding Numerous Features: Kernel Models

Lecture 1: Linear Support Vector Machine

Course Introduction

Large-Margin Separating Hyperplane Standard Large-Margin Problem Support Vector Machine

Reasons behind Large-Margin Hyperplane

2 Combining Predictive Features: Aggregation Models

3 Distilling Implicit Features: Extraction Models

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 4/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Linear Classification Revisited

PLA/pocket

h(x) = sign(s)

s x

x

x x

₀

1 2

d

h ( ) x

plausible err = 0/1

(small flipping noise)

minimize

specially

(linear separable)

linear (hyperplane) classifiers:

h(x) = sign(w

^T x)

Linear Support Vector Machine Large-Margin Separating Hyperplane

Which Line Is Best?

•

PLA? depending on randomness

•

VC bound? whichever you like! E

out

(w)≤

E _in (w)

| {z }

0

+

Ω( H)

| {z }

d

VC

=d +1

You?

rightmost one, possibly :-)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 6/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Which Line Is Best?

•

PLA? depending on randomness

•

VC bound? whichever you like! E

out

(w)≤

E _in (w)

| {z }

0

+

Ω( H)

| {z }

d

VC

=d +1

You?

rightmost one, possibly :-)

Linear Support Vector Machine Large-Margin Separating Hyperplane

Which Line Is Best?

•

PLA? depending on randomness

•

VC bound? whichever you like!

E

_out

(w)≤

E _in (w)

| {z }

0

+

Ω( H)

| {z }

d

VC

=d +1

You?

rightmost one, possibly :-)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 6/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Which Line Is Best?

•

PLA? depending on randomness

•

VC bound? whichever you like!

E

_out

(w)≤

E _in (w)

| {z }

0

+

Ω( H)

| {z }

d

VC

=d +1

You?

rightmost one, possibly :-)

Linear Support Vector Machine Large-Margin Separating Hyperplane

Why Rightmost Hyperplane?

informal argument

if (Gaussian-like) noise on future

x

≈ x

ⁿ

:

⇐⇒

x _n further from hyperplane

⇐⇒

tolerate more noise

⇐⇒

more robust to overfitting

⇐⇒

distance to closest x _n

⇐⇒

amount of noise tolerance

⇐⇒

robustness of hyperplane

rightmost one:

more robust

because of

larger distance to closest x n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 7/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Why Rightmost Hyperplane?

informal argument

if (Gaussian-like) noise on future

x

≈ x

ⁿ

:

⇐⇒

x _n further from hyperplane

⇐⇒

tolerate more noise

⇐⇒

more robust to overfitting

⇐⇒

distance to closest x _n

⇐⇒

amount of noise tolerance

⇐⇒

robustness of hyperplane

rightmost one:

more robust

because of

larger distance to closest x n

Linear Support Vector Machine Large-Margin Separating Hyperplane

Why Rightmost Hyperplane?

informal argument

if (Gaussian-like) noise on future

x

≈ x

ⁿ

:

⇐⇒

x _n further from hyperplane

⇐⇒

tolerate more noise

⇐⇒

more robust to overfitting

⇐⇒

distance to closest x _n

⇐⇒

amount of noise tolerance

⇐⇒

robustness of hyperplane

rightmost one:

more robust

because of

larger distance to closest x n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 7/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Why Rightmost Hyperplane?

informal argument

if (Gaussian-like) noise on future

x

≈ x

ⁿ

:

⇐⇒

x _n further from hyperplane

⇐⇒

tolerate more noise

⇐⇒

more robust to overfitting

⇐⇒

distance to closest x _n

⇐⇒

amount of noise tolerance

⇐⇒

robustness of hyperplane

rightmost one:

more robust

because of

larger distance to closest x n

Linear Support Vector Machine Large-Margin Separating Hyperplane

Why Rightmost Hyperplane?

informal argument

if (Gaussian-like) noise on future

x

≈ x

ⁿ

:

⇐⇒

x _n further from hyperplane

⇐⇒

tolerate more noise

⇐⇒

more robust to overfitting

⇐⇒

distance to closest x _n

⇐⇒

amount of noise tolerance

⇐⇒

robustness of hyperplane

rightmost one:

more robust

because of

larger distance to closest x n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 7/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Why Rightmost Hyperplane?

informal argument

if (Gaussian-like) noise on future

x

≈ x

ⁿ

:

⇐⇒

x _n further from hyperplane

⇐⇒

tolerate more noise

⇐⇒

more robust to overfitting

⇐⇒

distance to closest x _n

⇐⇒

amount of noise tolerance

⇐⇒

robustness of hyperplane

rightmost one:

more robust

because of

larger distance to closest x n

Linear Support Vector Machine Large-Margin Separating Hyperplane

Why Rightmost Hyperplane?

informal argument

if (Gaussian-like) noise on future

x

≈ x

ⁿ

:

⇐⇒

x _n further from hyperplane

⇐⇒

tolerate more noise

⇐⇒

more robust to overfitting

⇐⇒

distance to closest x _n

⇐⇒

amount of noise tolerance

⇐⇒

robustness of hyperplane

rightmost one:

more robust

because of

larger distance to closest x _n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 7/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Fat Hyperplane

• robust

separating hyperplane:

fat

—far from both sides of examples

• robustness

≡

fatness: distance to closest x _n

goal: find

fattest

separating hyperplane

Linear Support Vector Machine Large-Margin Separating Hyperplane

Fat Hyperplane

• robust

separating hyperplane:

fat

—far from both sides of examples

• robustness

≡

fatness: distance to closest x _n

goal: find

fattest

separating hyperplane

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 8/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Fat Hyperplane

• robust

separating hyperplane:

fat

—far from both sides of examples

• robustness

≡

fatness: distance to closest x _n

goal: find

fattest

separating hyperplane

Linear Support Vector Machine Large-Margin Separating Hyperplane

Large-Margin Separating Hyperplane

max

w fatness(w)

subject to

w classifies every (x _n

, y

n

)correctly

fatness(w) =

min

n=1,...,N

distance(x

_n

, w)

max

w margin(w)

subject to every

y _n w ^T x _n > 0

margin(w) =

min

n=1,...,N

distance(x

_n

, w)

•

fatness: formally called

margin

• correctness: y _n

=sign(w

^T x _n

)

goal: find

largest-margin separating

hyperplane

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 9/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Large-Margin Separating Hyperplane

max

w margin(w)

subject to

w classifies every (x _n

, y

n

)correctly

margin(w) =

min

n=1,...,N

distance(x

_n

, w)

max

w margin(w)

subject to every

y _n w ^T x _n > 0

margin(w) =

min

n=1,...,N

distance(x

_n

, w)

•

fatness: formally called

margin

• correctness: y _n

=sign(w

^T x _n

)

goal: find

largest-margin

separating

hyperplane

Linear Support Vector Machine Large-Margin Separating Hyperplane

Large-Margin Separating Hyperplane

max

w margin(w)

subject to every

y _n w ^T x _n > 0

margin(w) =

min

n=1,...,N

distance(x

_n

, w)

•

fatness: formally called

margin

• correctness: y _n

=sign(w

^T x _n

)

goal: find

largest-margin separating

hyperplane

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 9/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Large-Margin Separating Hyperplane

max

w margin(w)

subject to every

y _n w ^T x _n > 0

margin(w) =

min

n=1,...,N

distance(x

_n

, w)

•

fatness: formally called

margin

• correctness: y _n

=sign(w

^T x _n

)

goal: find

largest-margin

separating

hyperplane

Linear Support Vector Machine Large-Margin Separating Hyperplane

Fun Time

Consider two examples (v, +1) and (−v, −1) where v ∈ R

²

(without padding the v

₀

=1). Which of the following hyperplane is the

largest-margin separating

one for the two examples? You are highly encouraged to visualize by considering, for instance,

v = (3, 2).

1

x

₁

=0

2

x

₂

=0

3

v

₁

x

₁

+v

₂

x

₂

=0

4

v

₂

x

₁

+v

₁

x

₂

=0

Reference Answer: 3

Here the

largest-margin separating

hyperplane (line) must be a perpendicular bisector of the line segment between

v and

−v. Hence v is a normal vector of the largest-margin line. The result can be extended to the more general case of

v

∈ R

^d

.

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 10/28

Linear Support Vector Machine Large-Margin Separating Hyperplane

Fun Time

Consider two examples (v, +1) and (−v, −1) where v ∈ R

²

(without padding the v

₀

=1). Which of the following hyperplane is the

largest-margin separating

one for the two examples? You are highly encouraged to visualize by considering, for instance,

v = (3, 2).

1

x

₁

=0

2

x

₂

=0

3

v

₁

x

₁

+v

₂

x

₂

=0

4

v

₂

x

₁

+v

₁

x

₂

=0

Reference Answer: 3

Here the

largest-margin separating

hyperplane (line) must be a perpendicular bisector of the line segment between

v and

−v. Hence v is a normal vector of the largest-margin line. The result can be extended to the more general case of

v

∈ R

^d

.

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane: Preliminary

max w margin(w)

subject to every y n w ^T x _n > 0 margin(w) = min

n=1,...,N distance(x _n , w)

‘shorten’ x and w

distance

needs

w ₀

and

(w ₁ , . . . , w _d )

differently (to be derived)

b

=

w ₀







| w

|







=





 w ₁

.. . w _d







;

   XX x ₀ = X X 1







| x

|







=





 x ₁

.. . x _d







for this part: h(x) = sign(w

^T x

+

b)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 11/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane: Preliminary

max w margin(w)

subject to every y n w ^T x _n > 0 margin(w) = min

n=1,...,N distance(x _n , w)

‘shorten’ x and w

distance

needs

w ₀

and

(w ₁ , . . . , w _d )

differently (to be derived)

b

=

w ₀







| w

|







=





 w ₁

.. . w _d







;

   XX x ₀ = X X 1







| x

|







=





 x ₁

.. . x _d







for this part: h(x) = sign(w

^T x

+

b)

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane: Preliminary

max w margin(w)

subject to every y n w ^T x _n > 0 margin(w) = min

n=1,...,N distance(x _n , w)

‘shorten’ x and w

distance

needs

w ₀

and

(w ₁ , . . . , w _d )

differently (to be derived)

b

=

w ₀







| w

|







=





 w ₁

.. . w _d







;

   XX x ₀ = X X 1







| x

|







=





 x ₁

.. . x _d







for this part: h(x) = sign(w

^T x

+

b)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 11/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane: Preliminary

max w margin(w)

subject to every y n w ^T x _n > 0 margin(w) = min

n=1,...,N distance(x _n , w)

‘shorten’ x and w

distance

needs

w ₀

and

(w ₁ , . . . , w _d )

differently (to be derived)

b

=

w ₀







| w

|







=





 w ₁

.. . w _d







;

   XX x ₀ = X X 1







| x

|







=





 x ₁

.. . x _d







for this part: h(x) = sign(w

^T x

+

b)

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

+

b

=0

consider

x ⁰

,

x ⁰⁰

on hyperplane

1 w ^T x ⁰

=

−

b

,

w ^T x ⁰⁰

=

−

b

2 w

⊥ hyperplane:









w ^T

(x

⁰⁰

−

x ⁰

)

| {z } vector on hyperplane









=0

3

distance = project (x−

x ⁰

)to

⊥ hyperplane

dist(x, h)

x^′ x^′′ w x

distance(x,

b, w) =

   

w ^T

k

w

k

(x−

x ⁰

)    

=

1

1 k

w

k|

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

+

b

=0

consider

x ⁰

,

x ⁰⁰

on hyperplane

1 w ^T x ⁰

=

−

b

,

w ^T x ⁰⁰

=

−

b

2 w

⊥ hyperplane:









w ^T

(x

⁰⁰

−

x ⁰

)

| {z } vector on hyperplane









=

0

3

distance = project (x−

x ⁰

)to

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

   

w ^T

k

w

k

(x−

x ⁰

)    

=

1

1 k

w

k|

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

+

b

=0

consider

x ⁰

,

x ⁰⁰

on hyperplane

1 w ^T x ⁰

=

−

b

,

w ^T x ⁰⁰

=

−

b

2 w

⊥ hyperplane:









w ^T

(x

⁰⁰

−

x ⁰

)

| {z } vector on hyperplane









=

0

3

distance = project (x−

x ⁰

)to

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

   

w ^T

k

w

k

(x−

x ⁰

)    

=

1

1 k

w

k|

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

+

b

=0

consider

x ⁰

,

x ⁰⁰

on hyperplane

1 w ^T x ⁰

= −

b, w ^T x ⁰⁰

= −

b

2 w

⊥ hyperplane:









w ^T

(x

⁰⁰

−

x ⁰

)

| {z } vector on hyperplane









=

0

3

distance = project (x−

x ⁰

)to

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

   

w ^T

k

w

k

(x−

x ⁰

)    

=

1

1 k

w

k|

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

+

b

=0

consider

x ⁰

,

x ⁰⁰

on hyperplane

1 w ^T x ⁰

= −

b, w ^T x ⁰⁰

= −

b

2 w

⊥ hyperplane:









w ^T

(x

⁰⁰

−

x ⁰

)

| {z } vector on hyperplane









=

0

3

distance = project (x−

x ⁰

)to

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

   

w ^T

k

w

k

(x−

x ⁰

)    

=

1

1 k

w

k|

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

+

b

=0

consider

x ⁰

,

x ⁰⁰

on hyperplane

1 w ^T x ⁰

= −

b, w ^T x ⁰⁰

= −

b

2 w

⊥ hyperplane:









w ^T

(x

⁰⁰

−

x ⁰

)

| {z } vector on hyperplane









=0

3

distance = project (x−

x ⁰

)to

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

   

w ^T

k

w

k

(x−

x ⁰

)    

=

1

1 k

w

k|

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

+

b

=0

consider

x ⁰

,

x ⁰⁰

on hyperplane

1 w ^T x ⁰

= −

b, w ^T x ⁰⁰

= −

b

2 w

⊥ hyperplane:









w ^T

(x

⁰⁰

−

x ⁰

)

| {z } vector on hyperplane









=0

3

distance = project (x−

x ⁰

)to

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

   

w ^T

k

w

k

(x−

x ⁰

)    

=

1

1 k

w

k|

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

+

b

=0

consider

x ⁰

,

x ⁰⁰

on hyperplane

1 w ^T x ⁰

= −

b, w ^T x ⁰⁰

= −

b

2 w

⊥ hyperplane:









w ^T

(x

⁰⁰

−

x ⁰

)

| {z } vector on hyperplane









=0

3

distance = project (x−

x ⁰

)to

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

   

w ^T

k

w

k

(x−

x ⁰

)    

=

1

1 k

w

k|

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

+

b

=0

consider

x ⁰

,

x ⁰⁰

on hyperplane

1 w ^T x ⁰

= −

b, w ^T x ⁰⁰

= −

b

2 w

⊥ hyperplane:









w ^T

(x

⁰⁰

−

x ⁰

)

| {z } vector on hyperplane









=0

3

distance = project (x−

x ⁰

)to

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

   

w ^T

k

w

k(x−

x ⁰

)    

=

1

1 k

w

k|

w ^T x

+

b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

+

b

=0

consider

x ⁰

,

x ⁰⁰

on hyperplane

1 w ^T x ⁰

= −

b, w ^T x ⁰⁰

= −

b

2 w

⊥ hyperplane:









w ^T

(x

⁰⁰

−

x ⁰

)

| {z } vector on hyperplane









=0

3

distance = project (x−

x ⁰

)to

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

   

w ^T

k

w

k(x−

x ⁰

)    

=

1

1 k

w

k|

w ^T x

+

b

|

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Hyperplane

want: distance(x,

b, w), with hyperplane w ^T x ⁰

+

b

=0

consider

x ⁰

,

x ⁰⁰

on hyperplane

1 w ^T x ⁰

= −

b, w ^T x ⁰⁰

= −

b

2 w

⊥ hyperplane:









w ^T

(x

⁰⁰

−

x ⁰

)

| {z } vector on hyperplane









=0

3

distance = project (x−

x ⁰

)to

⊥ hyperplane

dist(x, h)

x^′ x^′′

w x

distance(x,

b, w) =

   

w ^T

k

w

k(x−

x ⁰

)    

=

1

1

k

w

k|

w ^T x + b

|

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 12/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Separating Hyperplane

distance(x,

b, w) =

1

k

w

k|

w ^T x + b

|

• separating

hyperplane: for every n

y n (w ^T x _n + b) > 0

•

distance to

separating

hyperplane: distance(x

_n

,

b, w) =

1

k

w

k

y _n

(w

^T x _n

+

b)

max

b,w

margin(b,

w)

subject to every

y n (w ^T x _n + b) > 0

margin(b,

w) =

min

n=1,...,N

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Separating Hyperplane

distance(x,

b, w) =

1

k

w

k|

w ^T x + b

|

• separating

hyperplane: for every n

y n (w ^T x n + b) > 0

•

distance to

separating

hyperplane: distance(x

_n

,

b, w) =

1

k

w

k

y _n

(w

^T x _n

+

b)

max

b,w

margin(b,

w)

subject to every

y n (w ^T x _n + b) > 0

margin(b,

w) =

min

n=1,...,N

distance(x

_n

,

b, w)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 13/28

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Separating Hyperplane

distance(x,

b, w) =

1

k

w

k|

w ^T x + b

|

• separating

hyperplane: for every n

y n (w ^T x n + b) > 0

•

distance to

separating

hyperplane:

distance(x

_n

,

b, w) =

1

k

w

k

y _n

(w

^T x _n

+

b)

max

b,w

margin(b,

w)

subject to every

y n (w ^T x _n + b) > 0

margin(b,

w) =

min

n=1,...,N

distance(x

_n

,

b, w)

Linear Support Vector Machine Standard Large-Margin Problem

Distance to Separating Hyperplane

distance(x,

b, w) =

1

k

w

k|

w ^T x + b

|

• separating

hyperplane: for every n

y n (w ^T x n + b) > 0

•

distance to

separating

hyperplane:

distance(x

_n

,

b, w) =

1

k

w

k

y _n

(w

^T x _n

+

b)

max

b,w

margin(b,

w)

subject to every

y n (w ^T x _n + b) > 0

margin(b,

w) =

min

n=1,...,N 1

kwk y _n

(w

^T x _n

+

b)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 13/28

Linear Support Vector Machine Standard Large-Margin Problem

Margin of Special Separating Hyperplane

max

b,w

margin(b,

w)

subject to every y

_n

(w

^T x _n

+

b)

> 0 margin(b,

w) =

min

n=1,...,N 1

kwk

y

_n

(w

^T x _n

+

b)

• w ^T x + b

=0 same as 3w

^T x + 3b

=0: scaling does not matter

• special

scaling: only consider separating (b,

w)

such that

min

n=1,...,N y _n (w ^T x _n + b) = 1

=⇒

margin(b,

w) = _kwk ¹

max

b,w 1 kwk

subject to every y

n

(w

^T x _n

+b)> 0

min

n=1,...,N y _n (w ^T x _n + b) = 1

Linear Support Vector Machine Standard Large-Margin Problem

Margin of Special Separating Hyperplane

max

b,w

margin(b,

w)

subject to every y

_n

(w

^T x _n

+

b)

> 0 margin(b,

w) =

min

n=1,...,N 1

kwk

y

_n

(w

^T x _n

+

b)

• w ^T x + b

=0 same as 3w

^T x + 3b

=0: scaling does not matter

• special

scaling: only consider separating (b,

w)

such that

min

n=1,...,N y _n (w ^T x _n + b) = 1

=⇒

margin(b,

w) = _kwk ¹

max

b,w 1 kwk

subject to every y

n

(w

^T x _n

+b)> 0

min

n=1,...,N y _n (w ^T x _n + b) = 1

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 14/28

Linear Support Vector Machine Standard Large-Margin Problem

Margin of Special Separating Hyperplane

max

b,w

margin(b,

w)

subject to every y

_n

(w

^T x _n

+

b)

> 0 margin(b,

w) =

min

n=1,...,N 1

kwk

y

_n

(w

^T x _n

+

b)

• w ^T x + b

=0 same as 3w

^T x + 3b

=0: scaling does not matter

• special

scaling: only consider separating (b,

w)

such that

n=1,...,N min y n (w ^T x _n + b) = 1

=⇒

margin(b,

w) = _kwk ¹

max

b,w 1 kwk

subject to every y

n

(w

^T x _n

+b)> 0

min

n=1,...,N y _n (w ^T x _n + b) = 1

Linear Support Vector Machine Standard Large-Margin Problem

Margin of Special Separating Hyperplane

max

b,w

margin(b,

w)

subject to every y

_n

(w

^T x _n

+

b)

> 0 margin(b,

w) =

min

n=1,...,N 1

kwk

y

_n

(w

^T x _n

+

b)

• w ^T x + b

=0 same as 3w

^T x + 3b

=0: scaling does not matter

• special

scaling: only consider separating (b,

w)

such that

n=1,...,N min y n (w ^T x _n + b) = 1

=⇒ margin(

b, w) = _kwk ¹

max

b,w 1 kwk

subject to every y

n

(w

^T x _n

+b)> 0

min

n=1,...,N y _n (w ^T x _n + b) = 1

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 14/28

Linear Support Vector Machine Standard Large-Margin Problem

Margin of Special Separating Hyperplane

max

b,w

margin(b,

w)

subject to every y

_n

(w

^T x _n

+

b)

> 0 margin(b,

w) =

min

n=1,...,N 1

kwk

y

_n

(w

^T x _n

+

b)

• w ^T x + b

=0 same as 3w

^T x + 3b

=0: scaling does not matter

• special

scaling: only consider separating (b,

w)

such that

n=1,...,N min y n (w ^T x _n + b) = 1

=⇒ margin(

b, w) = _kwk ¹

max

b,w 1 kwk

subject to every y

n

(w

^T x _n

+b)> 0

min

n=1,...,N y _n (w ^T x _n + b) = 1

Linear Support Vector Machine Standard Large-Margin Problem

Margin of Special Separating Hyperplane

max

b,w

margin(b,

w)

subject to every y

_n

(w

^T x _n

+

b)

> 0 margin(b,

w) =

min

n=1,...,N 1

kwk

y

_n

(w

^T x _n

+

b)

• w ^T x + b

=0 same as 3w

^T x + 3b

=0: scaling does not matter

• special

scaling: only consider separating (b,

w)

such that

n=1,...,N min y n (w ^T x _n + b) = 1

=⇒ margin(

b, w) = _kwk ¹

max

b,w 1 kwk

subject to

every y n (w ^T x _n + b) > 0 min

n=1,...,N y _n (w ^T x _n + b) = 1

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 14/28

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

k subject to

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

(w

^T x _n

+

b)

≥ 1 for all n

original constraint:

min _n=1,...,N y n (w ^T x n + b) = 1

want: optimal (b,

w) here (inside)

if optimal (b,

w)

outside, e.g. y

_n

(w

^T x _n

+

b)

>

1.126

for all n

—can scale (b,

w)

to “more optimal” (

_1.126 ^b

,

_1.126 ^w

)

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

min

b,w

1 2 w ^T w

subject to y

_n

(w

^T x _n

+

b)

≥ 1

for all n

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

k subject to

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

(w

^T x _n

+

b)

≥ 1 for all n original constraint:

min _n=1,...,N y n (w ^T x n + b) = 1

want: optimal (b,

w) here (inside)

if optimal (b,

w)

outside, e.g. y

_n

(w

^T x _n

+

b)

>

1.126

for all n

—can scale (b,

w)

to “more optimal” (

_1.126 ^b

,

_1.126 ^w

)

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

min

b,w

1 2 w ^T w

subject to y

_n

(w

^T x _n

+

b)

≥ 1

for all n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 15/28

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

k subject to

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

(w

^T x _n

+

b)

≥ 1 for all n original constraint:

min _n=1,...,N y n (w ^T x n + b) = 1

want: optimal (b,

w) here (inside)

if optimal (b,

w)

outside, e.g. y

_n

(w

^T x _n

+

b)

>

1.126

for all n

—can scale (b,

w)

to “more optimal” (

_1.126 ^b

,

_1.126 ^w

)

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

min

b,w

1 2 w ^T w

subject to y

_n

(w

^T x _n

+

b)

≥ 1

for all n

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

k subject to

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

(w

^T x _n

+

b)

≥ 1 for all n original constraint:

min _n=1,...,N y n (w ^T x n + b) = 1

want: optimal (b,

w) here (inside)

if optimal (b,

w)

outside,

e.g. y

_n

(w

^T x _n

+

b)

>

1.126

for all n

—can scale (b,

w)

to “more optimal” (

_1.126 ^b

,

_1.126 ^w

)

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

min

b,w

1 2 w ^T w

subject to y

_n

(w

^T x _n

+

b)

≥ 1

for all n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 15/28

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

k subject to

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

(w

^T x _n

+

b)

≥ 1 for all n original constraint:

min _n=1,...,N y n (w ^T x n + b) = 1

want: optimal (b,

w) here (inside)

if optimal (b,

w)

outside, e.g. y

_n

(w

^T x _n

+

b)

>

1.126

for all n

—can scale (b,

w)

to “more optimal” (

_1.126 ^b

,

_1.126 ^w

)

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

min

b,w

1 2 w ^T w

subject to y

_n

(w

^T x _n

+

b)

≥ 1

for all n

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

k subject to

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

(w

^T x _n

+

b)

≥ 1 for all n original constraint:

min _n=1,...,N y n (w ^T x n + b) = 1

want: optimal (b,

w) here (inside)

if optimal (b,

w)

outside, e.g. y

_n

(w

^T x _n

+

b)

> 1.126 for all n

—can scale (b,

w)

to “more optimal” (

_1.126 ^b

,

_1.126 ^w

)

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

min

b,w

1 2 w ^T w

subject to y

_n

(w

^T x _n

+

b)

≥ 1

for all n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 15/28

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

k subject to

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

(w

^T x _n

+

b)

≥ 1 for all n original constraint:

min _n=1,...,N y n (w ^T x n + b) = 1

want: optimal (b,

w) here (inside)

if optimal (b,

w)

outside, e.g. y

_n

(w

^T x _n

+

b)

> 1.126 for all n

—can scale (b,

w)

to “more optimal” (

_1.126 ^b

,

_1.126 ^w

)

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

min

b,w

1 2 w ^T w

subject to y

_n

(w

^T x _n

+

b)

≥ 1

for all n

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

k subject to

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

(w

^T x _n

+

b)

≥ 1 for all n original constraint:

min _n=1,...,N y n (w ^T x n + b) = 1

want: optimal (b,

w) here (inside)

if optimal (b,

w)

outside, e.g. y

_n

(w

^T x _n

+

b)

> 1.126 for all n

—can scale (b,

w)

to “more optimal” (

_1.126 ^b

,

_1.126 ^w

)

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

min

b,w

1 2 w ^T w

subject to y

_n

(w

^T x _n

+

b)

≥ 1

for all n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 15/28

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

k subject to

min

n=1,...,N y _n (w ^T x _n + b) = 1

necessary constraints: y

_n

(w

^T x _n

+

b)

≥ 1 for all n original constraint:

min _n=1,...,N y n (w ^T x n + b) = 1

want: optimal (b,

w) here (inside)

if optimal (b,

w)

outside, e.g. y

_n

(w

^T x _n

+

b)

> 1.126 for all n

—can scale (b,

w)

to “more optimal” (

_1.126 ^b

,

_1.126 ^w

)

(contradiction!)

final change: max =⇒ min, remove√

w

, add

¹ ₂

max

b,w

min

b,w 1 2 w ^T w

1 kwk

subject to y

_n

(w

^T x _n

+

b)

≥ 1 for all n

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

k subject to

min

n=1,...,N y _n (w ^T x _n + b) = 1

necessary constraints: y

_n

(w

^T x _n

+

b)

≥ 1 for all n original constraint:

min _n=1,...,N y n (w ^T x n + b) = 1

want: optimal (b,

w) here (inside)

if optimal (b,

w)

outside, e.g. y

_n

(w

^T x _n

+

b)

> 1.126 for all n

—can scale (b,

w)

to “more optimal” (

_1.126 ^b

,

_1.126 ^w

)

(contradiction!)

final change: max =⇒ min, remove√

w, add ¹ ₂

max

b,w

min

b,w 1 2 w ^T w

1 kwk

subject to y

_n

(w

^T x _n

+

b)

≥ 1 for all n

Hsuan-Tien Lin (NTU CSIE) Machine Learning Techniques 15/28

Linear Support Vector Machine Standard Large-Margin Problem

Standard Large-Margin Hyperplane Problem

max

b,w

1

k

w

k subject to

min

n=1,...,N y n (w ^T x _n + b) = 1

necessary constraints: y

_n

(w

^T x _n

+

b)

≥ 1 for all n original constraint:

min _n=1,...,N y n (w ^T x n + b) = 1

want: optimal (b,

w) here (inside)

if optimal (b,

w)

outside, e.g. y

_n

(w

^T x _n

+

b)

> 1.126 for all n

—can scale (b,

w)

to “more optimal” (

_1.126 ^b

,

_1.126 ^w

)

(contradiction!)

final change: max =⇒ min, remove√

w, add ¹ ₂

min

b,w

1 2

w ^T w

subject to y

_n

(w

^T x _n

+

b)

≥ 1 for all n