Machine Learning Foundations (ᘤ9M)

Machine Learning Foundations ( 機器學習基石)

Lecture 12: Nonlinear Transformation

Hsuan-Tien Lin (林軒田) htlin@csie.ntu.edu.tw

Department of Computer Science

& Information Engineering

National Taiwan University

( 國立台灣大學資訊工程系)

Nonlinear Transformation

Roadmap

1 When Can Machines Learn?

2 Why Can Machines Learn?

3 How

Can Machines Learn?

Lecture 11: Linear Models for Classification binary classification

via

(logistic) regression;

multiclass

via

OVA/OVO decomposition Lecture 12: Nonlinear Transformation

Quadratic Hypotheses Nonlinear Transform

Price of Nonlinear Transform Structured Hypothesis Sets

4 How Can Machines Learn Better?

Nonlinear Transformation Quadratic Hypotheses

Linear Hypotheses

up to now: linear hypotheses

•

visually:

‘line’-like

boundary

•

mathematically: linear scores

s

=

w ^T x

but limited . . .

−1 0 1

−1 0 1

•

theoretically:

d

VC

under control :-)

•

practically: on someD,

large E _in

for every line

:-(

how to

break the limit

of linear hypotheses

Nonlinear Transformation Quadratic Hypotheses

Circular Separable

−1 0 1

−1 0 1

−1 0 1

−1 0 1

•

D not linear separable

•

but

circular separable

by a circle of radius√

0.6 centered at origin:

hSEP(x) = sign

−x

1 ²

− x

2 ²

+0.6

re-derive

Circular-PLA, Circular-Regression,

blahblah. . . all over again?

:-)

Nonlinear Transformation Quadratic Hypotheses

Circular Separable and Linear Separable

h(x) = sign







0.6

|{z}

w ˜

0

·

1

|{z}

z

₀

+(

−1

)

| {z }

w ˜

₁

·

x ₁ ²

|{z}

z

1

+(

−1

)

| {z }

w ˜

₂

·

x ₂ ²

|{z}

z

2







= sign

w ˜ ^T z



x

1

x

2

−1 0 1

−1 0

1

•

{(x

ⁿ

, y

n

)} circular separable

=⇒ {(

z _n

, y

_n

)}

linear

separable

• x

∈ X 7−→

^Φ z ∈ Z

:

(nonlinear) feature

transform Φ z

1

z

2

0 0.5 1

0 0.5 1

circular separable inX =⇒

linear

separable in

Z

vice versa?

Nonlinear Transformation Quadratic Hypotheses

Linear Hypotheses in Z -Space

(z ₀ , z ₁ , z ₂ )

=

z

=

Φ(x) = (1, x ₁ ²

,

x ₂ ²

) h(x) =

˜ h(z) =

sign

w ˜ ^T Φ(x)



=sign

w ˜ ₀

+

w ˜ ₁ x ₁ ²

+

w ˜ ₂ x ₂ ²



w ˜ = ( w ˜ 0 , w ˜ 1 , w ˜ 2 )

•

(0.6,−1, −1): circle (◦inside)

•

(−0.6, +1, +1): circle (◦outside)

•

(0.6,−1, −2): ellipse

•

(0.6,−1, +2): hyperbola

•

(0.6, +1, +2):

constant ◦ :-)

lines in

Z

-space

⇐⇒

special

quadratic curves inX -space

Nonlinear Transformation Quadratic Hypotheses

General Quadratic Hypothesis Set

a ‘bigger’

Z

-space with

Φ ₂

(x) = (1,

x ₁

,

x ₂

,

x ₁ ²

,

x ₁ x ₂

,

x ₂ ²

) perceptrons in

Z

-space⇐⇒ quadratic hypotheses in X -space

H

Φ

₂ = n

h(x) : h(x) =

h(Φ ˜ ₂

(x)) for some linear

˜ h

on

Z

o

•

can

implement all possible quadratic curve boundaries:

circle, ellipse,

rotated ellipse, hyperbola, parabola,

. . .

⇐=

ellipse 2(x

₁

+x

₂

− 3)

²

+ (x

₁

− x

2

− 4)

²

=1

⇐=

w ˜ ^T

=

[33, −20, −4, 3, 2, 3]

•

include

lines and constants as degenerate cases

next:

learn

a good quadratic hypothesis g

Nonlinear Transformation Quadratic Hypotheses

Fun Time

Using the transform

Φ ₂

(x) = (1,

x ₁

,

x ₂

,

x ₁ ²

,

x ₁ x ₂

,

x ₂ ²

), which of the following weights

w ˜ ^T

in the

Z

-space implements the parabola 2x

₁ ²

+x

₂

=1?

1 [ −1, 2, 1, 0, 0, 0]

2 [0, 2, 1, 0, −1, 0]

3 [ −1, 0, 1, 2, 0, 0]

4 [ −1, 2, 0, 0, 0, 1]

Reference Answer: 3

Too simple, uh? :-)

Flexibility to implement arbitrary quadratic curves opens new possibilities for minimizing E

_in

!

Nonlinear Transformation Quadratic Hypotheses

Fun Time

Using the transform

Φ ₂

(x) = (1,

x ₁

,

x ₂

,

x ₁ ²

,

x ₁ x ₂

,

x ₂ ²

), which of the following weights

w ˜ ^T

in the

Z

-space implements the parabola 2x

₁ ²

+x

₂

=1?

1 [ −1, 2, 1, 0, 0, 0]

2 [0, 2, 1, 0, −1, 0]

3 [ −1, 0, 1, 2, 0, 0]

4 [ −1, 2, 0, 0, 0, 1]

Reference Answer: 3

Too simple, uh? :-)

Flexibility to implement arbitrary quadratic curves opens new possibilities for minimizing E

_in

!

Nonlinear Transformation Nonlinear Transform

Good Quadratic Hypothesis

Z

-space X -space

perceptrons

⇐⇒ quadratic hypotheses

good perceptron

⇐⇒

good quadratic hypothesis separating perceptron

⇐⇒ separating quadratic hypothesis

z1

z2

0 0.5 1

0 0.5 1

⇐⇒

x1

x2

−1 0 1

−1 0 1

•

want: get

good perceptron

in

Z

-space

•

known: get

good perceptron

in

X

-space with data{(

x _n

, y

n

)} todo: get

good perceptron

in

Z

-space with data{(

z _n

=

Φ ₂

(x

_n

), y

n

)}

Nonlinear Transformation Nonlinear Transform

The Nonlinear Transform Steps

−1 0 1

−1 0 1

−→

Φ

0 0.5 1

0 0.5 1

↓ A

−1 0 1

−1 0 1

Φ

⁻¹

←−

−→

Φ

0 0.5 1

0 0.5 1

1

transform original data{(x

ⁿ

, y

n

)} to {(

z n

=

Φ(x n

), y

n

)} by

Φ

2

get a good perceptron

w ˜

using{(

z n

, y

n

)}

and your favorite linear classification algorithmA

3

return g(x) = sign

w ˜ ^T Φ(x)

Nonlinear Transformation Nonlinear Transform

Nonlinear Model via Nonlinear Φ + Linear Models

−1 0 1

−1 0 1

−→

Φ

0 0.5 1

0 0.5 1

↓ A

−1 0 1

−1 0 1

Φ

⁻¹

←−

−→

Φ

0 0.5 1

0 0.5 1

two choices:

•

feature transform

Φ

•

linear modelA,

not just binary classification

Pandora’s box :-):

can now freely do

quadratic PLA, quadratic regression,

cubic regression, . . ., polynomial regression

Nonlinear Transformation Nonlinear Transform

Feature Transform Φ

−→

Φ

Average Intensity

Symmetry

not 1 1

↓ A

Φ

⁻¹

←−

−→

Φ

Average Intensity

Symmetry

not new, not just polynomial:

raw (pixels)

domain knowledge

−→

concrete (intensity, symmetry)

the force, too good to be true? :-)

Nonlinear Transformation Nonlinear Transform

Fun Time

Consider the quadratic transform

Φ ₂

(x) for x∈ R

^d

instead of in R

²

. The transform should include all different quadratic, linear, and constant terms formed by (x

₁

, x

2

, . . . , x

d

). What is the number of dimensions of

z

=

Φ ₂

(x)?

1

d

2 d

²

2

+

^3d ₂

+1

3

d

²

+d + 1

4

2

^d

Reference Answer: 2

Number of different quadratic terms is

^d ₂

+ d; number of different linear terms is d ;

number of different constant term is 1.

Nonlinear Transformation Nonlinear Transform

Fun Time

Consider the quadratic transform

Φ ₂

(x) for x∈ R

^d

instead of in R

²

. The transform should include all different quadratic, linear, and constant terms formed by (x

₁

, x

2

, . . . , x

d

). What is the number of dimensions of

z

=

Φ ₂

(x)?

1

d

2 d

²

2

+

^3d ₂

+1

3

d

²

+d + 1

4

2

^d

Reference Answer: 2

Number of different quadratic terms is

^d ₂

+ d;

number of different linear terms is d ; number of different constant term is 1.

Nonlinear Transformation Price of Nonlinear Transform

Computation/Storage Price

Q-th order polynomial transform: Φ

_Q

(x) =  1,

x

₁

, x

2

, . . . , x

d

, x

₁²

, x

1

x

2

, . . . , x

_d²

, . . . ,

x

₁^Q

, x

₁^Q−1

x

2

, . . . , x

_d^Q



=

1

|{z}

w ˜

0

+

d ˜

|{z}

others

dimensions

= # ways of≤ Q-combination from d kinds with repetitions

=

^Q+d _Q

=

^Q+d _d

=

O Q ^d

= efforts needed for computing/storing

z

=

Φ _Q

(x) and

w ˜

Q large =⇒

difficult to compute/store

Nonlinear Transformation Price of Nonlinear Transform

Model Complexity Price

Q-th order polynomial transform: Φ

_Q

(x) =  1,

x

₁

, x

2

, . . . , x

d

, x

₁²

, x

1

x

2

, . . . , x

_d²

, . . . ,

x

₁^Q

, x

₁^Q−1

x

₂

, . . . , x

_d^Q



1

|{z}

w ˜

0

+

d ˜

|{z}

others

dimensions =

O Q ^d

•

number of free parameters ˜w

_i

=

d ˜

+1

≈ d

^VC

( H Φ

_Q

)

• d

VC

( H Φ

_Q

) ≤ ˜d + 1

, why?

=⇒

any ˜d + 2 inputs not shattered inZ

=⇒ any ˜d + 2 inputs not shattered in X Q large =⇒

large d

VC

Nonlinear Transformation Price of Nonlinear Transform

Generalization Issue

Φ

1

(original

x)

which one do you prefer? :-)

•

Φ

1

‘visually’ preferred

•

Φ

4

: E

_in

(g) = 0 but overkill

Φ

₄

1 can we make sure that E out (g) is close enough to E _in (g)?

2 can we make E _in (g) small enough?

trade-off:

d (Q)˜

1 2

higher :-(

:-D

lower

:-D

:-(

how to pick Q?

visually, maybe?

Nonlinear Transformation Price of Nonlinear Transform

Danger of Visual Choices

first of all, can you really ‘visualize’ whenX = R

¹⁰

?

(well, I can’t :-)) Visualize X = R ²

•

full Φ

₂

:

z = (1, x ₁

, x

₂

, x

₁ ²

, x

₁

x

₂

, x

₂ ²

), dVC =6

•

or

z = (1, x ₁ ²

, x

₂ ²

), d_VC =3,

after visualizing?

•

or better

z = (1, x ₁ ²

+x

₂ ²

), d_VC=2?

•

or even better

z = sign(0.6 − x 1 ² − x 2 ² )
?

—careful about

your brain’s ‘model complexity’

−1 0 1

−1 0 1

for VC-safety, Φ shall be decided

without ‘peeking’

data

Nonlinear Transformation Price of Nonlinear Transform

Fun Time

Consider the Q-th order polynomial transform

Φ _Q

(x) for x∈ R

²

. Recall that ˜d =

^Q+2 ₂

− 1. When Q = 50, what is the value of ˜d?

1

1126

2

1325

3

2651

4

6211

Reference Answer: 2

It’s just a simple calculation, but shows you how ˜d becomes hundreds of times of d = 2 after the transform.

Nonlinear Transformation Price of Nonlinear Transform

Fun Time

Consider the Q-th order polynomial transform

Φ _Q

(x) for x∈ R

²

. Recall that ˜d =

^Q+2 ₂

− 1. When Q = 50, what is the value of ˜d?

1

1126

2

1325

3

2651

4

6211

Reference Answer: 2

It’s just a simple calculation, but shows you how ˜d becomes hundreds of times of d = 2 after the transform.

Nonlinear Transformation Structured Hypothesis Sets

Polynomial Transform Revisited

Φ

₀

(x) =  1 

, Φ

₁

(x) = 

Φ

₀

(x), x

1

, x

2

, . . . , x

d



Φ

₂

(x) = 

Φ

₁

(x), x

₁²

, x

1

x

2

, . . . , x

_d²



Φ

₃

(x) = 

Φ

₂

(x), x

₁³

, x

₁²

x

2

, . . . , x

_d³

 . . . . . .

Φ

_Q

(x) = 

Φ

_Q−1

(x), x

₁^Q

, x

₁^Q−1

x

₂

, . . . , x

_d^Q



H

^Φ0

⊂ H

^Φ1

⊂ H

^Φ2

⊂ H

^Φ3

⊂ . . . ⊂ H

^ΦQ

k k k k k

H

⁰

H

¹

H

²

H

³

. . . H

^Q

H0 H1 H2 H3 · · ·

structure:

nested H i

Nonlinear Transformation Structured Hypothesis Sets

Structured Hypothesis Sets

H0 H1 H2 H3 · · ·

Let

g _i = argmin _h∈H

i

E _in (h):

H

⁰

⊂ H

¹

⊂ H

²

⊂ H

³

⊂ . . .

d

VC

( H

⁰

) ≤ d

VC

( H

¹

) ≤ d

VC

( H

²

) ≤ d

VC

( H

³

) ≤ . . . E

in

(g

0

) ≥ E

in

(g

1

) ≥ E

in

(g

2

) ≥ E

in

(g

3

) ≥ . . .

in-sample error model complexity out-of-sample error

VC dimension, dvc

Error

d^∗vc

use

H 1126

won’t be good!

:-(

Nonlinear Transformation Structured Hypothesis Sets

Linear Model First

in-sample error model complexity out-of-sample error

VC dimension, dvc

Error

d^∗_vc

•

tempting sin: useH

1126

, low

E _in (g ₁₁₂₆ )

to fool your boss

—really? :-( a dangerous path of no return

•

safe route: H

1

first

• if E

_in

(g

₁

) good enough, live happily thereafter :-)

• otherwise, move right of the curve

with nothing lost except ‘wasted’ computation

linear model first:

simple, efficient,

safe, and workable!

Nonlinear Transformation Structured Hypothesis Sets

Fun Time

Consider two hypothesis sets,H

1

andH

1126

, whereH

1

⊂ H

1126

. Which of the following relationship between d_VC(H

1

)and d_VC(H

1126

)is not possible?

1

dVC(H

1

) =dVC(H

1126

)

2

d_VC(H

1

)6= dVC(H

1126

)

3

d_VC(H

1

)< dVC(H

1126

)

4

dVC(H

1

)> dVC(H

1126

)

Reference Answer: 4

Every input combination thatH

1

shatters can be shattered byH

1126

, so d_VCcannot

decrease.

Nonlinear Transformation Structured Hypothesis Sets

Fun Time

Consider two hypothesis sets,H

1

andH

1126

, whereH

1

⊂ H

1126

. Which of the following relationship between d_VC(H

1

)and d_VC(H

1126

)is not possible?

1

dVC(H

1

) =dVC(H

1126

)

2

d_VC(H

1

)6= dVC(H

1126

)

3

d_VC(H

1

)< dVC(H

1126

)

4

dVC(H

1

)> dVC(H

1126

)

Reference Answer: 4

Every input combination thatH

1

shatters can be shattered byH

1126

, so dVCcannot

decrease.

Nonlinear Transformation Structured Hypothesis Sets

Summary

1 When Can Machines Learn?

2 Why Can Machines Learn?

3 How

Can Machines Learn?

Lecture 11: Linear Models for Classification Lecture 12: Nonlinear Transformation

Quadratic Hypotheses

linear hypotheses on quadratic-transformed data Nonlinear Transform

happy linear modeling after Z = Φ(X ) Price of Nonlinear Transform

computation/storage/[model complexity]

Structured Hypothesis Sets

linear/simpler model first

• next: dark side of the force :-)

4 How Can Machines Learn Better?