# Machine Learning Techniques (ᘤᢈ)

(1)

## Machine Learning Techniques ( 機器學習技巧)

### Lecture 13: RBF Networks

Hsuan-Tien Lin (林軒田) htlin@csie.ntu.edu.tw

(2)

(3)

## Disclaimer

### Prof. Yaser S. Abu-Mostafa’s slides with permission.

Learning From Data

YaserS.Abu-Mostafa

CaliforniaInstituteofTe hnology

(4)

### RBF Networks Full RBF Model

Basi RBFmodel

LearningFromData-Le ture16 3/20

Basi RBFmodel

Ea h

inuen es

basedon

### kx − x | {z } n k

LearningFromData-Le ture16 3/20

Basi RBFmodel

Ea h

inuen es

basedon

### k | {z } x − x n k

LearningFromData-Le ture16 3/20

Basi RBFmodel

Ea h

inuen es

basedon

Standard form:

### | {z }

basisfun tion

LearningFromData-Le ture16 3/20

Basi RBFmodel

Ea h

inuen es

basedon

Standard form:

### | {z }

basisfun tion

LearningFromData-Le ture16 3/20

Basi RBFmodel

Ea h

inuen es

basedon

Standard form:

### | {z }

basisfun tion

LearningFromData-Le ture16 3/20

Basi RBFmodel

Ea h

inuen es

basedon

Standard form:

### | {z }

basisfun tion

LearningFromData-Le ture16 3/20

Basi RBFmodel

Ea h

inuen es

basedon

Standard form:

### | {z }

basisfun tion

LearningFromData-Le ture16 3/20

Basi RBFmodel

Ea h

inuen es

basedon

Standard form:

### | {z }

basisfun tion

LearningFromData-Le ture16 3/20

Basi RBFmodel

Ea h

inuen es

basedon

Standard form:

### | {z }

basisfun tion

LearningFromData-Le ture16 3/20

Basi RBFmodel

Ea h

inuen es

basedon

Standard form:

### | {z }

basisfun tion

LearningFromData-Le ture16 3/20

(5)

### RBF Networks Full RBF Model

Thelearning algorithm

Finding

:

basedon

in

:

for

:

### = y n

LearningFromData-Le ture16 4/20

Thelearning algorithm

Finding

:

basedon

in

:

for

:

### = y n

LearningFromData-Le ture16 4/20

Thelearning algorithm

Finding

:

basedon

in

:

for

:

### = y n

LearningFromData-Le ture16 4/20

Thelearning algorithm

Finding

:

basedon

in

:

for

:

### = y n

LearningFromData-Le ture16 4/20

Thelearning algorithm

Finding

:

basedon

in

:

for

:

### = y n

LearningFromData-Le ture16 4/20

Thelearning algorithm

Finding

:

basedon

in

:

for

:

### = y n

LearningFromData-Le ture16 4/20

Thelearning algorithm

Finding

:

basedon

in

:

for

:

### = y n

LearningFromData-Le ture16 4/20

Thelearning algorithm

Finding

:

basedon

in

:

for

:

### = y n

LearningFromData-Le ture16 4/20

Thelearning algorithm

Finding

:

basedon

in

:

for

:

### = y n

LearningFromData-Le ture16 4/20

Thelearning algorithm

Finding

:

basedon

in

:

for

:

### = y n

LearningFromData-Le ture16 4/20

(6)

Thesolution

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

isinvertible,

### w = Φ −1 y

exa tinterpolation

LearningFromData-Le ture16 5/20

Thesolution

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

isinvertible,

### w = Φ −1 y

exa tinterpolation

LearningFromData-Le ture16 5/20

Thesolution

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

isinvertible,

### w = Φ −1 y

exa tinterpolation

LearningFromData-Le ture16 5/20

Thesolution

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

isinvertible,

### w = Φ −1 y

exa tinterpolation

LearningFromData-Le ture16 5/20

Thesolution

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

isinvertible,

### w = Φ −1 y

exa tinterpolation

LearningFromData-Le ture16 5/20

Thesolution

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

isinvertible,

### w = Φ −1 y

exa tinterpolation

LearningFromData-Le ture16 5/20

Thesolution

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

isinvertible,

### w = Φ −1 y

exa tinterpolation

LearningFromData-Le ture16 5/20

Thesolution

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

isinvertible,

### w = Φ −1 y

exa tinterpolation

LearningFromData-Le ture16 5/20

Thesolution

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

isinvertible,

### w = Φ −1 y

exa tinterpolation

LearningFromData-Le ture16 5/20

Thesolution

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

isinvertible,

### w = Φ −1 y

exa tinterpolation

LearningFromData-Le ture16 5/20

Thesolution

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

isinvertible,

### w = Φ −1 y

exa tinterpolation

LearningFromData-Le ture16 5/20

Thesolution

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

isinvertible,

### w = Φ −1 y

exa tinterpolation

LearningFromData-Le ture16 5/20

(7)

### RBF Networks Full RBF Model

RBFfor lassi ation

sign

Learning:

### ∼

linearregressionfor lassi ation

Minimize

on

sign

### (s)

LearningFromData-Le ture16 7/20

RBFfor lassi ation

sign

Learning:

### ∼

linearregressionfor lassi ation

Minimize

on

sign

### (s)

LearningFromData-Le ture16 7/20

RBFfor lassi ation

sign

Learning:

### ∼

linearregressionfor lassi ation

Minimize

on

sign

### (s)

LearningFromData-Le ture16 7/20

RBFfor lassi ation

sign

Learning:

### ∼

linearregressionfor lassi ation

Minimize

on

sign

### (s)

LearningFromData-Le ture16 7/20

RBFfor lassi ation

sign

Learning:

### ∼

linearregressionfor lassi ation

Minimize

on

sign

### (s)

LearningFromData-Le ture16 7/20

RBFfor lassi ation

sign

Learning:

### ∼

linearregressionfor lassi ation

Minimize

on

sign

### (s)

LearningFromData-Le ture16 7/20

RBFfor lassi ation

sign

Learning:

### ∼

linearregressionfor lassi ation

Minimize

on

sign

### (s)

LearningFromData-Le ture16 7/20

RBFfor lassi ation

sign

Learning:

### ∼

linearregressionfor lassi ation

Minimize

on

sign

### (s)

LearningFromData-Le ture16 7/20

RBFfor lassi ation

sign

Learning:

### ∼

linearregressionfor lassi ation

Minimize

on

sign

### (s)

LearningFromData-Le ture16 7/20

RBFfor lassi ation

sign

Learning:

### ∼

linearregressionfor lassi ation

Minimize

on

sign

### (s)

LearningFromData-Le ture16 7/20

RBFfor lassi ation

sign

Learning:

### ∼

linearregressionfor lassi ation

Minimize

on

sign

### (s)

LearningFromData-Le ture16 7/20

RBFfor lassi ation

sign

Learning:

### ∼

linearregressionfor lassi ation

Minimize

on

sign

### (s)

LearningFromData-Le ture16 7/20

(8)

### RBF Networks Full RBF Model

Relationshipto nearest-neighbor method

LearningFromData-Le ture16 8/20

Relationshipto nearest-neighbor method

### y

valueofanearbypoint: similaree tbyabasisfun tion:

LearningFromData-Le ture16 8/20

Relationshipto nearest-neighbor method

### y

valueofanearbypoint: similaree tbyabasisfun tion:

LearningFromData-Le ture16 8/20

Relationshipto nearest-neighbor method

### y

valueofanearbypoint: similaree tbyabasisfun tion:

LearningFromData-Le ture16 8/20

Relationshipto nearest-neighbor method

### y

valueofanearbypoint: similaree tbyabasisfun tion:

LearningFromData-Le ture16 8/20

(9)

## Fun Time

(10)

RBFwith

enters

parameters

basedon

datapoints

Use

enters:

### −γ kx − µ k k 2 

1.Howto hoosethe enters

### µ k

2.Howto hoosetheweights

### w k

LearningFromData-Le ture16 9/20

RBFwith

enters

parameters

basedon

datapoints

Use

enters:

### −γ kx − µ k k 2 

1.Howto hoosethe enters

### µ k

2.Howto hoosetheweights

### w k

LearningFromData-Le ture16 9/20

RBFwith

enters

parameters

basedon

datapoints

Use

enters:

### −γ kx − µ k k 2 

1.Howto hoosethe enters

### µ k

2.Howto hoosetheweights

### w k

LearningFromData-Le ture16 9/20

RBFwith

enters

parameters

basedon

datapoints

Use

enters:

### −γ kx − µ k k 2 

1.Howto hoosethe enters

### µ k

2.Howto hoosetheweights

### w k

LearningFromData-Le ture16 9/20

RBFwith

enters

parameters

basedon

datapoints

Use

enters:

### −γ kx − µ k k 2 

1.Howto hoosethe enters

### µ k

2.Howto hoosetheweights

### w k

LearningFromData-Le ture16 9/20

RBFwith

enters

parameters

basedon

datapoints

Use

enters:

### −γ kx − µ k k 2 

1.Howto hoosethe enters

### µ k

2.Howto hoosetheweights

### w k

LearningFromData-Le ture16 9/20

RBFwith

enters

parameters

basedon

datapoints

Use

enters:

### −γ kx − µ k k 2 

1.Howto hoosethe enters

### µ k

2.Howto hoosetheweights

### w k

LearningFromData-Le ture16 9/20

RBFwith

enters

parameters

basedon

datapoints

Use

enters:

### −γ kx − µ k k 2 

1.Howto hoosethe enters

### µ k

2.Howto hoosetheweights

### w k

LearningFromData-Le ture16 9/20

RBFwith

enters

parameters

basedon

datapoints

Use

enters:

### −γ kx − µ k k 2 

1.Howto hoosethe enters

### µ k

2.Howto hoosetheweights

### w k

LearningFromData-Le ture16 9/20

RBFwith

enters

parameters

basedon

datapoints

Use

enters:

### −γ kx − µ k k 2 

1.Howto hoosethe enters

### µ k

2.Howto hoosetheweights

### w k

LearningFromData-Le ture16 9/20

RBFwith

enters

parameters

basedon

datapoints

Use

enters:

### −γ kx − µ k k 2 

1.Howto hoosethe enters

### µ k

2.Howto hoosetheweights

### w k

LearningFromData-Le ture16 9/20

RBFwith

enters

parameters

basedon

datapoints

Use

enters:

### −γ kx − µ k k 2 

1.Howto hoosethe enters

### µ k

2.Howto hoosetheweights

### w k

LearningFromData-Le ture16 9/20

(11)

### RBF Networks Prototype Extraction

Choosingthe enters

Minimizethedistan ebetween

### x n

andthe losest enter

:

-means lustering

Split

into lusters

Minimize

### kx n − µ k k 2

Unsupervisedlearning

NP-hard

LearningFromData-Le ture16 10/20

Choosingthe enters

Minimizethedistan ebetween

### x n

andthe losest enter

:

-means lustering

Split

into lusters

Minimize

### kx n − µ k k 2

Unsupervisedlearning

NP-hard

LearningFromData-Le ture16 10/20

Choosingthe enters

Minimizethedistan ebetween

### x n

andthe losest enter

:

-means lustering

Split

into lusters

Minimize

### kx n − µ k k 2

Unsupervisedlearning

NP-hard

LearningFromData-Le ture16 10/20

Choosingthe enters

Minimizethedistan ebetween

### x n

andthe losest enter

:

-means lustering

Split

into lusters

Minimize

### kx n − µ k k 2

Unsupervisedlearning

NP-hard

LearningFromData-Le ture16 10/20

Choosingthe enters

Minimizethedistan ebetween

### x n

andthe losest enter

:

-means lustering

Split

into lusters

Minimize

### kx n − µ k k 2

Unsupervisedlearning

NP-hard

LearningFromData-Le ture16 10/20

Choosingthe enters

Minimizethedistan ebetween

### x n

andthe losest enter

:

-means lustering

Split

into lusters

Minimize

### kx n − µ k k 2

Unsupervisedlearning

NP-hard

LearningFromData-Le ture16 10/20

Choosingthe enters

Minimizethedistan ebetween

### x n

andthe losest enter

:

-means lustering

Split

into lusters

Minimize

### kx n − µ k k 2

Unsupervisedlearning

NP-hard

LearningFromData-Le ture16 10/20

Choosingthe enters

Minimizethedistan ebetween

### x n

andthe losest enter

:

-means lustering

Split

into lusters

Minimize

### kx n − µ k k 2

Unsupervisedlearning

NP-hard

LearningFromData-Le ture16 10/20

Choosingthe enters

Minimizethedistan ebetween

### x n

andthe losest enter

:

-means lustering

Split

into lusters

Minimize

### kx n − µ k k 2

Unsupervisedlearning

NP-hard

LearningFromData-Le ture16 10/20

Choosingthe enters

Minimizethedistan ebetween

### x n

andthe losest enter

:

-means lustering

Split

into lusters

Minimize

### kx n − µ k k 2

Unsupervisedlearning

NP-hard

LearningFromData-Le ture16 10/20

Choosingthe enters

Minimizethedistan ebetween

### x n

andthe losest enter

:

-means lustering

Split

into lusters

Minimize

### kx n − µ k k 2

Unsupervisedlearning

NP-hard

LearningFromData-Le ture16 10/20

(12)

### RBF Networks Prototype Extraction

Aniterativealgorithm

Lloyd'salgorithm:Iterativelyminimize

w.r.t.

all

Convergen e

### −→

lo al minimum

LearningFromData-Le ture16 11/20

Aniterativealgorithm

Lloyd'salgorithm:Iterativelyminimize

w.r.t.

all

Convergen e

### −→

lo al minimum

LearningFromData-Le ture16 11/20

Aniterativealgorithm

Lloyd'salgorithm:Iterativelyminimize

w.r.t.

all

Convergen e

### −→

lo al minimum

LearningFromData-Le ture16 11/20

Aniterativealgorithm

Lloyd'salgorithm:Iterativelyminimize

w.r.t.

all

Convergen e

### −→

lo al minimum

LearningFromData-Le ture16 11/20

Aniterativealgorithm

Lloyd'salgorithm:Iterativelyminimize

w.r.t.

all

Convergen e

### −→

lo al minimum

LearningFromData-Le ture16 11/20

Aniterativealgorithm

Lloyd'salgorithm:Iterativelyminimize

w.r.t.

all

Convergen e

### −→

lo al minimum

LearningFromData-Le ture16 11/20

Aniterativealgorithm

Lloyd'salgorithm:Iterativelyminimize

w.r.t.

all

Convergen e

### −→

lo al minimum

LearningFromData-Le ture16 11/20

Aniterativealgorithm

Lloyd'salgorithm:Iterativelyminimize

w.r.t.

all

Convergen e

### −→

lo al minimum

LearningFromData-Le ture16 11/20

Aniterativealgorithm

Lloyd'salgorithm:Iterativelyminimize

w.r.t.

all

Convergen e

### −→

lo al minimum

LearningFromData-Le ture16 11/20

Aniterativealgorithm

Lloyd'salgorithm:Iterativelyminimize

w.r.t.

all

Convergen e

### −→

lo al minimum

LearningFromData-Le ture16 11/20

Aniterativealgorithm

Lloyd'salgorithm:Iterativelyminimize

w.r.t.

all

Convergen e

### −→

lo al minimum

LearningFromData-Le ture16 11/20

(13)

### RBF Networks Prototype Extraction

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

Lloyd'salgorithm ina tion

### Hi

1.Getthedatapoints

2.Onlytheinputs!

3.Initializethe enters

4.Iterate

5.Theseareyour

### µ k

's

LearningFromData-Le ture16 12/20

(14)

## Fun Time

(15)

### RBF Networks RBF Network

Choosingtheweights

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

T

isinvertible,

T

T

### y

pseudo-inverse

LearningFromData-Le ture16 14/20

Choosingtheweights

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

T

isinvertible,

T

T

### y

pseudo-inverse

LearningFromData-Le ture16 14/20

Choosingtheweights

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

T

isinvertible,

T

T

### y

pseudo-inverse

LearningFromData-Le ture16 14/20

Choosingtheweights

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

T

isinvertible,

T

T

### y

pseudo-inverse

LearningFromData-Le ture16 14/20

Choosingtheweights

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

T

isinvertible,

T

T

### y

pseudo-inverse

LearningFromData-Le ture16 14/20

Choosingtheweights

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

T

isinvertible,

T

T

### y

pseudo-inverse

LearningFromData-Le ture16 14/20

Choosingtheweights

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

T

isinvertible,

T

T

### y

pseudo-inverse

LearningFromData-Le ture16 14/20

Choosingtheweights

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

T

isinvertible,

T

T

### y

pseudo-inverse

LearningFromData-Le ture16 14/20

Choosingtheweights

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

T

isinvertible,

T

T

### y

pseudo-inverse

LearningFromData-Le ture16 14/20

Choosingtheweights

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

T

isinvertible,

T

T

### y

pseudo-inverse

LearningFromData-Le ture16 14/20

Choosingtheweights

equationsin

unknowns

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

If

T

isinvertible,

T

T

### y

pseudo-inverse

LearningFromData-Le ture16 14/20

Updating...

## References

Related subjects :