XXOutputXX

N

X

m=1

β m RBF(x, x _m

)

!

•

just linear regression on

RBF-transformed data

z n

= [RBF(x

n

,

x ₁

),

RBF(x n

,

x ₂

), . . . ,

RBF(x n

,

x _N

)]

•

optimal

β? β

= (Z

^T Z) ⁻¹ Z ^T y, if Z ^T Z

invertible,

remember? :-)

•

size of

Z?

N (examples) by

N (centers)

—symmetric square matrix

•

theoretical fact: if

x _n all different, Z

with

Gaussian RBF invertible

optimal

β

with

invertible Z: β

=

Z ⁻¹

y

Radial Basis Function Network RBF Network Learning

Interpolation by Full RBF Network

full RBF

Network for squared error regression:

h(x) =

   XX Output X X

N

X

m=1

β m RBF(x, x _m

)

!

•

just linear regression on

RBF-transformed data

z n

= [RBF(x

n

,

x ₁

),

RBF(x n

,

x ₂

), . . . ,

RBF(x n

,

x _N

)]

•

optimal

β? β

= (Z

^T Z) ⁻¹ Z ^T y, if Z ^T Z

invertible,

remember? :-)

•

size of

Z?

N (examples) by

N (centers)

—symmetric square matrix

•

theoretical fact: if

x _n all different, Z

with

Gaussian RBF invertible

optimal

β

with

invertible Z: β

=

Z ⁻¹

y

Radial Basis Function Network RBF Network Learning

Interpolation by Full RBF Network

full RBF

Network for squared error regression:

h(x) =

   XX Output X X

N

X

m=1

β m RBF(x, x _m

)

!

•

just linear regression on

RBF-transformed data

z n

= [RBF(x

n

,

x ₁

),

RBF(x n

,

x ₂

), . . . ,

RBF(x n

,

x _N

)]

•

optimal

β? β

= (Z

^T Z) ⁻¹ Z ^T y, if Z ^T Z

invertible,

remember? :-)

•

size of

Z?

N (examples) by

N (centers)

—symmetric square matrix

•

theoretical fact: if

x _n all different, Z

with

Gaussian RBF invertible

optimal

β

with

invertible Z: β

=

Z ⁻¹

y

Radial Basis Function Network RBF Network Learning

Interpolation by Full RBF Network

full RBF

Network for squared error regression:

h(x) =

   XX Output X X

N

X

m=1

β m RBF(x, x _m

)

!

•

just linear regression on

RBF-transformed data

z n

= [RBF(x

n

,

x ₁

),

RBF(x n

,

x ₂

), . . . ,

RBF(x n

,

x _N

)]

•

optimal

β? β

= (Z

^T Z) ⁻¹ Z ^T y, if Z ^T Z

invertible,

remember? :-)

•

size of

Z?

N (examples) by

N (centers)

—symmetric square matrix

•

theoretical fact: if

x _n all different, Z

with

Gaussian RBF invertible

optimal

β

with

invertible Z: β

=

Z ⁻¹

y

Radial Basis Function Network RBF Network Learning

Interpolation by Full RBF Network

full RBF

Network for squared error regression:

h(x) =

   XX Output X X

N

X

m=1

β m RBF(x, x _m

)

!

•

just linear regression on

RBF-transformed data

z n

= [RBF(x

n

,

x ₁

),

RBF(x n

,

x ₂

), . . . ,

RBF(x n

,

x _N

)]

•

optimal

β? β

= (Z

^T Z) ⁻¹ Z ^T y, if Z ^T Z

invertible,

remember? :-)

•

size of

Z? N (examples) by N (centers)

—symmetric square matrix

•

theoretical fact: if

x _n all different, Z

with

Gaussian RBF invertible

optimal

β

with

invertible Z: β

=

Z ⁻¹

y

Radial Basis Function Network RBF Network Learning

Interpolation by Full RBF Network

full RBF

Network for squared error regression:

h(x) =

   XX Output X X

N

X

m=1

β m RBF(x, x _m

)

!

•

just linear regression on

RBF-transformed data

z n

= [RBF(x

n

,

x ₁

),

RBF(x n

,

x ₂

), . . . ,

RBF(x n

,

x _N

)]

•

optimal

β? β

= (Z

^T Z) ⁻¹ Z ^T y, if Z ^T Z

invertible,

remember? :-)

•

size of

Z? N (examples) by N (centers)

—symmetric square matrix

•

theoretical fact: if

x _n all different, Z

with

Gaussian RBF invertible

optimal

β

with

invertible Z: β

=

Z ⁻¹

y

Radial Basis Function Network RBF Network Learning

Interpolation by Full RBF Network

full RBF

Network for squared error regression:

h(x) =

   XX Output X X

N

X

m=1

β m RBF(x, x _m

)

!

•

just linear regression on

RBF-transformed data

z n

= [RBF(x

n

,

x ₁

),

RBF(x n

,

x ₂

), . . . ,

RBF(x n

,

x _N

)]

•

optimal

β? β

= (Z

^T Z) ⁻¹ Z ^T y, if Z ^T Z

invertible,

remember? :-)

•

size of

Z? N (examples) by N (centers)

—symmetric square matrix

•

theoretical fact: if

x _n all different, Z

with

Gaussian RBF invertible

Z ⁻¹

Radial Basis Function Network RBF Network Learning

Interpolation by Full RBF Network

full RBF

Network for squared error regression:

h(x) =

   XX Output X X

N

X

m=1

β m RBF(x, x _m

)

!

•

just linear regression on

RBF-transformed data

z n

= [RBF(x

n

,

x ₁

),

RBF(x n

,

x ₂

), . . . ,

RBF(x n

,

x _N

)]

•

optimal

β? β

= (Z

^T Z) ⁻¹ Z ^T y, if Z ^T Z

invertible,

remember? :-)

•

size of

Z? N (examples) by N (centers)

—symmetric square matrix

•

theoretical fact: if

x _n all different, Z

with

Gaussian RBF invertible

optimal

β

with

invertible Z: β

=

Z ⁻¹ y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

=

y

^T

Z

⁻¹

(first column of Z) = y

^T



1

0 . . . 0 

T

=

y

₁

—g_RBF(x

_n

) =

y

_n

, i.e. E

_in

(g_RBF) =

0

,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

=

y

^T

Z

⁻¹

(first column of Z)

= y

^T



1

0 . . . 0 

T

=

y

₁

—g_RBF(x

_n

) =

y

_n

, i.e. E

_in

(g_RBF) =

0

,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z)

= y

^T



1

0 . . . 0 

T

=

y

₁

—g_RBF(x

_n

) =

y

_n

, i.e. E

_in

(g_RBF) =

0

,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1

0 . . . 0 

T

=

y

₁

—g_RBF(x

_n

) =

y

_n

, i.e. E

_in

(g_RBF) =

0

,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1 0 . . . 0 

T

=

y

₁

—g_RBF(x

_n

) =

y

_n

, i.e. E

_in

(g_RBF) =

0

,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1 0 . . . 0 

T

=

y

₁

—g_RBF(x

_n

) =

y

_n

, i.e. E

_in

(g_RBF) =

0

,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1 0 . . . 0 

T

= y

₁

—g_RBF(x

_n

) =

y

_n

, i.e. E

_in

(g_RBF) =

0

,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1 0 . . . 0 

T

= y

₁

—g_RBF(x

_n

) =

y

_n

,

i.e. E

_in

(g_RBF) =

0

,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1 0 . . . 0 

T

= y

₁

—g_RBF(x

_n

) =y

_n

,

i.e. E

_in

(g_RBF) =

0

,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1 0 . . . 0 

T

= y

₁

—g_RBF(x

_n

) =y

_n

, i.e. E

_in

(g_RBF) =

0,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1 0 . . . 0 

T

= y

₁

—g_RBF(x

_n

) =y

_n

, i.e. E

_in

(g_RBF) =0,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =

Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1 0 . . . 0 

T

= y

₁

—g_RBF(x

_n

) =y

_n

, i.e. E

_in

(g_RBF) =0,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =

Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1 0 . . . 0 

T

= y

₁

—g_RBF(x

_n

) =y

_n

, i.e. E

_in

(g_RBF) =0,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =

Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1 0 . . . 0 

T

= y

₁

—g_RBF(x

_n

) =y

_n

, i.e. E

_in

(g_RBF) =0,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =

Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1 0 . . . 0 

T

= y

₁

—g_RBF(x

_n

) =y

_n

, i.e. E

_in

(g_RBF) =0,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =

Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1 0 . . . 0 

T

= y

₁

—g_RBF(x

_n

) =y

_n

, i.e. E

_in

(g_RBF) =0,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1 0 . . . 0 

T

= y

₁

—g_RBF(x

_n

) =y

_n

, i.e. E

_in

(g_RBF) =0,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

Radial Basis Function Network RBF Network Learning

Regularized Full RBF Network

full Gaussian RBF Network for regression:

β

=

Z ⁻¹ y g

RBF

(x

₁

) = β

^T

z

₁

= y

^T

Z

⁻¹

(first column of Z) = y

^T



1 0 . . . 0 

T

= y

₁

—g_RBF(x

_n

) =y

_n

, i.e. E

_in

(g_RBF) =0,

yeah!! :-)

•

called

exact interpolation

for

function approximation

•

but

overfitting for learning? :-(

•

how about

regularization? e.g. ridge

regression for

β

instead

—optimal

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

•

seen

Z? Z

= [Gaussian(x

_n

,

x _m

)] =Gaussian kernel matrix

K

effect of

regularization

in different spaces:

kernel

ridge

regression:

β

= (K+

λI) ⁻¹ y;

regularized

full RBFNet:

β

= (Z

^T Z

+

λI) ⁻¹ Z ^T y

Radial Basis Function Network RBF Network Learning

Fewer Centers as Regularization

recall:

gSVM(x) =

sign X

SV

α m y m exp



−γkx − x m k ² 

+

b

!

—only ‘ N’

SVs

needed in ‘network’

•

next:

M  N

instead of

M = N

•

effect:

regularization

在文檔中 Machine Learning Techniques (ᘤᢈ) (頁 42-68)