Machine Learning Techniques (ᘤᢈ)

(1)

Machine Learning Techniques

( 機器學習技巧)

Lecture 7: Blending and Bagging

Hsuan-Tien Lin (林軒田) htlin@csie.ntu.edu.tw

Department of Computer Science

& Information Engineering

National Taiwan University

( 國立台灣大學資訊工程系)

(2)

Blending and Bagging

Agenda

Lecture 7: Blending and Bagging Motivation of Aggregation Uniform Blending

Linear and Any Blending

Bagging

(3)

Blending and Bagging Motivation of Aggregation

An Aggregation Story

Your T friends g

₁

, · · · ,g

_T

predicts whether stock will go up as g

t

(x).

You can . . .

• select

the most trust-worthy friend from their

usual performance

—validation!

• mix

the predictions from all your friends

uniformly

—let them

vote!

• mix

non-uniformly

—let them vote, but

give some more ballots

• combine

the predictions

conditionally

—if

[condition t true]

give some ballots to friend t

•

...

aggregation

models:

mix

or

combine

hypotheses (for better performance)

(4)

Aggregation with Math Notations

Your T friends g

₁

, · · · ,g

_T

predicts whether stock will go up as g

t

(x).

• select

the most trust-worthy friend from their

usual performance

G(x) = g

t

∗(x) with t

∗

=argmin

t∈{1,2,··· ,T } E _val (g t )

• mix

uniformly

G(x) = sign

P

T

t=1 1

· g

t

(x)

• mix

non-uniformly

G(x) = sign

P

T

t=1 α t

· g

t

(x)

with

α t ≥ 0

• include select: α

_t

= J E

val

(g

t

) smallest K

• include uniformly: α

_t

= 1

• combine

the predictions

conditionally

G(x) = sign

P

T

t=1 q _t (x)

· g

_t

(x)

with

q _t (x) ≥ 0

• include non-uniformly: q

t

(x) = α

t

aggregation models: a

rich family

(5)

Recall: Selection by Validation

G(x) = g

_t

_∗(x) with t

_∗

= argmin

t∈{1,2,··· ,T }

E _val (g t )

• simple

and popular

•

can also use E

_in

instead of

E _val

(with

complexity price on d

VC)

•

need

one strong

g

_t

to guarantee small

E _val

(and small E

_out

)

selection:

rely on one strong hypothesis

aggregation:

can we do better with many (possibly weaker) hypotheses?

(6)

Why Might Aggregation Work?

•

mix

different weak hypotheses

uniformly

—G(x) ‘strong’

•

aggregation

=⇒

feature transform (?)

•

mix

different random-PLA hypotheses

uniformly

—G(x) ‘moderate’

•

aggregation

=⇒

regularization (?)

proper aggregation =⇒

better performance

(7)

Fun Time

(8)

Blending and Bagging Uniform Blending

Uniform Blending (Voting) for Classification

uniform blending: known g t

, each with

1

ballot

G(x) = sign

T

X

t=1

1

·

g t

(x)

!

•

same

g _t

(autocracy):

as good as one single

g _t

•

very different

g _t

(diversity+

democracy):

majority can

correct

minority

•

similar results with uniform voting for multiclass

G(x) = argmax

1≤k ≤K T

X

t=1

Jg

^t

(x) = kK

how about

regression?

(9)

Uniform Blending for Regression

G(x) = 1 T

T

X

t=1

g _t

(x)

•

same

g _t

(autocracy):

as good as one single

g t

•

very different

g _t

(diversity+

democracy):

=⇒

some

g t

(x) > f (x), some

g t

(x) < f (x)

=⇒average

could be

more accurate than individual

diverse hypotheses: even simple uniform

blending

can be better than one

(10)

Theoretical Analysis of Uniform Blending

G(x)

=

1 T

T

X

t=1

g _t (x)

avg (g

_t

(x) − f (x))

²

= avg g

²_t

− 2g

_t

f + f

²

= avg g

²_t

− 2Gf + f

²

= avg g

²_t

− G

²

+ (G − f )

²

= avg g

²_t

− 2G

²

+ G

²

+ (G − f )

²

= avg g

²_t

− 2g

_t

G + G

²

+ (G − f )

²

= avg (g

_t

− G)

²

+ (G − f )

²

avg

(E

out

(g

t

)) =

avg

E(g

_t

−

G) ²

+ E

out

(G)

(11)

Some Special g _t

consider a

virtual

iterative process that for t = 1, 2, . . . , T

1

request size-N data D

_t

from P

^N

(i.i.d.)

2

obtain

g t

by A(D

t

)

g ¯

= lim

T →∞ G

= lim

T →∞

1 T

T

X

t=1

g t

=

E

D A(D)

avg

(E

_out

(g

_t

)) =

avg

E(g

_t

−

¯ g) ²

+E

_out

(

g) ¯ expected

performance of A =

expected deviation

to

consensus

+performance of

consensus

•

performance of

consensus: called bias

• expected deviation

to

consensus: called variance

uniform blending:

reduces

variance

for stabler performance

(12)

Fun Time

(13)

Blending and Bagging Linear and Any Blending

Linear Blending

linear blending: known g t

, each to be given

α t

ballot

G(x) = sign

T

X

t=1

α t

· g

t

(x)

!

with

α t ≥ 0

computing ‘good’ α

t

: min

α

t

≥0

E

_in

(α)

linear blending for regression

α

mint

≥0

1 N

N

X

n=1



y

n

−

T

X

t =1

α t g t

(x

n

)





2 LinReg + transformation

min

w

i

1 N

N

X

n=1



y

n

−

˜ d

X

i=1

w _i φ _i

(x

n

)





2

linear blending = LinModel +

hypotheses as transform

+

constraints

(14)

Constraint on α _t

hypotheses as transform

+

constraints:

min

α

t

≥0

1 N

N

X

n=1

err y

_n

,

T

X

t=1

α t g _t

(x

_n

)

!

linear blending for binary classification

if

α t

<0 =⇒

α t g _t

(x) =

|α _t |

(−g

t

(x))

•

negative

α t

for

g t

≡ positive

|α _t |

for

−g t

• if you have a stock up/down classifier with 99% error, tell me!

:-)

in practice, often

hypotheses as transform

(( (( (( (

hhh

+

constraints hhh h

(15)

Linear Blending versus Selection

in practice, often

g ₁

∈ H

₁

,

g ₂

∈ H

₂

, . . . ,

g _T

∈ H

_T

by

minimum E _in

•

recall:

selection by minimum E _in

—bestof

best, paying d

_VC

_T S

t=1

H _t

•

recall: linear blending includes

selection

as special case

—by setting

α t

=J

E _val (g t )

smallestK

•

complexity price of linear blending

with E _in

(aggregationof

best):

d

VC

_T S

t=1

H _t

like

selection, blending practically done with

(E

_val

instead of

E _in

) + (g

_t

from

E _train

)

(16)

Any Blending

Linear Blending

Given

g ₁

,

g ₂

, . . .,

g _T

1

transform (x

_n

,y

_n

)in

D _val

to (z

_n

=

Φ(x _n

),y

_n

), where

Φ(x) = (g ₁

(x), . . . ,

g _T

(x))

2

compute

α

= Lin

{(z

n

,y

_n

)}

return GLINB(x) = LinH(α

^T Φ(x))

Any Blending (Stacking)

Given

g ₁

,

g ₂

, . . .,

g _T

1

transform (x

_n

,y

_n

)in

D _val

to (z

_n

=

Φ(x _n

),y

n

), where

Φ(x) = (g ₁

(x), . . . ,

g _T

(x))

2

compute

g ˜

=

Any

{(z

n

,y

n

)} return GANYB(x) =

g(Φ(x)) ˜

if

AnyModel = quadratic polynomial:

G

ANYB

(x) =

T

X

t=1

α

_t

+

T

X

τ =1

α

_τ,t

g

τ

(x)

!

| {z }

q(x)

·g

_t

(x) —conditional aggregation

danger:

overfitting with

any

blending!

(17)

Blending in Practice

KDDCup 2012 Track 1: World Champion Solution by NTU

• validation set blending: a special any blending

model E

_test

(squared):

519.45

=⇒

456.24

—helped

secure the lead

in

last two weeks

• test set blending: linear blending

using ˜E

_test

E

_test

(squared):

456.24

=⇒

442.06

—helped

turn the tables

in

last hour

blending ‘useful’ in practice,

despite the

computational burden

(18)

Fun Time

(19)

Blending and Bagging Bagging

What We Have Done

blending: aggregate

after getting g _t

; learning: aggregate

as well as getting g _t aggregation type blending learning

uniform voting/averaging ?

non-uniform linear ?

conditional stacking ?

learning

g

_t

for uniform aggregation:

diversity

important

• diversity

by different models:

g ₁

∈ H

₁

,

g ₂

∈ H

₂

, . . . ,

g _T

∈ H

_T

• diversity

by different parameters: GD with η = 0.001, 0.01, . . ., 10

• diversity

by algorithmic randomness:

random PLA with different random seeds

• diversity

by data randomness:

within-cross-validation hypotheses g

_v ⁻

by data randomness

without

g

⁻

(20)

Revisit of Bias-Variance

expected

performance of A =

expected deviation

to

consensus

+performance of

consensus consensus ¯ g

=

expected g _t

from

D _t

∼ P

^N

• consensus

more stable than direct A(D),

but comes from many more

D _t

than the D on hand

•

want: approximate

g ¯

by

• finite (large) T

• approximate g

t

= A(D

_t

) from D

_t

∼ P

^N

using only D

bootstrapping: a statistical tool that

re-samples

from D to ‘simulate’

D _t

(21)

Bootstrap Aggregation

bootstrapping

bootstrap sample

D ˜ _t

: re-sample N examples from D

with replacement

virtual aggregation

consider a

virtual

1

request size-N data D

t

from P

^N

(i.i.d.)

2

obtain

g _t

by A(D

_t

)

G

=Uniform(g

t

)

bootstrap aggregation

consider a

physical

1

request size-N data

D ˜ _t

from

bootstrapping

2

obtain

g _t

by A(

D ˜ _t

)

G

=Uniform(g

t

)

bootstrap aggregation (BAGging):

a simple

meta algorithm

on top of

base algorithm

A

(22)

Bagging Pocket in Action

TPOCKET =1000; TBAG=25

•

very

diverse

g

_t

from bagging

•

proper

non-linear

boundary after aggregating binary classifiers

bagging works reasonably well

if base

algorithm sensitive to data randomness

(23)

Fun Time

(24)

Machine Learning Techniques (ᘤᢈ)