Introduction to Adaptive Boosting

### Introduction to Adaptive Boosting

Hsuan-Tien Lin

National Taiwan University

Machine Learning, Fall 2008

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Apple Recognition Problem

Is this a picture of an apple?

We want to teach a class of 6 year olds.

Gather photos from NY Apple Asso. and Google Image.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Our Fruit Class Begins

Teacher: How would you describe an apple? Michael?

Michael: I think apples are circular.

(Class): Apples are circular.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Our Fruit Class Begins

Teacher: How would you describe an apple? Michael?

Michael: I think apples are circular.

(Class): Apples are circular.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Our Fruit Class Begins

Teacher: How would you describe an apple? Michael?

Michael: I think apples are circular.

(Class): Apples are circular.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Our Fruit Class Continues

Teacher: Being circular is a good feature for the apples. However, if you only say circular, you could make several mistakes.

What else can we say for an apple? Tina?

Tina: It looks like apples are red.

(Class): Apples are somewhat circular and somewhat red.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Our Fruit Class Continues

Teacher: Being circular is a good feature for the apples. However, if you only say circular, you could make several mistakes.

What else can we say for an apple? Tina?

Tina: It looks like apples are red.

(Class): Apples are somewhat circular and somewhat red.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Our Fruit Class Continues

Teacher: Being circular is a good feature for the apples. However, if you only say circular, you could make several mistakes.

What else can we say for an apple? Tina?

Tina: It looks like apples are red.

(Class): Apples are somewhat circular and somewhat red.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Our Fruit Class Continues

Teacher: Yes. Many apples are red. However, you could still make mistakes based on circular and red. Do you have any other suggestions, Joey?

Joey: Apples could also be green.

(Class): Apples are somewhat circular and somewhat red and possibly green.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Our Fruit Class Continues

Teacher: Yes. Many apples are red. However, you could still make mistakes based on circular and red. Do you have any other suggestions, Joey?

Joey: Apples could also be green.

(Class): Apples are somewhat circular and somewhat red and possibly green.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Our Fruit Class Continues

Teacher: Yes. Many apples are red. However, you could still make mistakes based on circular and red. Do you have any other suggestions, Joey?

Joey: Apples could also be green.

(Class): Apples are somewhat circular and somewhat red and possibly green.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Our Fruit Class Continues

Teacher: Yes. It seems that apples might be circular, red, green. But you may confuse them with tomatoes or peaches, right?

Any more suggestions, Jessica?

Jessica: Apples have stems at the top.

(Class): Apples are somewhat circular, somewhat red, possibly green, and may have stems at the top.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Our Fruit Class Continues

Teacher: Yes. It seems that apples might be circular, red, green. But you may confuse them with tomatoes or peaches, right?

Any more suggestions, Jessica?

Jessica: Apples have stems at the top.

(Class): Apples are somewhat circular, somewhat red, possibly green, and may have stems at the top.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Our Fruit Class Continues

Teacher: Yes. It seems that apples might be circular, red, green. But you may confuse them with tomatoes or peaches, right?

Any more suggestions, Jessica?

Jessica: Apples have stems at the top.

(Class): Apples are somewhat circular, somewhat red, possibly green, and may have stems at the top.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Put Intuition to Practice

Intuition

Combine simple rules to approximate complex function.

Emphasize incorrect data to focus on valuable information.

AdaBoost Algorithm (Freund and Schapire 1997)
Input: training examples Z = {(xn,yn)}^{N}_{n=1}.
For t = 1, 2, · · · , T ,

Learn a simple rule ht from emphasized training examples.

Get the confidence αt of such rule

Emphasize the training examples that do not agree with ht. Output: combined function H(x ) = sign

PT

t=1α_{t}h_{t}(x )

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Put Intuition to Practice

Intuition

Combine simple rules to approximate complex function.

Emphasize incorrect data to focus on valuable information.

AdaBoost Algorithm (Freund and Schapire 1997)
Input: training examples Z = {(xn,yn)}^{N}_{n=1}.
For t = 1, 2, · · · , T ,

Learn a simple rule ht from emphasized training examples.

Get the confidence αt of such rule

Emphasize the training examples that do not agree with ht. Output: combined function H(x ) = sign

PT

t=1α_{t}h_{t}(x )

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Put Intuition to Practice

Intuition

Combine simple rules to approximate complex function.

Emphasize incorrect data to focus on valuable information.

AdaBoost Algorithm (Freund and Schapire 1997)
Input: training examples Z = {(xn,yn)}^{N}_{n=1}.
For t = 1, 2, · · · , T ,

Learn a simple rule ht from emphasized training examples.

Get the confidence αt of such rule

Emphasize the training examples that do not agree with ht. Output: combined function H(x ) = sign

PT

t=1α_{t}h_{t}(x )

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Put Intuition to Practice

Intuition

Combine simple rules to approximate complex function.

Emphasize incorrect data to focus on valuable information.

^{N}_{n=1}.
For t = 1, 2, · · · , T ,

Learn a simple rule ht from emphasized training examples.

Get the confidence αt of such rule

Emphasize the training examples that do not agree with ht. Output: combined function H(x ) = sign

PT

t=1α_{t}h_{t}(x )

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Put Intuition to Practice

Intuition

Combine simple rules to approximate complex function.

Emphasize incorrect data to focus on valuable information.

^{N}_{n=1}.
For t = 1, 2, · · · , T ,

Learn a simple rule ht from emphasized training examples.

Get the confidence αt of such rule

Emphasize the training examples that do not agree with ht. Output: combined function H(x ) = sign

PT

t=1α_{t}h_{t}(x )

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Put Intuition to Practice

Intuition

Combine simple rules to approximate complex function.

Emphasize incorrect data to focus on valuable information.

^{N}_{n=1}.
For t = 1, 2, · · · , T ,

Learn a simple rule ht from emphasized training examples.

Get the confidence αt of such rule

Emphasize the training examples that do not agree with ht. Output: combined function H(x ) = sign

PT

t=1α_{t}h_{t}(x )

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Some More Details

AdaBoost Algorithm

Input: training examples Z = {(x_{n},y_{n})}^{N}_{n=1}.
For t = 1, 2, · · · , T ,

Learn a simple rule h_{t} from emphasized training examples.

How? Choose a ht ∈ H with minimum emphasized error.

Get the confidence α_{t} of such rule

How? An ht with lower error should get higher αt.

Emphasize the training examples that do not agree with ht. How? Maintain an emphasis value unper example.

Output: combined function H(x ) = sign PT

t=1α_{t}h_{t}(x )
Let’s see some demos.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Some More Details

AdaBoost Algorithm

Input: training examples Z = {(x_{n},y_{n})}^{N}_{n=1}.
For t = 1, 2, · · · , T ,

Learn a simple rule h_{t} from emphasized training examples.

How? Choose a ht ∈ H with minimum emphasized error.

Get the confidence α_{t} of such rule

How? An ht with lower error should get higher αt.

Emphasize the training examples that do not agree with ht. How? Maintain an emphasis value unper example.

Output: combined function H(x ) = sign PT

t=1α_{t}h_{t}(x )
Let’s see some demos.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Some More Details

AdaBoost Algorithm

Input: training examples Z = {(x_{n},y_{n})}^{N}_{n=1}.
For t = 1, 2, · · · , T ,

Learn a simple rule h_{t} from emphasized training examples.

How? Choose a ht ∈ H with minimum emphasized error.

Get the confidence α_{t} of such rule

How? An ht with lower error should get higher αt.

Emphasize the training examples that do not agree with ht. How? Maintain an emphasis value unper example.

Output: combined function H(x ) = sign PT

t=1α_{t}h_{t}(x )
Let’s see some demos.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Some More Details

AdaBoost Algorithm

Input: training examples Z = {(x_{n},y_{n})}^{N}_{n=1}.
For t = 1, 2, · · · , T ,

Learn a simple rule h_{t} from emphasized training examples.

How? Choose a ht ∈ H with minimum emphasized error.

Get the confidence α_{t} of such rule

How? An ht with lower error should get higher αt.

Output: combined function H(x ) = sign PT

t=1α_{t}h_{t}(x )
Let’s see some demos.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Some More Details

AdaBoost Algorithm

Input: training examples Z = {(x_{n},y_{n})}^{N}_{n=1}.
For t = 1, 2, · · · , T ,

Learn a simple rule h_{t} from emphasized training examples.

How? Choose a ht ∈ H with minimum emphasized error.

Get the confidence α_{t} of such rule

How? An ht with lower error should get higher αt.

Output: combined function H(x ) = sign PT

t=1α_{t}h_{t}(x )
Let’s see some demos.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Some More Details

AdaBoost Algorithm

Input: training examples Z = {(x_{n},y_{n})}^{N}_{n=1}.
For t = 1, 2, · · · , T ,

Learn a simple rule h_{t} from emphasized training examples.

How? Choose a ht ∈ H with minimum emphasized error.

Get the confidence α_{t} of such rule

How? An ht with lower error should get higher αt.

Output: combined function H(x ) = sign PT

t=1α_{t}h_{t}(x )
Let’s see some demos.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Some More Details

AdaBoost Algorithm

Input: training examples Z = {(x_{n},y_{n})}^{N}_{n=1}.
For t = 1, 2, · · · , T ,

Learn a simple rule h_{t} from emphasized training examples.

How? Choose a ht ∈ H with minimum emphasized error.

Get the confidence α_{t} of such rule

How? An ht with lower error should get higher αt.

Output: combined function H(x ) = sign PT

t=1α_{t}h_{t}(x )
Let’s see some demos.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### Some More Details

AdaBoost Algorithm

Input: training examples Z = {(x_{n},y_{n})}^{N}_{n=1}.
For t = 1, 2, · · · , T ,

Learn a simple rule h_{t} from emphasized training examples.

How? Choose a ht ∈ H with minimum emphasized error.

Get the confidence α_{t} of such rule

How? An ht with lower error should get higher αt.

Output: combined function H(x ) = sign PT

t=1α_{t}h_{t}(x )
Let’s see some demos.

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### The Final Version

Input: Z = {(xn,yn)}^{N}_{n=1}.**Set u**_{n}= _{N}^{1} **for all n.**

For t = 1, 2, · · · , T ,

Learn a simple rule h_{t} such that h_{t} solves

minh N

X

n=1

u_{n}· I[y_{n}6= h(x_{n})].

Compute the error t =PN

n=1 un

PN

m=1um · I[y_{n}6= h(x_{n})]and the
confidence

α_{t} =1

2ln1 − t

_{t}

Emphasize the training examples that do not agree with h_{t}:
un=un· exp

−α_{t}ynht(xn)
.
Output: combined function H(x ) = sign

PT

t=1αth_{t}(x )

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### The Final Version

Input: Z = {(xn,yn)}^{N}_{n=1}.**Set u**_{n}= _{N}^{1} **for all n.**

For t = 1, 2, · · · , T ,

Learn a simple rule h_{t} such that h_{t} solves

minh N

X

n=1

u_{n}· I[y_{n}6= h(x_{n})].

Compute the error t =PN n=1 un

PN

m=1um · I[y_{n}6= h(x_{n})]and the
confidence

α_{t} =1

2ln1 − t

_{t}

Emphasize the training examples that do not agree with h_{t}:
un=un· exp

−α_{t}ynht(xn)
.
Output: combined function H(x ) = sign

PT

t=1αth_{t}(x )

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### The Final Version

Input: Z = {(xn,yn)}^{N}_{n=1}.**Set u**_{n}= _{N}^{1} **for all n.**

For t = 1, 2, · · · , T ,

Learn a simple rule h_{t} such that h_{t} solves

minh N

X

n=1

u_{n}· I[y_{n}6= h(x_{n})].

Compute the error t =PN n=1 un

PN

m=1um · I[y_{n}6= h(x_{n})]and the
confidence

α_{t} =1

2ln1 − t

_{t}

Emphasize the training examples that do not agree with h_{t}:
un=un· exp

−α_{t}ynht(xn)
.
Output: combined function H(x ) = sign

PT

t=1αth_{t}(x )

Hsuan-Tien Lin AdaBoost

Introduction to Adaptive Boosting Intuition

Adaptive Boosting (AdaBoost)

### The Final Version

Input: Z = {(xn,yn)}^{N}_{n=1}.**Set u**_{n}= _{N}^{1} **for all n.**

For t = 1, 2, · · · , T ,

Learn a simple rule h_{t} such that h_{t} solves

minh N

X

n=1

u_{n}· I[y_{n}6= h(x_{n})].

Compute the error t =PN n=1 un

PN

m=1um · I[y_{n}6= h(x_{n})]and the
confidence

α_{t} =1

2ln1 − t

_{t}

Emphasize the training examples that do not agree with h_{t}:
un=un· exp

−α_{t}ynht(xn)
.
Output: combined function H(x ) = sign

PT

t=1αth_{t}(x )

Hsuan-Tien Lin AdaBoost