Introduction to Adaptive Boosting
Hsuan-Tien Lin
National Taiwan University
Machine Learning, Fall 2009
Hsuan-Tien Lin AdaBoost
Apple Recognition Problem
Is this a picture of an apple?
We want to teach a class of 6 year olds.
Gather photos from NY Apple Asso. and Google Image.
Hsuan-Tien Lin AdaBoost
Our Fruit Class Begins
Teacher: How would you describe an apple? Michael?
Michael: I think apples are circular.
(Class): Apples are circular.
Hsuan-Tien Lin AdaBoost
Our Fruit Class Continues
Teacher: Being circular is a good feature for the apples. However, if you only say circular, you could make several mistakes.
What else can we say for an apple? Tina?
Tina: It looks like apples are red.
(Class): Apples are somewhat circular and somewhat red.
Hsuan-Tien Lin AdaBoost
Our Fruit Class Continues
Teacher: Yes. Many apples are red. However, you could still make mistakes based on circular and red. Do you have any other suggestions, Joey?
Joey: Apples could also be green.
(Class): Apples are somewhat circular and somewhat red and possibly green.
Hsuan-Tien Lin AdaBoost
Our Fruit Class Continues
Teacher: Yes. It seems that apples might be circular, red, green. But you may confuse them with tomatoes or peaches, right?
Any more suggestions, Jessica?
Jessica: Apples have stems at the top.
(Class): Apples are somewhat circular, somewhat red, possibly green, and may have stems at the top.
Hsuan-Tien Lin AdaBoost
Put Intuition to Practice
Intuition
Combine simple rules to approximate complex function.
Emphasize incorrect data to focus on valuable information.
AdaBoost Algorithm (Freund and Schapire 1997) Input: training examples D = {(xn,yn)}Nn=1. For t = 1, 2, · · · , T ,
Learn a simple rule ht from emphasized training examples.
Get the confidence αt of such rule
Emphasize the training examples that do not agree with ht. Output: combined function H(x ) = sign
PT
t=1αtht(x)
Hsuan-Tien Lin AdaBoost
Some More Details
AdaBoost Algorithm
Input: training examples Z = {(xn,yn)}Nn=1. For t = 1, 2, · · · , T ,
Learn a simple rule ht from emphasized training examples.
How? Choose a ht ∈ H with minimum emphasized error.
Get the confidence αt of such rule
How? An ht with lower error should get higher αt.
Emphasize the training examples that do not agree with ht. How? Maintain an emphasis value unper example.
Output: combined function H(x) = sign PT
t=1αtht(x) Let’s see some demos.
Hsuan-Tien Lin AdaBoost
The Final Version
Input: D = {(xn,yn)}Nn=1.Set un= N1 for all n.
For t = 1, 2, · · · , T ,
Learn a simple rule ht such that ht solves
minh N
X
n=1
un· [yn6= h(xn)].
Compute the error t =PN n=1 un
PN
m=1um · I[yn6= h(xn)]and the confidence
αt =1
2ln1 − t
t
Emphasize the training examples that do not agree with ht: un=un· exp
−αtynht(xn) . Output: combined function H(x) = sign
PT
t=1αtht(x)
Hsuan-Tien Lin AdaBoost
