Inﬁnite Ensemble Learning with Support Vector Machines

(1)

Infinite Ensemble Learning with Support Vector Machines

Hsuan-Tien Lin

in collaboration with Ling Li Learning Systems Group, Caltech

Second Symposium on Vision and Learning, 2005/09/21

(2)

Outline

1 Setup of our Learning Problem

2 Motivation of Infinite Ensemble Learning

3 Connecting SVM and Ensemble Learning

4 SVM-Based Framework of Infinite Ensemble Learning

5 Examples of the Framework

6 Experimental Comparison

7 Conclusion and Discussion

(3)

Setup of our Learning Problem

binary classification problem:

does this image represent an apple?

features of the image: a vector x ∈ X ⊆ R^D.

e.g.:(x)₁can describe the shape,(x)₂can describe the color, etc.

difference to the features in vision: a vector of properties, not a

“set of interest points.”

label (whether the image is an apple): y ∈ {+1, −1}.

learning problem: give many images and their labels (training examples){(x_i,y_i)}^N_i=1, find a classifier g(x) : X → {+1, −1}that predicts unseen images well.

hypotheses (classifiers): functions fromX → {+1, −1}.

(4)

Motivation of Infinite Ensemble Learning

g(x) : X → {+1, −1}

ensemble learning: popular paradigm.

ensemble: weighted vote of a committee of hypotheses.

g(x) =sign(P

w_th_t(x)) ,w_t ≥0.

traditional ensemble learning: infinite size committee with finite number of nonzero weights.

is finiteness restriction and/or regularization?

how to handle infinite number of nonzero weights?

SVM (large-margin hyperplane): also popular.

hyperplane: a weighted combination of features.

SVM: infinite dimensional hyperplane through kernels.

g(x) =sign(P

w_dφ_d(x) +b) .

can we use SVM for infinite ensemble learning?

(5)

Connecting SVM and Ensemble Learning

Illustration of SVM

{(x_i,y_i)}^N_i=1

φd

implicitly computed

φ₁(x)

φ₂(x)

· · ·

φ∞(x)

wd

via duality

w₁

w₂

· · ·

w_∞ (λ_i)^N_i=1

g(x) =sign(P∞

d=1w_dφ_d(x) +b) SVM implicit

computation with K(x,x⁰) = P∞

d=1φd(x)φd(x⁰).

optimal solution (w,b)represented by the dual

variablesλ_i.

(6)

Property of SVM

g(x) =sign(P∞

d=1w_dφ_d(x) +b) =sign PN

i=1λ_iy_iK(x_i,x) +b

optimal hyperplane: represented through duality.

key for handling infinity: kernel tricksK(x,x⁰) =P∞

d=1φd(x)φd(x⁰).

quadratic programming of a margin-related criteria.

goal: (infinite dimensional) large-margin hyperplane.

min

w,b

1

2kwk²₂+C

N

X

i=1

ξi, s.t. y_i

∞

X

d=1

w_dφd(x_i) +b

!

≥1− ξ_i, ξi ≥0.

regularization: controlled with the trade-off parameter C.

(7)

Illustration of AdaBoost

{(x_i,y_i)}^N_i=1

ht∈ H iteratively selected

h₁(x)

h₂(x)

· · ·

h_T(x)

wt≥0 iteratively assigned

w₁

w₂

· · ·

w_T u₁(i)

u₂(i)

· · · g(x) =sign

PT

t=1w_th_t(x) AdaBoost

most successful ensemble learning algorithm.

boosts up the performance of each individual h_t. emphasizes difficult examples by ut and finds (h_t,w_t)iteratively.

(8)

Property of AdaBoost

g(x) =sign PT

t=1wtht(x)

iterative coordinate descent of a margin-related criteria.

min

N

X

i=1

exp(−ρ_i) , s.t.ρ_i =y_i

∞

X

t=1

w_th_t(x_i)

!

,w_t ≥0.

goal: asymptotically, large-margin ensemble.

minw,h kwk₁, s.t. y_i

∞

X

t=1

wtht(x_i)

!

≥1,wt ≥0.

optimal ensemble: approximated by finite one.

key for good approximation: sparsity

– some optimal ensemble has many zero weights.

regularization: finite approximation.

(9)

Connection between SVM and AdaBoost

φ_d(x) ⇔h_t(x)

SVM AdaBoost

G(x) =P

kw_kφk(x) +b G(x) =P

kw_kh_k(x) w_k ≥0 hard-goal

minkwk_p, s.t. y_iG(x_i) ≥1

p=2 p=1

optimization

quadratic programming iterative coordinate descent key for infinity

kernel trick sparsity

regularization

soft-margin trade-off finite approximation

(10)

SVM-Based Framework of Infinite Ensemble Learning

Challenge

designing an infinite ensemble learning algorithm traditional ensemble learning: iterative and cannot directly be generalized.

another approach: embedding infinite number of hypotheses in SVM kernel, i.e.,K(x,x⁰) =P∞

t=1h_t(x)h_t(x⁰).

then, SVM classifier: g(x) =sign(P∞

t=1w_th_t(x) +b).

does the kernel exist?

how to ensure wt ≥0?

our main contribution: a framework that conquers the challenge.

(11)

Embedding Hypotheses into the Kernel

Definition

The kernel that embodiesH = {h_α: α ∈ C}is defined as K_H,r(x,x⁰) =

Z

C

φ_x(α)φ_x⁰(α)dα,

where C is a measure space, φx(α) = r(α)h_α(x), and r: C → R⁺ is chosen such that the integral always exists.

integral instead of sum: works even for uncountableH.

K_H,r(x,x⁰): an inner product forφx andφ_x⁰ inF = L₂(C).

the classifier: g(x) =sign R

Cw(α)r(α)h_α(x)dα +b .

(12)

Negation Completeness and Constant Hypotheses

g(x) =sign

Z

C

w(α)r(α)h_α(x)dα +b

not an ensemble classifier yet.

w(α) ≥0?

hard to handle: possibly uncountable constraints.

simple with negation completeness assumption onH.

negation completeness: h∈ Hif and only if(−h) ∈ H.

for any w , exists nonnegativew that produces same g.˜ What is b?

equivalently, the weight on a constant hypothesis.

another assumption:Hcontains a constant hypothesis.

both assumptions: mild in practice.

g(x)is equivalent to an ensemble classifier.

(13)

Framework of Infinite Ensemble Learning

Algorithm

1 Consider a hypothesis setH(negation complete and contains a constant hypothesis).

2 Construct a kernelKH,r with proper r(·).

3 Properly choose other SVM parameters.

4 Train SVM withK_H,r and{(x_i,y_i)}^N_i=1to obtainλ_i and b.

5 Output g(x) =sign PN

i=1y_iλiK_H(x_i,x) +b . easy: SVM routines.

hard: kernel construction.

shall inherit the profound properties of SVM.

(14)

Examples of the Framework

Decision Stump

decision stump: s_q,d,α(x) =q·sign((x)_d − α).

simplicity: popular for ensemble learning (e.g., Viola and Jones)

(x)₂≥ α?

Y ^@_@

@ R

N

+1

−1

(a) Decision Process

- 6

s_+1,2,α(x) = +1

(x)2= α (x)2

(x)1

(b) Decision Boundary Figure:Illustration of the decision stump s+1,2,α(x)

(15)

Stump Kernel

consider the set of decision stumps S =

s_q,d,α_d:q ∈ {+1, −1} ,d ∈ {1, . . . ,D} , α_d ∈ [L_d,R_d] . whenX ⊆ [L₁,R₁] × [L₂,R₂] × · · · × [L_D,R_D],S is negation complete, and contains a constant hypothesis.

Definition

The stump kernelK_S is defined forS with r(q,d, α_d) = ¹₂.

K_S(x,x⁰) = ∆S−

D

X

d=1

(x)_d− (x⁰)_d

= ∆S− kx−x⁰k₁,

where∆S = ¹₂PD

d=1(R_d−L_d)is a constant.

(16)

Property of Stump Kernel

simple to compute: the constant∆S can even be dropped K˜(x,x⁰) = −kx−x⁰k₁.

infinite power: under mild assumptions, SVM with C= ∞can perfectly classify training examples with stump kernel.

the popular Gaussian kernel exp(−γkx−x⁰k²₂)also.

fast parameter selection: scaling the stump kernel is equivalent to scaling soft-margin parameter C.

Gaussian kernel depends on a good(γ,C)pair.

stump kernel only needs good C: roughly ten times faster.

feature space explanation for`₁-norm similarity.

well suited in some specific applications:

cancer prediction with gene expressions.

(17)

Perceptron

perceptron: p_θ,α(x) =sign θ^Tx− α .

not easy for ensemble learning: hard to design good algorithm.

θ^Tx ≥ α?

Y ^@_@

@ R

N

+1

−1

(a) Decision Process

- 6

p_θ,α(x) = +1 θ^Tx = α (x)2

(x)1

@ I

−θ

(b) Decision Boundary Figure:Illustration of the perceptron pθ,α(x)

(18)

Perceptron Kernel

consider the set of perceptrons P =

p_θ,α: θ ∈ R^D, kθk2=1, α ∈ [−R,R]

.

whenX is within a ball of radius R centered at the origin,P is negation complete, and contains a constant hypothesis.

Definition

The perceptron kernel isK_P with r(θ, α) =rP, K_P(x,x⁰) = ∆P− kx−x⁰k₂, where r_P and∆P are constants.

(19)

Property of Perceptron Kernel

similar properties to the stump kernel.

also simple to compute.

infinite power: equivalent to a D-∞-1 neural network.

fast parameter selection: also shown in (Fleuret and Sahbi, ICCV 2003 workshop, called triangular kernel) without feature space explanation.

(20)

Histogram Intersection Kernel

introduced for scene recognition (Odone et al., IEEE TIP, 2005).

assume(x)d: counts in the histogram (how many pixels are red?) – an integer between[0,size of image].

histogram intersection kernel:

K(x,x⁰) =PD

d=1min((x)_d, (x⁰)_d).

generalized with difficult math when(x)_d is not an integer (Boughorbel et al., ICIP, 2005), similar tasks.

letˆs(x) = (s(x) +1)/2: HIK can be constructed easily from the framework.

furthermore, HIK equivalent to stump kernel.

insights on why HI (stump) kernel works well for the task?

(21)

Other Kernels

Laplacian kernel:K(x,x⁰) =exp(−γkx−x⁰k₁).

provably embodies infinite number of decision trees.

generalized Laplacian: K(x,x⁰) =exp −γP |(x)^a_d− (x⁰)^a_d| . can be similarly constructed with a slightly different r function.

standard kernel for histogram-based image classification with SVM (Chappelle et al., IEEE TNN, 1999).

insights on why it should work well?

exponential kernel: K(x,x⁰) =exp(−γkx−x⁰k₂).

provably embodies infinite number of decision trees of perceptrons.

(22)

Experimental Comparison

Comparison between SVM and AdaBoost

tw twn th thn ri rin aus bre ger hea ion pim son vot 0

5 10 15 20 25 30 35

error (%)

SVM−Stump AdaBoost−Stump(100) AdaBoost−Stump(1000)

5 10 15 20 25 30 35

error (%)

SVM−Perc AdaBoost−Perc(100) AdaBoost−Perc(1000)

Results

fair comparison between AdaBoost and SVM.

SVM is usually best – benefits to go to infinity.

sparsity

(finiteness) is a restriction.

(23)

Experimental Comparison

Comparison of SVM Kernels

5 10 15 20 25 30 35

error (%)

SVM−Stump SVM−Perc SVM−Gauss

Results

SVM-Perc very similar to SVM-Gauss.

SVM-Stump comparable to, but sometimes a bit worse than others.

(24)

Conclusion and Discussion

constructed: general framework for infinite ensemble learning.

infinite ensemble learning could be better – existing AdaBoost-Stump applications may switch.

derived new and meaningful kernels.

stump kernel: succeeded in specific applications.

perceptron kernel: similar to Gaussian, faster in parameter selection.

gave novel interpretation to existing kernels.

histogram intersection kernel: equivalent to stump kernel.

Laplacian kernel: ensemble of decision trees.

possible thoughts for vision

would fast parameter selection be important for some problems?

any vision applications in which those kernel models are reasonable?

do the novel interpretations give any insights?

any domain knowledge that can be brought into kernel construction?