### Training Support Vector Machines:

### Status and Challenges

Chih-Jen Lin

Department of Computer Science National Taiwan University

ICML Workshop on Large Scale Learning Challenge

### Outline

SVM is popular

But its training isn’t easy We check existing techniques Large data sets

We show several approaches, and discuss various considerations

Will try to partially answer why there are controversial comparisons

Introduction to SVM

### Outline

Introduction to SVM

Solving SVM Quadratic Programming Problem Training large-scale data

Linear SVM

Discussion and Conclusions

### Support Vector Classification

Training data (x_{i}, y_{i}), i = 1, . . . , l , x_{i} ∈ R^{n}, y_{i} = ±1
Maximizing the margin

[Boser et al., 1992, Cortes and Vapnik, 1995]

min

w,b,ξ

1

2w^{T}w +C

l

X

i =1

ξ_{i}

subject to y_{i}(w^{T}φ(x_{i})+ b) ≥ 1 −ξ_{i},
ξ_{i} ≥ 0, i = 1, . . . , l .

High dimensional ( maybe infinite ) feature space
φ(x) = (φ_{1}(x), φ_{2}(x), . . .).

Introduction to SVM

### Support Vector Classification (Cont’d)

w: maybe infinite variables

The dual problem (finite # variables) minα

1

2α^{T}Qα − e^{T}α

subject to 0 ≤ α_{i} ≤ C , i = 1, . . . , l
y^{T}α = 0,

where Q_{ij} = y_{i}y_{j}φ(x_{i})^{T}φ(x_{j}) and e = [1, . . . , 1]^{T}
At optimum

w =Pl

i =1α_{i}y_{i}φ(x_{i})

### Outline

Introduction to SVM

Solving SVM Quadratic Programming Problem Training large-scale data

Linear SVM

Discussion and Conclusions

Solving SVM Quadratic Programming Problem

### Large Dense Quadratic Programming

minα

1

2α^{T}Qα − e^{T}α

subject to 0 ≤ α_{i} ≤ C , i = 1, . . . , l
y^{T}α = 0

Q_{ij} 6= 0, Q : an l by l fully dense matrix
50,000 training points: 50,000 variables:

(50, 000^{2} × 8/2) bytes = 10GB RAM to store Q
Traditional methods:

### Decomposition Methods

Working on some variables each time (e.g.,

[Osuna et al., 1997, Joachims, 1998, Platt, 1998]) Similar to coordinate-wise minimization

Working set B , N = {1, . . . , l }\B fixed Sub-problem at the kth iteration:

minαB

1

2α^{T}_{B} (α^{k}_{N})^{T}Q_{BB} QBN

Q_{NB} Q_{NN}

α_{B}
α^{k}_{N}

−

e^{T}_{B} (e^{k}_{N})^{T}α_{B}
α^{k}_{N}

subject to 0 ≤ α_{t} ≤ C , t ∈ B, y^{T}_{B}α_{B} = −y^{T}_{N}α^{k}_{N}

Solving SVM Quadratic Programming Problem

### Avoid Memory Problems

The new objective function 1

2α^{T}_{B}QBBαB + (−eB + QBNα^{k}_{N})^{T}αB + constant
Only B columns of Q needed (|B| ≥ 2)

Calculated when used Trade time for space

Popular software such as SVM^{light}, LIBSVM,
SVMTorch are of this type

### How Decomposition Methods Perform?

Convergence not very fast

But, no need to have very accurate α Prediction not affected much

In some situations, # support vectors # training points

Initial α^{1} = 0, some instances never used

Solving SVM Quadratic Programming Problem

An example of training 50,000 instances using LIBSVM

$svm-train -c 16 -g 4 -m 400 22features Total nSV = 3370

Time 79.524s

On a Xeon 2.0G machine

Calculating Q may have taken more time

#SVs = 3,370 50,000

A good case where some remain at zero all the time

### Issues of Decomposition Methods

Techniques for faster decomposition methods store recently used kernel elements working set size/selection

theoretical issues: convergence

and others (details not discussed here) But training large data still difficult

Kernel square to the number of data Training millions of data time consuming Will discuss some possible approaches

Training large-scale data

### Outline

Introduction to SVM

Solving SVM Quadratic Programming Problem Training large-scale data

Linear SVM

Discussion and Conclusions

### Parallel: Multi-core/Shared Memory

Most computation of decomposition methods:

kernel evaluations

Easily parallelized via openMP One line change of LIBSVM

Each core/CPU calculates part of a kernel column Multicore Shared-memory

1 80 1 100

2 48 2 57

4 32 4 36

8 27 8 28

Same 50,000 data (kernel evaluations: 80% time) Using GPU [Catanzaro et al., 2008]

Training large-scale data

### Parallel: Distributed Environments

What if data data cannot fit into memory?

Use distributed environments PSVM: [Chang et al., 2007]

http://code.google.com/p/psvm/

π-SVM: http://pisvm.sourceforge.net, Parallel GPDT [Zanni et al., 2006]

All use MPI

They report good speed-up

But on certain environments, communication cost a concern

### Approximations

Subsampling

Simple and often effective Many more advanced techniques

Incremental training: (e.g., [Syed et al., 1999]) Data ⇒ 10 parts

train 1st part ⇒ SVs, train SVs + 2nd part, . . . Select and train good points: KNN or heuristics e.g., [Bakır et al., 2005]

Training large-scale data

### Approximations (Cont’d)

Approximate the kernel; e.g., [Fine and Scheinberg, 2001, Williams and Seeger, 2001]

Use part of the kernel; e.g.,

[Lee and Mangasarian, 2001, Keerthi et al., 2006]

And many others

Some simple but some sophisticated

### Parallelization or Approximation

Difficult to say Parallel: general

Approximation: simpler in some cases We can do both

For certain problems, approximation doesn’t easily work

Training large-scale data

### Parallelization or Approximation (Cont’d)

covtype: 500k training and 80k testing rcv1: 550k training and 14k testing

covtype rcv1

Training size Accuracy Training size Accuracy

50k 92.5% 50k 97.2%

100k 95.3% 100k 97.4%

500k 98.2% 550k 97.8%

For large sets, selecting a right approach is essential We illustrate this point using linear SVM for

document classification

### Outline

Introduction to SVM

Solving SVM Quadratic Programming Problem Training large-scale data

Linear SVM

Discussion and Conclusions

Linear SVM

### Linear Support Vector Machines

Data not mapped to another space

In theory, RBF kernel with certain parameters

⇒ as good geleralization performance as linear [Keerthi and Lin, 2003]

But sometimes can easily solve much larger linear SVMs

Training of linear/nonlinear SVMs should be separately considered

### Linear Support Vector Machines (Cont’d)

Linear SVM useful if accuracy similar to nonlinear Will discuss an example of linear SVM for document classification

Linear SVM

### Linear SVM for Large Document Sets

Document classification

Bag of words model (TF-IDF or others) A large # of features

Can solve larger problems than kernelized cases Recently an active research topic

SVM^{perf} [Joachims, 2006]

Pegasos [Shalev-Shwartz et al., 2007]

LIBLINEAR [Lin et al., 2007, Hsieh et al., 2008]

and others

### Linear SVM

Primal without the bias term b minw

1

2w^{T}w + C

l

X

i =1

max 0, 1 − y_{i}w^{T}x_{i}
Dual

minα f (α) = 1

2α^{T}Qα − e^{T}α
subject to 0 ≤ α_{i} ≤ C , ∀i

No linear constraint y^{T}α = 0
Q_{ij} = y_{i}y_{j}x^{T}_{i} x_{j}

Linear SVM

### A Comparison: LIBSVM and LIBLINEAR

rcv1: # data: > 600k, # features: > 40k TF-IDF

Using LIBSVM (linear kernel)

> 10 hours

Using LIBLINEAR

Computation: < 5 seconds; I/O: 60 seconds Same stopping condition

Accuracy similar to nonlinear

### Revisit Decomposition Methods

The extreme: update one variable at a time Reduced to

α_{i} ← min

max

α_{i} − ∇_{i}f (α)
Q_{ii} , 0

, C

where

∇_{i}f (α) = (Qα)_{i} − 1 =

l

X

j =1

Q_{ij}α_{j} − 1
O(nl ) to calculate i th row of Q

n: # features, l : # data

Linear SVM

For linear SVM, define
w = X^{l}

j =1y_{j}α_{j}x_{j},
Much easier: O(n)

∇_{i}f (α) = y_{i}w^{T}x_{i} − 1
All we need is to maintain w. If

¯

α_{i} ← α_{i}
then

w ← w + (αi − ¯αi)yixi

### Testing Accuracy (Time in Seconds)

L1-SVM: news20 L2-SVM: news20

L1-SVM: rcv1 L2-SVM: rcv1

Linear SVM

### Analysis

Other implementation details in [Hsieh et al., 2008]

Decomposition method for linear/nonlinear kernels:

O(nl ) per iteration

New way for linear: O(n) per iteration Faster if # iterations not l times more

A few seconds for million data; Any limitation?

Less effective if

# features small: should solve primal Large penalty parameter C

### Analysis (Cont’d)

One must be careful on comparisons Now we have two decomposition methods (nonlinear and linear)

Similar theoretical convergence rates

Very different practical behaviors for certain problems

This partially explains controversial comparisons in some recent work

Linear SVM

### Analysis (Cont’d)

A lesson: different SVMs

To handle large data ⇒ may need different training strategies

Even just for linear SVM

# data # features

# data # features

# data, # features both large Should use different methods

For example, # data # features

primal based method; (but why not nonlinear?)

### Outline

Introduction to SVM

Solving SVM Quadratic Programming Problem Training large-scale data

Linear SVM

Discussion and Conclusions

Discussion and Conclusions

### Discussion and Conclusions

Linear versus nonlinear

In this competition, most use linear (wild track) Even accuracy may be worse

Recall I mention “parallelization” &

“approximation”

Linear is essentially an approximation of nonlinear For large data, selecting a right approach seems to be essential

But finding a suitable one is difficult

### Discussion and Conclusions (Cont’d)

This (i.e., “too many approaches”) is indeed bad from the viewpoint of designing machine learning software

The success of LIBSVM and SVM^{light}
Simple and general

Developments in both directions (general and specific) will help to advance SVM training