Label Space Coding for Multi-label Classification

Label Space Coding for Multi-label Classification

Hsuan-Tien Lin National Taiwan University 3rd TWSIAM Annual Meeting, 05/30/2015

joint works with

Farbound Tai(MLD Workshop 2010, NC Journal 2012)&

Yao-Nan Chen(NIPS Conference 2012)

Multi-label Classification

Which Fruit?

?

apple orange strawberry kiwi

multi-class classification:

classify input (picture) toone category (label)

Multi-label Classification

Which Fruits?

?: {orange, strawberry, kiwi}

apple orange strawberry kiwi

multi-labelclassification:

classify input tomultiple (or no)categories

Multi-label Classification

What Tags?

?: {machine learning,data structure,data mining,object oriented programming,artificial intelligence,compiler, architecture,chemistry,textbook,children book,. . .etc.}

anothermulti-label classification problem:

tagginginput to multiple categories

Multi-label Classification

Binary Relevance: Multi-label Classification via Yes/No

Binary

Classification {yes,no}

Multi-label w/ L classes: Lyes/no questions

machine learning(Y), data structure(N), data mining(Y), OOP(N), AI(Y), compiler(N), architecture(N), chemistry(N), textbook(Y),

children book(N),etc.

• Binary Relevance approach:

transformation tomultiple isolated binary classification

• disadvantages:

• isolation—hidden relations not exploited (e.g. ML and DMhighly correlated, MLsubset ofAI, textbook & children bookdisjoint)

• unbalanced—fewyes, manyno

Binary Relevance: simple (& good) benchmark with known disadvantages

Multi-label Classification

Multi-label Classification Setup

Given

N examples (inputx_n,label-set Y_n) ∈ X ×2^{{1,2,···L}}

• fruits: X = encoding(pictures), Yn⊆ {1, 2, · · · , 4}

• tags: X = encoding(merchandise), Yn⊆ {1, 2, · · · , L}

Goal

a multi-label classifier g(x) thatclosely predictsthe label-set Y associated with someunseen inputs x (byexploiting hidden relations/combinations between labels)

• Hamming loss: averaged symmetric difference ¹_L|g(x) 4 Y|

multi-label classification: hot and important

Compression Coding

From Label-set to Coding View

label set apple orange strawberry binary code Y₁= {o} 0 (N) 1 (Y) 0 (N) y₁= [0, 1, 0]

Y₂= {a, o} 1 (Y) 1 (Y) 0 (N) y₂= [1, 1, 0]

Y₃= {a, s} 1 (Y) 0 (N) 1 (Y) y₃= [1, 0, 1]

Y₄= {o} 0 (N) 1 (Y) 0 (N) y₄= [0, 1, 0]

Y₅= {} 0 (N) 0 (N) 0 (N) y₅= [0, 0, 0]

subset Y of 2{1,2,··· ,L}⇔ length-L binary code y

Compression Coding

Existing Approach: Compressive Sensing

General Compressive Sensing

sparse (many0) binary vectorsy ∈ {0, 1}^Lcan berobustly

compressed by projecting to M  L basis vectors {p₁,p₂, · · · ,p_M} Compressive Sensing for Multi-label Classification(Hsu et al., 2009)

1 compress: transform {(x_n,y_n)}to {(x_n,c_n)}byc_n=Py_nwith some M by LrandommatrixP = [p₁,p₂, · · · ,p_M]^T

2 learn: getregressionfunctionr(x) from x_n toc_n

3 decode: g(x) = findclosest sparse binary vectortoP^Tr(x) Compressive Sensing:

• efficient in training: random projectionw/M  L

• inefficient in testing:time-consuming decoding

Compression Coding

From Coding View to Geometric View

label set binary code Y₁= {o} y₁= [0, 1, 0]

Y₂= {a, o} y₂= [1, 1, 0]

Y₃= {a, s} y₃= [1, 0, 1]

Y₄= {o} y₄= [0, 1, 0]

Y₅= {} y₅= [0, 0, 0]

y₁,y₄ y₂ y₃

y₅

length-L binary code ⇔vertex of hypercube {0, 1}^L

Compression Coding

Geometric Interpretation of Binary Relevance

y₁,y₄ y₂ y₃

y₅

Binary Relevance: project to thenatural axes& classify

Compression Coding

Geometric Interpretation of Compressive Sensing

y₁,y₄ y₂ y₃

y₅

Compressive Sensing:

• project torandom flat (linear subspace)

• learn “on” the flat; decode toclosest sparse vertex other (better) flat? other (faster) decoding?

Compression Coding

Our Contributions

Compression Coding &

Learnable-Compression Coding

A Novel Approach for Label Space Compression

• algorithmic: first known algorithm forfeature-aware dimension reduction with fast decoding

• theoretical: justification forbestlearnableprojection

• practical: consistently better performance than compressive sensing (& binary relevance)

will now introduce the key ideas behind the approach

Compression Coding

Faster Decoding: Round-based

Compressive Sensing Revisited

• decode: g(x) = findclosest sparse binary vectorto ˜y = P^Tr(x)

For any given “intermediate prediction” (real-valued vector) ˜y,

• find closestsparsebinary vector to ˜y:slow optimization of`₁-regularizedobjective

• find closestanybinary vector to ˜y:fast g(x) =round(y)

round-based decoding: simple & faster alternative

Compression Coding

Better Projection: Principal Directions

Compressive Sensing Revisited

• compress: transform {(x_n,y_n)}to {(x_n,c_n)}byc_n=Py_nwith some M by LrandommatrixP

• randomprojection: arbitrarydirections

• bestprojection: principaldirections

principal directions: best approximation to desired out- puty_nduringcompression(why?)

Compression Coding

Novel Theoretical Guarantee

Linear Transform+Learn+Round-based Decoding

Theorem (Tai and Lin, 2012) Ifg(x) = round(P^Tr(x)),

1

L|g(x) 4 Y|

| {z }

Hamming loss

≤ const ·





kr(x) −

c

z}|{Py k²

| {z }

learn

+ ky − P^T

c

z}|{Py k²

| {z }

compress







• kr(x) − ck²: prediction errorfrom input to codeword

• ky − P^Tck²: encoding errorfrom desired output to codeword principal directions: best approximation to

desired outputy_nduringcompression(indeed)

Compression Coding

Proposed Approach 1:

Principal Label Space Transform

From Compressive Sensing to PLST

1 compress: transform {(xn,yn)}to {(xn,cn)}bycn=Pynwith the M by L principal matrixP

2 learn: getregressionfunctionr(x) from x_n toc_n

3 decode: g(x) = round(P^Tr(x))

• principal directions: viaPrincipal Component Analysison {y_n}^N_n=1

• physical meaning behindp_m: key (linear)label correlations PLST: improving CS by projecting tokey correlations

Compression Coding

Theoretical Guarantee of PLST Revisited

Linear Transform+Learn+Round-based Decoding

Theorem (Tai and Lin, 2012) Ifg(x) = round(P^Tr(x)),

1

L|g(x) 4 Y|

| {z }

Hamming loss

≤ const ·





kr(x) −

c

z}|{Py k²

| {z }

learn

+ ky − P^T

c

z}|{Py k²

| {z }

compress







• ky − P^Tck²: encoding error, minimized during encoding

• kr(x) − ck²: prediction error, minimized during learning

• but goodencodingmay not be easy tolearn; vice versa PLST: minimize two errors separately (sub-optimal)

(can we do better by minimizingjointly?)

Compression Coding

Proposed Approach 2:

Conditional Principal Label Space Transform

can we do better by minimizingjointly?

Yes and easy for ridge regression (closed-form solution)

From PLST to CPLST

1 compress: transform {(x_n,y_n)}to {(x_n,c_n)}byc_n=Py_nwith the M by L conditional principal matrixP

2 learn: getregressionfunctionr(x) from xn tocn, ideally using ridge regression

3 decode: g(x) = round(P^Tr(x))

• conditional principal directions: top eigenvectors ofY^TXX^†Y, key (linear) label correlationsthat are “easy to learn”

CPLST: project tokeylearnablecorrelations

—can also pair withkernel regression (non-linear)

Compression Coding

Hamming Loss Comparison: Full-BR, PLST & CS

0 20 40 60 80 100

0.03 0.035 0.04 0.045 0.05

Full−BR (no reduction) CS

PLST

mediamill (Linear Regression)

0 20 40 60 80 100

0.03 0.035 0.04 0.045 0.05

Full−BR (no reduction) CS

PLST

mediamill (Decision Tree)

• PLSTbetter thanFull-BR: fewer dimensions, similar (or better) performance

• PLSTbetter thanCS: faster,better performance

• similar findings acrossdata sets and regression algorithms

Compression Coding

Hamming Loss Comparison: PLST & CPLST

0 5 10 15

0.2 0.205 0.21 0.215 0.22 0.225 0.23 0.235 0.24 0.245

# of dimension

Hamning loss

pbr cpa plst cplst

yeast (Linear Regression)

• CPLSTbetter thanPLST: better performance across all dimensions

• similar findings acrossdata sets and regression algorithms

Conclusion

1 CompressionCoding(Tai & Lin, MLD Workshop 2010; NC Journal 2012)

—condense for efficiency: better (than BR) approach PLST

— key tool: PCA from Statistics/Signal Processing

2 Learnable-CompressionCoding(Chen & Lin, NIPS Conference 2012)

—condense learnably forbetterefficiency: better (than PLST) approach CPLST

— key tool: Ridge Regression from Statistics (+ PCA)

More...

• error-correcting code instead of compression, with improved decoding (Ferng and Lin, IEEE TNNLS 2013)

• multi-label classification with arbitrary loss (Li and Lin, ICML 2014)

• dynamic instead of static coding, binary instead of real coding, (...)

Thank you! Questions?