© Deng Cai, College of Computer Science, Zhejiang University

(1)

So Far…

It’s time for

 Unsupervised learning

 We are only given inputs

 Goal: find “interesting patterns”

 Discovering clusters

– Clustering

 Discovering latent factors

– Dimensionality reduction

– Topic modeling

– Matrix factorization

(2)

Deng Cai (蔡登)

College of Computer Science Zhejiang University

[email protected]

2 Matrix Factorization

(3)

What Is Matrix Factorization?

Is this factorization unique?

 Every column of and every row of are normalized

Does this factorization always exist?

∈ , ∈

Σ ∈ ΣΣ

Σ

(4)

Why Matrix Factorization?

⋯

⋮ ⋮ ⋮

⋯

⋮ ⋮

⋯

⋮ ⋮ ⋮

⋯

Each column vector of can be represented as a linear combination of column vectors of

Each column vector of can be regarded as a low dimensional

representation of corresponding column vector of

(5)

Relation to Dimensionality Reduction

If there is a matrix which satisfies:

In DR, we learn the transformation matrix In MF, we learn the basis matrix

, , ⋯ , , ⋯

∈ , ∈

(6)

Relation to Topic Modeling

Terms Documents

| ,

∑ ,

| ⋯ |

⋮ ⋱ ⋮

| ⋯ |

Latent Concepts

Terms Documents

TRADE

econom

imports

trade

| | |

(7)

Relation to Topic Modeling

| | |

| ⋯ |

⋮ ⋱ ⋮

| ⋯ |

⋮ ⋱ ⋮

| ⋯ |

⋮ ⋱ ⋮

| ⋯ |

| ,

∑ ,

| ⋯ |

⋮ ⋱ ⋮

| ⋯ |

(8)

Algorithms

Singular Value Decomposition

Nonnegative Matrix Factorization

Sparse Coding

(9)

Singular Value Decomposition (SVD)

For an arbitrary matrix ∈ there exists a factorization as follows:

Σ where

∈ , ∈ , ,

diagonal matrix Σ ∈ If

∈ , ∈ , ,

Σ diag , , ⋯ ⋯ 0

(10)

SVD: Low‐rank Approximation

SVD can be used to compute optimal low‐rank approximations.

Approximation problem:

Solution via SVD

C. Eckart, G. Young, The approximation of a matrix by another of lower rank. Psychometrika, 1, 211‐218, 1936.

∗

set small singular values to zero

∗

(11)

Low rank approximation by SVD

(12)

Relation to PCA

Given an SVD of X, the following two relations hold:

Σ ΣV V Σ Σ V

Σ Σ ΣΣ

(13)

Latent Semantic Analysis (Indexing)

The Latent Semantic Analysis via SVD can be summarized as follows:

Document similarity

, Σ , Σ

...

LSA term vectors

... LSA document vectors

=

< ^, >

...

terms

...

documents

(14)

Latent Semantic Analysis

Latent semantic space: illustrating example

(15)

Relation to Topic Modeling

| | |

| ⋯ |

⋮ ⋱ ⋮

| ⋯ |

⋮ ⋱ ⋮

| ⋯ |

⋮ ⋱ ⋮

| ⋯ |

| ,

∑ ,

| ⋯ |

⋮ ⋱ ⋮

| ⋯ |

(16)

Nonnegative Matrix Factorization

Low rank assumption (k hidden factors)

Nonnegative assumption

(17)

Non‐negative Matrix Factorization

Two cost functions

 Euclidean distance

 Divergence

(18)

Optimization Problems

Minimize with respect to and , subject to the constraints .

Minimize with respect to and ,

subject to the constraints .

(19)

NMF Optimization (Euclidean Distance)

min , . . 0, 0

tr tr

2 2 0

tr 2tr tr

Γ, same size as Φ, same size as

tr Γ tr Φ

Γ

←

2 2 Φ 0

←

(20)

Multiplicative Update Rules

The Euclidean distance is nonincreasing under the update rules

The Euclidean distance is invariant under these updates if

and only if and are at a stationary point of the distance.

(21)

NMF vs PLSA

≅ , 0, 0

| ∑ log

, 1

| |

max log

log log

max 1

, log | |

(22)

Sparse Coding

• Represent input vectors approximately as a weighted linear combination of a small number of “basis vectors.”