¡ ¡ http://www.mmds.org Assumption: Data lies on or near a low d -dimensional subspace Axes of this subspace are effective representation of the data

(1)

Mining of Massive Datasets

Jure Leskovec, Anand Rajaraman, Jeff Ullman

Stanford University

http://www.mmds.org

¡ Assumption: Data lies on or near a low d-dimensional subspace

¡ Axes of this subspace are effective

representation of the data

(2)

¡ Compress / reduce dimensionality:

§ 10 ⁶ rows; 10 ³ columns; no updates

§ Random access to any cell(s); small error: OK

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org 3

The above matrix is really “2-dimensional.” All rows can be reconstructed by scaling [1 1 1 0 0] or [0 0 0 1 1]

¡ Q: What is rank of a matrix A?

¡ A: Number of linearly independent columns of A

¡ For example:

§ Matrix A = has rank r=2

§ Why? The first two rows are linearly independent, so the rank is at least 2, but all three rows are linearly dependent (the first is equal to the sum of the second and third) so the rank must be less than 3.

¡ Why do we care about low rank?

§ We can write A as two “basis” vectors: [1 2 1] [-2 -3 1]

§ And new coordinates of : [1 0] [0 1] [1 -1]

(3)

¡ Cloud of points 3D space:

§ Think of point positions as a matrix:

¡ We can rewrite coordinates more efficiently!

§ Old basis vectors: [1 0 0] [0 1 0] [0 0 1]

§ New basis vectors: [1 2 1] [-2 -3 1]

§ Then A has new coordinates: [1 0]. B: [0 1], C: [1 -1]

§ Notice: We reduced the number of coordinates!

1 row per point:

A B

C A

¡ Goal of dimensionality reduction is to discover the axis of data!

Rather than representing every point with 2 coordinates we represent each point with 1 coordinate (corresponding to the position of the point on the red line).

By doing this we incur a bit of

error as the points do not

exactly lie on the line

(4)

Why reduce dimensions?

¡ Discover hidden correlations/topics

§ Words that occur commonly together

¡ Remove redundant and noisy features

§ Not all words are useful

¡ Interpretation and visualization

¡ Easier storage and processing of the data

A _[m x n] = U _[m x r] S _{[ r x r]} (V _[n x r] ) ^T

¡ A: Input data matrix

§ m x n matrix (e.g., m documents, n terms)

¡ U: Left singular vectors

§ m x r matrix (m documents, r concepts)

¡ S: Singular values

§ r x r diagonal matrix (strength of each ‘concept’) (r : rank of the matrix A)

¡ V: Right singular vectors

§ n x r matrix (n terms, r concepts)

(5)

9

m A

n

m S

n

U

V ^T

»

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

T

m A

n

» ⁺

s

1

u

₁

v

₁

s

₂

u

₂

v

₂

σ _i … scalar u _i … vector v _i … vector

T

(6)

It is always possible to decompose a real matrix A into A = U S V ^T , where

¡ U, S, V: unique

¡ U, V: column orthonormal

§ U ^T U = I; V ^T V = I (I: identity matrix)

§ (Columns are orthogonal unit vectors)

¡ S: diagonal

§ Entries (singular values) are positive,

and sorted in decreasing order (σ ₁ ³ σ ₂ ³ ... ³ 0)

Nice proof of uniqueness: http://www.mpi-inf.mpg.de/~bast/ir-seminar-ws04/lecture2.pdf

¡ A = U S V ^T - example: Users to Movies

=

SciFi

Romnce

Ma tr ix Alie n Se re n it y Ca sa b la n ca Am elie

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

m S

n

U

V ^T

“Concepts”

AKA Latent dimensions

AKA Latent factors

(7)

¡ A = U S V ^T - example: Users to Movies

=

SciFi

Romnce

x x

Ma tr ix Alie n Se re n it y Ca sa b la n ca Am elie

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

0.13 0.02 -0.01 0.41 0.07 -0.03 0.55 0.09 -0.04 0.68 0.11 -0.05 0.15 -0.59 0.65 0.07 -0.73 -0.67 0.07 -0.29 0.32

12.4 0 0 0 9.5 0 0 0 1.3

0.56 0.59 0.56 0.09 0.09 0.12 -0.02 0.12 -0.69 -0.69 0.40 -0.80 0.40 0.09 0.09

¡ A = U S V ^T - example: Users to Movies

SciFi-concept

Romance-concept

=

SciFi

Romnce

x x

Ma tr ix Alie n Se re n it y Ca sa b la n ca Am elie

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

0.13 0.02 -0.01 0.41 0.07 -0.03 0.55 0.09 -0.04 0.68 0.11 -0.05 0.15 -0.59 0.65 0.07 -0.73 -0.67 0.07 -0.29 0.32

12.4 0 0 0 9.5 0 0 0 1.3

0.56 0.59 0.56 0.09 0.09

0.12 -0.02 0.12 -0.69 -0.69

0.40 -0.80 0.40 0.09 0.09

(8)

¡ A = U S V ^T - example:

Romance-concept

U is “user-to-concept”

similarity matrix

SciFi-concept

=

SciFi

Romnce

x x

Ma tr ix Alie n Se re n it y Ca sa b la n ca Am elie

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

0.13 0.02 -0.01 0.41 0.07 -0.03 0.55 0.09 -0.04 0.68 0.11 -0.05 0.15 -0.59 0.65 0.07 -0.73 -0.67 0.07 -0.29 0.32

12.4 0 0 0 9.5 0 0 0 1.3

0.56 0.59 0.56 0.09 0.09 0.12 -0.02 0.12 -0.69 -0.69 0.40 -0.80 0.40 0.09 0.09

¡ A = U S V ^T - example:

SciFi

Romnce

SciFi-concept

“strength” of the SciFi-concept

=

SciFi

Romnce

x x

Ma tr ix Alie n Se re n it y Ca sa b la n ca Am elie

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

0.13 0.02 -0.01 0.41 0.07 -0.03 0.55 0.09 -0.04 0.68 0.11 -0.05 0.15 -0.59 0.65 0.07 -0.73 -0.67 0.07 -0.29 0.32

12.4 0 0 0 9.5 0 0 0 1.3

0.56 0.59 0.56 0.09 0.09

0.12 -0.02 0.12 -0.69 -0.69

0.40 -0.80 0.40 0.09 0.09

(9)

¡ A = U S V ^T - example:

SciFi-concept

V is “movie-to-concept”

similarity matrix

SciFi-concept

=

SciFi

Romnce

x x

Ma tr ix Alie n Se re n it y Ca sa b la n ca Am elie

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

0.13 0.02 -0.01 0.41 0.07 -0.03 0.55 0.09 -0.04 0.68 0.11 -0.05 0.15 -0.59 0.65 0.07 -0.73 -0.67 0.07 -0.29 0.32

12.4 0 0 0 9.5 0 0 0 1.3

0.56 0.59 0.56 0.09 0.09 0.12 -0.02 0.12 -0.69 -0.69 0.40 -0.80 0.40 0.09 0.09

‘movies’, ‘users’ and ‘concepts’:

¡ U: user-to-concept similarity matrix

¡ V: movie-to-concept similarity matrix

¡ S: its diagonal elements:

‘strength’ of each concept

(10)

v

₁

first right singular vector

Movie 1 rating

Mov ie 2 ra ti ng

¡ Instead of using two coordinates (𝒙, 𝒚) to describe point locations, let’s use only one coordinate 𝒛

¡ Point’s position is its location along vector 𝒗 _𝟏

¡ How to choose 𝒗 𝟏 ? Minimize reconstruction error

(11)

¡ Goal: Minimize the sum of reconstruction errors:

) ) 𝑥 _+, − 𝑧 _+, ^/

0 ,12 3

+12

§ where 𝒙 𝒊𝒋 are the “old” and 𝒛 𝒊𝒋 are the

“new” coordinates

¡ SVD gives ‘best’ axis to project on:

§ ‘best’ = minimizing the reconstruction errors

¡ In other words, minimum reconstruction error

v

₁

first right singular vector

Movie 1 rating

Mo vie 2 r atin g

¡ A = U S V ^T - example:

§ V: “movie-to-concept” matrix

§ U: “user-to-concept” matrix

v

₁

first right singular vector

Movie 1 rating

Mo vie 2 r atin g

= x x

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

0.13 0.02 -0.01 0.41 0.07 -0.03 0.55 0.09 -0.04 0.68 0.11 -0.05 0.15 -0.59 0.65 0.07 -0.73 -0.67 0.07 -0.29 0.32

12.4 0 0 0 9.5 0 0 0 1.3

0.56 0.59 0.56 0.09 0.09

0.12 -0.02 0.12 -0.69 -0.69

0.40 -0.80 0.40 0.09 0.09

(12)

¡ A = U S V ^T - example:

v

₁

first right singular vector

Movie 1 rating

Mo vie 2 r atin g

variance (‘spread’) on the v

₁

axis

= x x

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

0.13 0.02 -0.01 0.41 0.07 -0.03 0.55 0.09 -0.04 0.68 0.11 -0.05 0.15 -0.59 0.65 0.07 -0.73 -0.67 0.07 -0.29 0.32

12.4 0 0 0 9.5 0 0 0 1.3

0.56 0.59 0.56 0.09 0.09 0.12 -0.02 0.12 -0.69 -0.69 0.40 -0.80 0.40 0.09 0.09

A = U S V ^T - example:

¡ U S: Gives the coordinates of the points in the

projection axis

v

₁

first right singular vector

Movie 1 rating

Mo vie 2 r atin g

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

1.61 0.19 -0.01 5.08 0.66 -0.03 6.82 0.85 -0.05 8.43 1.04 -0.06 1.86 -5.60 0.84 0.86 -6.93 -0.87 0.86 -2.75 0.41 Projection of users

on the “Sci-Fi” axis

(U S) ^T :

(13)

More details

¡ Q: How exactly is dim. reduction done?

= x x

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

0.13 0.02 -0.01 0.41 0.07 -0.03 0.55 0.09 -0.04 0.68 0.11 -0.05 0.15 -0.59 0.65 0.07 -0.73 -0.67 0.07 -0.29 0.32

12.4 0 0 0 9.5 0 0 0 1.3

0.56 0.59 0.56 0.09 0.09 0.12 -0.02 0.12 -0.69 -0.69 0.40 -0.80 0.40 0.09 0.09

More details

¡ Q: How exactly is dim. reduction done?

¡ A: Set smallest singular values to zero

= x x

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

0.13 0.02 -0.01 0.41 0.07 -0.03 0.55 0.09 -0.04 0.68 0.11 -0.05 0.15 -0.59 0.65 0.07 -0.73 -0.67 0.07 -0.29 0.32

12.4 0 0 0 9.5 0 0 0 1.3

0.56 0.59 0.56 0.09 0.09

0.12 -0.02 0.12 -0.69 -0.69

0.40 -0.80 0.40 0.09 0.09

(14)

More details

¡ Q: How exactly is dim. reduction done?

¡ A: Set smallest singular values to zero

x x

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

0.13 0.02 -0.01 0.41 0.07 -0.03 0.55 0.09 -0.04 0.68 0.11 -0.05 0.15 -0.59 0.65 0.07 -0.73 -0.67 0.07 -0.29 0.32

12.4 0 0 0 9.5 0 0 0 1.3

0.56 0.59 0.56 0.09 0.09 0.12 -0.02 0.12 -0.69 -0.69 0.40 -0.80 0.40 0.09 0.09

»

More details

¡ Q: How exactly is dim. reduction done?

¡ A: Set smallest singular values to zero

x x

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

0.13 0.02 -0.01 0.41 0.07 -0.03 0.55 0.09 -0.04 0.68 0.11 -0.05 0.15 -0.59 0.65 0.07 -0.73 -0.67 0.07 -0.29 0.32

12.4 0 0 0 9.5 0 0 0 1.3

0.56 0.59 0.56 0.09 0.09 0.12 -0.02 0.12 -0.69 -0.69 0.40 -0.80 0.40 0.09 0.09

»

(15)

More details

¡ Q: How exactly is dim. reduction done?

¡ A: Set smallest singular values to zero

» x x

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

0.13 0.02 0.41 0.07 0.55 0.09 0.68 0.11 0.15 -0.59 0.07 -0.73 0.07 -0.29

12.4 0 0 9.5

0.56 0.59 0.56 0.09 0.09 0.12 -0.02 0.12 -0.69 -0.69

More details

¡ Q: How exactly is dim. reduction done?

¡ A: Set smallest singular values to zero

»

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

0.92 0.95 0.92 0.01 0.01 2.91 3.01 2.91 -0.01 -0.01 3.90 4.04 3.90 0.01 0.01 4.82 5.00 4.82 0.03 0.03 0.70 0.53 0.70 4.11 4.11 -0.69 1.34 -0.69 4.78 4.78 0.32 0.23 0.32 2.01 2.01 Frobenius norm:

ǁMǁ _F = ÖΣ _ij M _ij ² ^ǁA-Bǁ _{is “small”} ^F ^{= Ö Σ} îj ^(A îj ^-B îj ⁾ ²

(16)

A U

Sigma

V ^T

=

B U

Sigma

V ^T

=

B is best approximation of A

¡ Theorem:

Let A = U S V ^T and B = U S V ^T where

S = diagonal r ^x r matrix with s _i =σ _i (i=1…k) else s _i =0 then B is a best rank(B)=k approx. to A

What do we mean by “best”:

§ B is a solution to min _B ǁA-Bǁ _F where rank(B)=k

𝜎 ₂₂ Σ 𝜎 ₈₈

𝐴 − 𝐵 _; = ) 𝐴 _+, − 𝐵 _+, ^/

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

+,

32

(17)

¡ Theorem: Let A = U S V ^T (σ ₁ ³σ ₂ ³…, rank(A)=r)

then B = U S V ^T

§ S = diagonal r ^x r matrix where s _i =σ _i (i=1…k) else s _i =0

is a best rank-k approximation to A:

§ B is a solution to min _B ǁA-Bǁ _F where rank(B)=k

¡ We will need 2 facts:

§ 𝑀 _; = ∑ 𝑞 ₊ ₊₊ ^/ where M = P Q R is SVD of M

§ U S V ^T - U S V ^T = U (S - S) V ^T

𝜎 ₂₂ Σ 𝜎 ₈₈

¡ We will need 2 facts:

§ 𝑀 _; = ∑ 𝑞 _A _AA ^/ where M = P Q R is SVD of M

§ U S V ^T - U S V ^T = U (S - S) V ^T

We apply:

-- P column orthonormal -- R row orthonormal -- Q is diagonal

Details!

(18)

¡ A = U S V ^T , B = U S V ^T (σ ₁ ³σ ₂ ³… ³ 0, rank(A)=r)

§ S = diagonal n ^x n matrix where s _i =σ _i (i=1…k) else s _i =0

then B is solution to min _B ǁA-Bǁ _F , rank(B)=k

¡ Why?

¡ We want to choose s _i to minimize ∑ 𝜎 ₊ ₊ − 𝑠 ₊ ^/

¡ Solution is to set s _i =σ _i (i=1…k) and other s _i =0

å å

å = = + = +

= +

-

= ^r

k i

i r

k i

i k

i

i i

s s

i 1

2 1 2

1 ) 2

(

min s s s

å =

= - = S - = ^r -

i

i i F s

k F B rank

B A B S s

i 1

2 )

(

, min min min ( s )

We used: U S V ^T - U S V ^T = U (S - S) V ^T

Equivalent:

‘spectral decomposition’ of the matrix:

= x x

u

₁

u

₂

σ

₁

σ

₂

v

₁

v

₂

1 1 1 0 0

3 3 3 0 0

4 4 4 0 0

5 5 5 0 0

0 2 0 4 4

0 0 0 5 5

0 1 0 2 2

(19)

Equivalent:

‘spectral decomposition’ of the matrix

= σ ₁ ^u 1 v ^T ₁ + σ ₂ u ₂ v ^T ₂ +...

n

m

n x 1 1 x m

k terms

Assume: σ ₁ ³ σ ₂ ³ σ ₃ ³ ... ³ 0

Why is setting small σ _i to 0 the right thing to do?

Vectors u _i and v _i are unit length, so σ _i scales them.

So, zeroing small σ _i introduces less error.

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

Q: How many σ ^s to keep?

A: Rule-of-a thumb:

keep 80-90% of ‘energy’ = ∑ 𝝈 _𝒊 _𝒊 ^𝟐

= σ ₁ u ₁ v ^T ₁ + σ ₂ u ₂ v ^T ₂ +...

n

m

Assume: σ ₁ ³ σ ₂ ³ σ ₃ ³ ...

1 1 1 0 0

3 3 3 0 0

4 4 4 0 0

5 5 5 0 0

0 2 0 4 4

0 0 0 5 5

0 1 0 2 2

(20)

¡ To compute SVD:

§ O(nm ² ) or O(n ² m) (whichever is less)

¡ But:

§ Less work, if we just want singular values

§ or if we want first k singular vectors

§ or if the matrix is sparse

¡ Implemented in linear algebra packages like

§ LINPACK, Matlab, SPlus, Mathematica ...

¡ SVD: A= U S V ^T : unique

§ U: user-to-concept similarities

§ V: movie-to-concept similarities

§ S : strength of each concept

¡ Dimensionality reduction:

§ keep the few largest singular values (80-90% of ‘energy’)

§ SVD: picks up linear correlations

(21)

¡ SVD gives us:

§ A = U S V ^T

¡ Eigen-decomposition:

§ A = X L X ^T

§ A is symmetric

§ U, V, X are orthonormal (U ^T U=I),

§ L, S are diagonal

¡ Now let’s calculate:

§ AA ^T = US V ^T (US V ^T ) ^T = US V ^T (VS ^T U ^T ) = USS ^T U ^T

§ A ^T A = V S ^T U ^T (US V ^T ) = V SS ^T V ^T

¡ SVD gives us:

§ A = U S V ^T

¡ Eigen-decomposition:

§ A = X L X ^T

§ A is symmetric

§ U, V, X are orthonormal (U ^T U=I),

§ L, S are diagonal

¡ Now let’s calculate:

§ AA ^T = US V ^T (US V ^T ) ^T = US V ^T (VS ^T U ^T ) = USS ^T U ^T

§ A ^T A = V S ^T U ^T (US V ^T ) = V SS ^T V ^T

X L ² X ^T

Shows how to compute SVD using eigenvalue

decomposition!

(22)

¡ A A ^T = U S ² U ^T

¡ A ^T A = V S ² V ^T

¡ (A ^T A) ^k = V S ^2k V ^T

§ E.g.: (A ^T A) ² = V S ² V ^T V S ² V ^T = V S ⁴ V ^T

¡ (A ^T A) ^k ~ v ₁ σ ₁ ^2k v ₁ ^T for k>>1

(23)

¡ Q: Find users that like ‘Matrix’

¡ A: Map query into a ‘concept space’ – how?

=

SciFi

Romnce

x x

Ma tr ix Alie n Se re n it y Ca sa b la n ca Am elie

1 1 1 0 0 3 3 3 0 0 4 4 4 0 0 5 5 5 0 0 0 2 0 4 4 0 0 0 5 5 0 1 0 2 2

0.13 0.02 -0.01 0.41 0.07 -0.03 0.55 0.09 -0.04 0.68 0.11 -0.05 0.15 -0.59 0.65 0.07 -0.73 -0.67 0.07 -0.29 0.32

12.4 0 0 0 9.5 0 0 0 1.3

0.56 0.59 0.56 0.09 0.09 0.12 -0.02 0.12 -0.69 -0.69 0.40 -0.80 0.40 0.09 0.09

¡ Q: Find users that like ‘Matrix’

¡ A: Map query into a ‘concept space’ – how?

5 0 0 0 0 q =

Matrix

Al ie n

v1

q

v2 Ma tr ix Alie n Se re n it y Ca sa b la n ca Am elie

Project into concept space:

Inner product with each

‘concept’ vector v _i

(24)

¡ Q: Find users that like ‘Matrix’

¡ A: Map query into a ‘concept space’ – how?

v1

q

**q*v** ₁ 5 0 0 0 0

Ma tr ix Alie n Se re n it y Ca sa b la n ca Am elie

v2

Matrix

Al ie n

q =

Project into concept space:

Inner product with each

‘concept’ vector v _i

Compactly, we have:

q _concept = q V E.g.:

movie-to-concept similarities (V)

=

SciFi-concept

5 0 0 0 0 Ma tr ix Alie n Se re n it y Ca sa b la n ca Am elie

q =

0.56 0.12 0.59 -0.02 0.56 0.12 0.09 -0.69 0.09 -0.69

x 2.8 0.6

(25)

¡ How would the user d that rated (‘Alien’, ‘Serenity’) be handled?

d _concept = d V E.g.:

movie-to-concept similarities (V)

=

SciFi-concept

0 4 5 0 0 Ma tr ix Alie n Se re n it y Ca sa b la n ca Am elie

q =

0.56 0.12 0.59 -0.02 0.56 0.12 0.09 -0.69 0.09 -0.69

x 5.2 0.4

¡ Observation: User d that rated (‘Alien’,

‘Serenity’) will be similar to user q that rated (‘Matrix’), although d and q have zero ratings in common!

0 4 5 0 0

d =

SciFi-concept

5 0 0 0 0

q =

Ma tr ix Alie n Se re n it y Ca sa b la n ca Am elie

Zero ratings in common Similarity ≠ 0

2.8 0.6

5.2 0.4

(26)

+ Optimal low-rank approximation in terms of Frobenius norm

- Interpretability problem:

§ A singular vector specifies a linear

combination of all input columns or rows

- Lack of sparsity:

§ Singular vectors are dense!

=

U S

V

^T

(27)

¡ Goal: Express A as a product of matrices C,U,R Make ǁA-C·U·Rǁ _F small

¡ “Constraints” on C and R:

A C U R

¡ Goal: Express A as a product of matrices C,U,R Make ǁA-C·U·Rǁ _F small

¡ “Constraints” on C and R:

Pseudo-inverse of the intersection of C and R

A C U R

Frobenius norm:

ǁXǁ _F = Ö Σ _ij X _ij ²

(28)

¡ Let:

A _k be the “best” rank k approximation to A (that is, A _k is SVD of A)

Theorem [Drineas et al.]

CUR in O(m·n) time achieves

§ ǁ A-CURǁ F £ ǁA-A _k ǁ _F + eǁAǁ F

with probability at least 1-d, by picking

§ O(k log(1/d)/e ² ) columns, and

§ O(k ² log ³ (1/d)/e ⁶ ) rows

In practice:

Pick 4k cols/rows

¡ Sampling columns (similarly for rows):

Note this is a randomized algorithm, same

column can be sampled more than once

(29)

¡ Let W be the “intersection” of sampled columns C and rows R

§ Let SVD of W = X Z Y ^T

¡ Then: U = Y (Z ⁺ ) ² X ^T

§ Z ⁺ : reciprocals of non-zero singular values: Z ⁺ _ii =1/ Z _ii

A C

R

» W

¡ For example:

§ Select 𝒄 = 𝑶 ^{𝒌 𝒍𝒐𝒈 𝒌} _𝜺 _𝟐 columns of A using ColumnSelect algorithm

§ Select 𝒓 = 𝑶 ^{𝒌 𝒍𝒐𝒈 𝒌} _𝜺 _𝟐 rows of A using ColumnSelect algorithm

§ Set 𝑼 = 𝑾 ^P

¡ Then:

with probability 98%

In practice:

Pick 4k cols/rows

for a “rank-k” approximation

SVD error

CUR error

(30)

+ Easy interpretation

• Since the basis vectors are actual columns and rows

+ Sparse basis

• Since the basis vectors are actual columns and rows

- Duplicate columns and rows

• Columns of large norms will be sampled many times

Singular vector Actual column

¡ If we want to get rid of the duplicates:

§ Throw them away

§ Scale (multiply) the columns/rows by the square root of the number of duplicates

A C

_d

R

_d

C

_s

R

_s

Construct a

small U

(31)

SVD: A = U S V ^T

Huge but sparse Big and dense

CUR: A = C U R

Huge but sparse Big but sparse dense but small

sparse and small

¡ DBLP bibliographic data

§ Author-to-conference big sparse matrix

§ A _ij : Number of papers published by author i at conference j

§ 428K authors (rows), 3659 conferences (columns)

§ Very sparse

¡ Want to reduce dimensionality

§ How much time does it take?

§ What is the reconstruction error?

§ How much space do we need?

(32)

¡ Accuracy:

§ 1 – relative sum squared errors

¡ Space ratio:

§ #output matrix entries / #input matrix entries

¡ CPU time

J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org 63 CUR

CUR no duplicates

SVD CUR CUR no dup

Sun, Faloutsos: Less is More: Compact Matrix Decomposition for Large Sparse Graphs, SDM ’07.

CUR SVD

¡ SVD is limited to linear projections:

§ Lower-dimensional linear projection that preserves Euclidean distances

¡ Non-linear methods: Isomap

§ Data lies on a nonlinear low-dim curve aka manifold

§ Use the distance as measured along the manifold

§ How?

§ Build adjacency graph

§ Geodesic distance is graph distance

§ SVD/PCA the graph pairwise distance matrix

(33)

¡ Drineas et al., Fast Monte Carlo Algorithms for Matrices III:

Computing a Compressed Approximate Matrix Decomposition, SIAM Journal on Computing, 2006.

¡ J. Sun, Y. Xie, H. Zhang, C. Faloutsos: Less is More:

Compact Matrix Decomposition for Large Sparse Graphs, SDM 2007

¡ Intra- and interpopulation genotype reconstruction from tagging SNPs, P. Paschou, M. W. Mahoney, A. Javed, J. R.

Kidd, A. J. Pakstis, S. Gu, K. K. Kidd, and P. Drineas, Genome Research, 17(1), 96-107 (2007)

¡ Tensor-CUR Decompositions For Tensor-Based Data, M. W.

Mahoney, M. Maggioni, and P. Drineas, Proc. 12-th Annual SIGKDD, 327-336 (2006)

¡ ¡ http://www.mmds.org Assumption: Data lies on or near a low d -dimensional subspace Axes of this subspace are effective representation of the data

Mining of Massive Datasets

Jure Leskovec, Anand Rajaraman, Jeff Ullman

Stanford University

http://www.mmds.org

¡ Assumption: Data lies on or near a low d-dimensional subspace

¡ Axes of this subspace are effective

representation of the data

¡ Compress / reduce dimensionality:

§ 10 6 rows; 10 3 columns; no updates

§ Random access to any cell(s); small error: OK

The above matrix is really “2-dimensional.” All rows can be reconstructed by scaling [1 1 1 0 0] or [0 0 0 1 1]

¡ Q: What is rank of a matrix A?

¡ A: Number of linearly independent columns of A

¡ For example:

§ Matrix A = has rank r=2

§ Why? The first two rows are linearly independent, so the rank is at least 2, but all three rows are linearly dependent (the first is equal to the sum of the second and third) so the rank must be less than 3.

¡ Why do we care about low rank?

§ We can write A as two “basis” vectors: [1 2 1] [-2 -3 1]

§ And new coordinates of : [1 0] [0 1] [1 -1]

¡ Cloud of points 3D space:

§ Think of point positions as a matrix:

¡ We can rewrite coordinates more efficiently!

§ Old basis vectors: [1 0 0] [0 1 0] [0 0 1]

§ New basis vectors: [1 2 1] [-2 -3 1]

§ Then A has new coordinates: [1 0]. B: [0 1], C: [1 -1]

§ Notice: We reduced the number of coordinates!

1 row per point:

A B

C A

¡ Goal of dimensionality reduction is to discover the axis of data!

Rather than representing every point with 2 coordinates we represent each point with 1 coordinate (corresponding to the position of the point on the red line).

By doing this we incur a bit of

error as the points do not

exactly lie on the line

Why reduce dimensions?

¡ Discover hidden correlations/topics

§ Words that occur commonly together

¡ Remove redundant and noisy features

§ Not all words are useful

¡ Interpretation and visualization

¡ Easier storage and processing of the data

A [m x n] = U [m x r] S [ r x r] (V [n x r] ) T

¡ A: Input data matrix

§ m x n matrix (e.g., m documents, n terms)

¡ U: Left singular vectors

§ m x r matrix (m documents, r concepts)

¡ S: Singular values

§ r x r diagonal matrix (strength of each ‘concept’) (r : rank of the matrix A)

¡ V: Right singular vectors

§ n x r matrix (n terms, r concepts)

m A

n

m S

n

U

V T

»

T

m A

n

» +

s

u

v

s

u

v

σ i … scalar u i … vector v i … vector

T

It is always possible to decompose a real matrix A into A = U S V T , where

¡ U, S, V: unique

¡ U, V: column orthonormal

§ U T U = I; V T V = I (I: identity matrix)

§ (Columns are orthogonal unit vectors)

¡ S: diagonal

§ Entries (singular values) are positive,

and sorted in decreasing order (σ 1 ³ σ 2 ³ ... ³ 0)

Nice proof of uniqueness: http://www.mpi-inf.mpg.de/~bast/ir-seminar-ws04/lecture2.pdf

¡ A = U S V T - example: Users to Movies

§ 10 ⁶ rows; 10 ³ columns; no updates

A _[m x n] = U _[m x r] S _{[ r x r]} (V _[n x r] ) ^T

V ^T

» ⁺

σ _i … scalar u _i … vector v _i … vector

It is always possible to decompose a real matrix A into A = U S V ^T , where

§ U ^T U = I; V ^T V = I (I: identity matrix)

and sorted in decreasing order (σ ₁ ³ σ ₂ ³ ... ³ 0)

¡ A = U S V ^T - example: Users to Movies

V ^T

¡ A = U S V ^T - example: Users to Movies

¡ A = U S V ^T - example: Users to Movies

¡ A = U S V ^T - example:

¡ A = U S V ^T - example:

¡ A = U S V ^T - example: