Christopher M. Bishop

Part 2: Unsupervised Learning

Machine Learning Techniques for Computer Vision

Microsoft Research Cambridge

Christopher M. Bishop

x

x

Overview of Part 2

• Mixture models

• EM

• Variational Inference

• Bayesian model complexity

• Continuous latent variables

The Gaussian Distribution

• Multivariate Gaussian

• Maximum likelihood

mean covariance

Gaussian Mixtures

• Linear super-position of Gaussians

• Normalization and positivity require

Example: Mixture of 3 Gaussians

0 0.5 1

0 0.5 1

(a)

0 0.5 1

0 0.5 1

(b)

Maximum Likelihood for the GMM

• Log likelihood function

• Sum over components appears inside the log

– no closed form ML solution

EM Algorithm – Informal Derivation

EM Algorithm – Informal Derivation

• M step equations

EM Algorithm – Informal Derivation

• E step equation

EM Algorithm – Informal Derivation

• Can interpret the mixing coefficients as prior probabilities

• Corresponding posterior probabilities (responsibilities)

Old Faithful Data Set

Time

between

eruptions

(minutes)

Latent Variable View of EM

• To sample from a Gaussian mixture:

– first pick one of the components with probability – then draw a sample from that component

– repeat these two steps for each new data point

0 0.5 1

0 0.5 1

(a)

Latent Variable View of EM

• Goal: given a data set, find

• Suppose we knew the colours

– maximum likelihood would involve fitting each component to the corresponding cluster

• Problem: the colours are latent (hidden) variables

Incomplete and Complete Data

complete

0 0.5 1

0 0.5 1

(a)

0 0.5 1

0 0.5 1

(b)

incomplete

Latent Variable Viewpoint

Latent Variable Viewpoint

• Binary latent variables describing which component generated each data point

• Conditional distribution of observed variable

• Prior distribution of latent variables

• Marginalizing over the latent variables we obtain

X

Z

Graphical Representation of GMM

p z _n z _n

x n

x n

S m

N

Latent Variable View of EM

• Suppose we knew the values for the latent variables – maximize the complete-data log likelihood

– trivial closed-form solution: fit each component to the corresponding set of data points

• We don’t know the values of the latent variables

– however, for given parameter values we can compute

the expected values of the latent variables

Posterior Probabilities (colour coded)

0.5 1

0.5

1

Over-fitting in Gaussian Mixture Models

• Infinities in likelihood function when a component

‘collapses’ onto a data point:

with

• Also, maximum likelihood cannot determine the number

K of components

Cross Validation

• Can select model complexity using an independent validation data set

• If data is scarce use cross-validation:

– partition data into S subsets – train on S−1 subsets

– test on remainder – repeat and average

• Disadvantages

– computationally expensive

Bayesian Mixture of Gaussians

• Parameters and latent variables appear on equal footing

• Conjugate priors

p

z _n z _n x _n x _n

L m

N

Data Set Size

• Problem 1: learn the function

for from 100 (slightly) noisy examples

– data set is computationally small but statistically large

• Problem 2: learn to recognize 1,000 everyday objects from 5,000,000 natural images

– data set is computationally large but statistically small

• Bayesian inference

– computationally more demanding than ML or MAP

Variational Inference

• Exact Bayesian inference intractable

• Markov chain Monte Carlo – computationally expensive – issues of convergence

• Variational Inference

– broadly applicable deterministic approximation – let denote all latent variables and parameters

– approximate true posterior using a simpler distribution

– minimize Kullback-Leibler divergence

General View of Variational Inference

• For arbitrary

where

Variational Lower Bound

Factorized Approximation

• Goal: choose a family of q distributions which are:

– sufficiently flexible to give good approximation – sufficiently simple to remain tractable

• Here we consider factorized distributions

• No further assumptions are required!

• Optimal solution for one factor, keeping the remainder fixed

0 0.5 1 0

0.5 1

x 1

x 2

(a)

Lower Bound

• Can also be evaluated

• Useful for maths/code verification

• Also useful for model comparison:

Illustration: Univariate Gaussian

• Likelihood function

• Conjugate prior

• Factorized variational distribution

Initial Configuration

1 2

τ

(a)

After Updating

−1 0 0 1

1 2

µ τ

(b)

After Updating

1 2

τ

(c)

Converged Solution

−1 0 0 1

1 2

µ τ

(d)

Variational Mixture of Gaussians

• Assume factorized posterior distribution

• No other approximations needed!

Variational Equations for GMM

Lower Bound for GMM

VIBES

Bishop, Spiegelhalter and Winn (2002)

ML Limit

• If instead we choose

we recover the maximum likelihood EM algorithm

Bound vs. K for Old Faithful Data

Bayesian Model Complexity

Sparse Bayes for Gaussian Mixture

• Corduneanu and Bishop (2001)

• Start with large value of K

– treat mixing coefficients as parameters – maximize marginal likelihood

– prunes out excess components

Summary: Variational Gaussian Mixtures

• Simple modification of maximum likelihood EM code

• Small computational overhead compared to EM

• No singularities

• Automatic model order selection

Continuous Latent Variables

• Conventional PCA

– data covariance matrix

– eigenvector decomposition

• Minimizes sum-of-squares projection – not a probabilistic model

– how should we choose L ?

x

x

x

x

x

x

~x

~x

u

u

Probabilistic PCA

• Tipping and Bishop (1998)

• L dimensional continuous latent space

• D dimensional data space

x

x

z



{ w

PCA

factor analysis

Probabilistic PCA

• Marginal distribution

• Advantages

– exact ML solution

– computationally efficient EM algorithm

– captures dominant correlations with few parameters – mixtures of PPCA

– Bayesian PCA

– building block for more complex models







N W

z

z

x

x

EM for PCA

0

2 (a)

EM for PCA

−2 0 2

−2 0

2 (b)

EM for PCA

0

2 (c)

EM for PCA

−2 0 2

−2 0

2 (d)

EM for PCA

0

2 (e)

EM for PCA

−2 0 2

−2 0

2 (f)

0

2 (g)

EM for PCA

Bayesian PCA

• Bishop (1998)

• Gaussian prior over columns of

• Automatic relevance determination (ARD)

ML PCA Bayesian PCA





 N W

z

z

x

x



Non-linear Manifolds

• Example: images of a rigid object

x

x

x

x

x

x

Bayesian Mixture of BPCA Models



s

s

z z

x

x





N

M W

W







Flexible Sprites

• Jojic and Frey (2001)

• Automatic decomposition of video sequence into – background model

– ordered set of masks (one per object per frame)

– foreground model (one per object per frame)

Transformed Component Analysis

• Generative model

• Now include transformations (translations)

• Extend to L layers

• Inference intractable so use variational framework

 s

s

m m

T

T

x

x

N

L

Bayesian Constellation Model

• Li, Fergus and Perona (2003)

• Object recognition from small training sets

• Variational treatment of fully Bayesian model

Bayesian Constellation Model

Summary of Part 2

• Discrete and continuous latent variables – EM algorithm

• Build complex models from simple components – represented graphically

– incorporates prior knowledge

• Variational inference

– Bayesian model comparison