Introduction to Bayesian Statistics Lecture 4: Multiparameter models (I)

Introduction to Bayesian Statistics

Lecture 4: Multiparameter models (I)

Rung-Ching Tsai

Department of Mathematics National Taiwan Normal University

March 18, 2015

Noninformative prior distributions

• Proper and improper prior distributions

• Unnormalized densities

• Uniform prior distributions on different scales

• Some examples

◦ Probability parameter θ ∈ (0, 1)

• One possibility: p(θ) = 1 [proper]

• Another possibility: p(logitθ) ∝ 1 corresponds to p(θ) ∝ θ⁻¹(1 − θ)⁻¹ [improper]

◦ Location parameter θ unconstrained

• One possibility: p(θ) ∝ 1 [improper] ⇒ p(θ|y) ≈ normal(θ|¯y ,^σ_n²)

◦ Scale parameter σ > 0

• One possibility: p(σ) ∝ 1 [improper]

• Another possibility: p(logσ²) ∝ 1 corresponds to p(σ²) ∝ σ⁻² [improper]

Noninformative prior distributions: Jeffrey’s principle

• φ = h(θ), p(φ) = p(θ)|_{d φ}^{d θ}| = p(θ) |h⁰(θ)|⁻¹

• Jeffrey’s principle leads to a non informative prior density:

p(θ) ∝ [J(θ)]^1/2, where J(θ) is the Fisher information for θ:

J(θ) = E

"

 dlogp(y |θ) d θ

2

|θ

#

= −E d²logp(y |θ) d θ² |θ



• Jeffrey’s prior model isinvariant to parameterization, evaluate J(φ) at θ = h⁻¹(φ):

J(φ) = −E d²logp(y |φ) d φ²



= −E

"

d²logp(y |θ = h⁻¹(φ)) d θ²

d θ d φ    

2#

= J(θ)    

d θ d φ    

2

;

thus, J(φ)^1/2= J(θ)^1/2|^{d θ}_{d φ}|

3 of 17

Examples: Various noninformative prior distributions

• y |θ ∼ binomial(n, θ), p(y |θ) = ⁿ_y
θ^y(1 − θ)^n−y

• Jeffrey’s prior density p(θ) ∝ [J(θ)]^1/2:

logp(y |θ) = constant + y logθ + (n − y )log(1 − θ).

J(θ) = −E d²logp(y |θ) d θ² |θ



= n

θ(1 − θ) Jeffreys⁰prior ⇒ p(θ) ∝ θ^−1/2(1 − θ)^−1/2.

• Three alternatives of prior

◦ Jeffreys’ prior: θ ∼ Beta(¹₂,¹₂)

◦ uniform prior: θ ∼ Beta(1, 1), i.e., p(θ) = 1

◦ improper prior: θ ∼ Beta(0, 0) i.e., p(logθ) ∝ 1

From single-parameter to multiparameter models

• The reality of applied statistics: there are always several (maybe many) unknown parameters!

• BUT the interest usually lies in only a few of these (parameters of interest) while others are regarded as nuisance parameters for which we have no interest in making inferences but which are required in order to construct a realistic model.

• At this point the simple conceptual framework of the Bayesian approach reveals its principal advantage over other forms of inference.

5 of 17

Bayesian approach to multiparameter models

• The Bayesian approach is clear: Obtain the joint posterior distribution of all unknowns, then integrate over the nuisance parameters to leave the marginal posterior distribution for the parameters of interest.

• Alternatively using simulation, draw samples from the entire joint posterior distribution (even this may be computationally difficult), look at the parameters of interest and ignore the rest.

Parameter of interest and nuisance parameter

• Suppose model parameter θ has two parts θ = (θ₁, θ₂)

◦ Parameter of interest: θ₁

◦ Nuisance parameter: θ₂

• For example

y |µ, σ² ∼ normal(µ, σ²),

◦ Unknown: µ and σ²

◦ Parameter of interest (usually, not always): µ

◦ Nuisance parameter: σ²

• Approach to obtain p(θ₁|y )

◦ Averaging over nuisance parameters

◦ Factoring the joint posterior

◦ A strategy for computation: Conditional simulation via Gibbs sampler

7 of 17

Posterior distribution of θ = (θ

, θ

)

• Prior of θ:

p(θ) = p(θ₁, θ₂)

• Likelihood of θ:

p(y |θ) = p(y |θ₁, θ₂)

• Posterior of θ = (θ₁, θ₂) given y :

p(θ₁, θ₂|y ) ∝ p(θ₁, θ₂)p(y |θ₁, θ₂).

Approaches to obtain marginal posterior of θ

, p(θ

|y )

• Joint posterior of θ1 and θ2: p(θ1, θ2|y ) ∝ p(θ₁, θ2)p(y |θ1, θ2)

• Approaches to obtain marginal posterior density p(θ1|y )

◦ By averaging or integrating over the nuisance parameter θ2: p(θ1|y ) =

Z

p(θ1, θ2|y )d θ2.

◦ By factoring the joint posterior:

p(θ₁|y ) = Z

p(θ₁, θ₂|y )d θ₂

= Z

p(θ₁|θ2, y )p(θ₂|y )d θ2. (1)

• p(θ1|y ) is a mixture of the conditional posterior distributions given the nuisance parameter θ2, p(θ1|θ2, y ).

• The weighting function p(θ2|y ) combines evidence from data and prior.

• θ2can be categorical (discrete) and may take only a few possible values representing, for example, different sub-models.

9 of 17

A strategy for computation: Simulations instead of integration

We rarely evaluate integral (1) explicitly, but it suggests an important strategy for constructing and computing with multiparameter models, using simulations.

• Successive conditional simulations

◦ Draw θ2from its marginal posterior distribution, p(θ2|y ).

◦ Draw θ1from conditional posterior distribution given the drawn value of θ2, p(θ1|θ2, y ).

• All-Others conditional simulations (Gibbs sampler)

◦ Draw θ^(t+1)₁ from conditional posterior distribution given the previous drawn value of θ₂^(t), p(θ1|θ^(t)₂ , y ).

◦ Draw θ^(t+1)₂ from conditional posterior distribution given the drawn value of θ^(t)₁ , p(θ₂|θ₁^(t), y ).

Multiparameter model: the normal model (I)

• y₁, · · · , y_n^iid∼ normal(µ, σ²), both µ and σ² unknown, use Bayesian approach to estimate µ.

◦ choose a prior for (µ, σ²), take noninformative priors:

p(µ, σ²) = p(µ)p(σ²) ∝ 1 · (σ²)⁻¹= σ⁻²

• priorindependenceof location and scale

• p(µ) ∝ 1: noninformative or uniform but improper prior

• p(logσ²) ∝ 1 ⇒ p(σ²) ∝ (σ²)⁻¹: noninformative or uniform on logσ²

◦ likelihood:

p(y|µ, σ²) =

n

Y

i =1

√1 2πσexp



− 1

2σ²(y_i− µ)²



∝ σ⁻ⁿexp − 1 2σ²(

n

X

i =1

(yi− µ)²

!

11 of 17

Joint posterior distribution, p(µ, σ

|y)

• y₁, · · · , y_n^iid∼ normal(µ, σ²)

◦ prior of (µ, σ²): p(µ, σ²) = p(µ)p(σ²) ∝ 1 · (σ²)⁻¹= σ⁻²

◦ find the joint posterior distribution of (µ, σ²):

p(µ, σ²|y) ∝ p(µ, σ²)p(y|µ, σ²)

∝ σ⁻ⁿ⁻²exp − 1 2σ²(

n

X

i =1

(yi− µ)²

!

= σ⁻ⁿ⁻²exp − 1 2σ²(

n

X

i =1

(y_i− ¯y )²+ n(¯y − µ)²

!

= σ⁻ⁿ⁻²exp



− 1

2σ²[(n − 1)s²+ n(¯y − µ)²]

 . where s²= _n−1¹ Pn

i =1(yi− ¯y )², the sample variance. The sufficient

Conditional posterior distribution, p(µ|σ

, y)

•

p(µ, σ²|y) = p(µ|σ², y)p(σ²|y)

• Use the case with single parameter µ with known σ² and non informative prior p(µ) ∝ 1, we have

p(µ|σ², y) ∼ normal(¯y ,σ² n ).

13 of 17

Marginal posterior distribution, p(σ

|y)

• p(µ, σ²|y) = p(µ|σ², y)p(σ²|y)

• p(σ²|y) requires averaging the joint distribution

p(µ, σ²|y) ∝ σ⁻ⁿ⁻²exp −_2σ¹2[(n − 1)s²+ n(¯y − µ)²]
over µ, that is, evaluating the simple normal integral

Z exp



− 1

2σ²n(¯y − µ)²

 d µ =

r2πσ² n , thus,

p(σ²|y) ∝ (σ²)^−(n+1)/2exp



−(n − 1)s² 2σ²



σ²|y ∼ Inv − χ²(n − 1, s²),

Analytic form of marginal posterior distribution of µ

• µ is typically the estimand of interest, so ultimate objective of the Bayesian analysis is the marginal posterior distribution of µ. This can be obtained by integrating σ² out of the joint posterior

distribution. Easily done by simulation: first draw σ² from p(σ²|y), then draw µ from p(µ|σ², y).

• The posterior distribution of µ, p(µ|y), can be thought of as a mixture of normal distributions mixed over the scaled inverse chi-squared distribution for the variance - a rare case where analytic results are available.

15 of 17

Performing the integration

• We start by integrating the joint posterior density over σ² p(µ|y) =

Z ∞ 0

p(µ, σ²|y)d σ²

• With the substitution z = _2σ^A2, A = (n − 1)s²+ n(µ − ¯y )², the result is an unnormalized gamma integral:

p(µ|y) ∝ A^−n/2 Z ∞

0

z^(n−2)/2exp(−z)dz

∝ [(n − 1)s²+ n(µ − ¯y )²]^−n/2

∝



1 +n(µ − ¯y )² (n − 1)s²

^−n/2

Parallel between Bayesian & Frequentist results

• σ²: Bayes (under noninformative prior on logσ², p(σ²) ∝ (σ²)⁻¹) versus Frequentist:

(n − 1)s²

σ² |y ∼ χ²_n−1vs. (n − 1)s²

σ² |µ, σ² ∼ χ²_n−1

• µ: Bayes (under noninformative prior on (µ, logσ²), p(µ, σ²) ∝ (σ²)⁻¹) versus Frequentist:

µ − ¯y s/√

n|y ∼ t_n−1vs. y − µ¯ s/√

n|µ, σ²∼ t_n−1.

where the ratio _s/^{¯y −µ√}_n is called a pivotal quantity : Its sampling distribution does not depend on the nuisance parameter σ².

17 of 17