9.1 Maximum Likelihood Estimation : Motivation and Examples

Chapter 9 Point Estimation

In this chapter, the basic terminology and concepts of parametric point estimation were introduced briefly. In the present chapter, we are going to elaborate extensively on hereafter weeks. The methods of estimation to be discussed here are those listed in the first section of the previous chapter; namely, maximum likelihood estimation, estimation through the concepts of unbiased and minimum variance (which lead to uniformly minimum variance estimates), estimation based on decision-theoretic concepts, and estimation by the method of moments.

9.1 Maximum Likelihood Estimation : Motivation and Examples

【Example1】

Let X , , X

₁

"

₁₀

~ (1, ) , 0 1

₁₀

iid

B θ < < , and let θ x

₁

, " , x be the respective observed values. For convenience, set . Further, suppose that in the 10 trails, 6 resulted in successes, so that

1 2 1

t = + x x + + " x

₀

t = 6 . Then the likelihood function involved is . Thus,

6 4

1 1

( | ) (1 ) , 0 1, ( , , )

L θ x = θ − θ < < θ x = x " x

₀

L ( | ) θ x is the probability of observing exactly 6 successes in 10 independent Binormial trails, the successes occurring on those trials for which x

_i

= 1 , i = 1, 2, " ,10 ; this probability is a function of the (unknown) parameter θ . Calculate the values of this probability for θ ranging from 0.1 to 0.9. We find

Value of θ Values of ( | ) L θ x

0.1 0.000006656

0.2 0.000026200

0.3 0.000175000

0.4 0.000531000

0.5 0.000976000

0.6 0.003320000

0.7 0.003010000

0.8 0.000419000

0.9 0.000053000

【Definition 1】

Let

₁

, , ~ (.; ) , , and let

iid

X " X

n

f θ θ ∈Ω x x " x be the respective observed

₁

,

₂

, ,

_n

values and x = ( , x

₁

" , x

_n

) . The likelihood function, , is given by

, and a value of θ which maximizes is called a Maximum Likelihood Estimate (MLE) of . Clearly, the MLE depends on x, and we usually write . Thus,

( | ) L θ x

( | )

ⁿ1

( ; )

_i

L ^{θ x} = ∏

i₌

f ^{x θ L θ x ( | )}

θ ˆ =

^

( )

θ θ x

{ }

( | ) ˆ max ( | ) ;

L θ x = L θ x θ ∈ Ω

【Note】概似函數(Likelihood Function)

1. 若

₁

, , ~ ( ; )

iid

X " X

n

f x θ ， 定 義 X

₁

, " , X

_n

經 抽 樣 之 後 的 聯 合 機 率 密 度 函 數

為 θ 的概似函數，記作：

1 2

1

( , , , ; ) ( ; )

n

n i

θ

i

f x x x f x

=

= ∏

" θ

i

θ

1 2 1 2

1

( ; , , , ) ( , , , ; ) ( ; )

n

n n

i

L x x x f x x x f x

=

= = ∏

θ " " θ 代表已知樣本 X

₁

, " , X

_n

屬 於 的可能性大小。 θ

2.Data X = (X

1

,X

2

,…,X

n

) where X

i

之pdf為f(x;θ)

∏

−

=

ⁿ

i

i

n

f x

x x x f

1 2

1

, ,..., ; ) ( ; )

( θ θ ^{θ 固定；x 變動}

概似函數－ L(θ; x): likelihood function θ 變動；x 固定

∏

=

=

ⁿ

i

x

i

f x

L or L

1

)

; ( )

; ( )

( θ θ θ

(一) Find the Likelihood Function

1.若 ( ; ) f x θ 中，x 的範圍與 θ 無關時，則可直接的建立概似函數

【Example 2】

(1) If

₁

, , ~ ( )

iid

X " X

n

Ber θ , please find the likelihood function of θ .

(2) If

₁

, , ~ ( )

iid

X " X

n

Poisson θ , please find the likelihood function of θ .

【sol】

2.若 ( ; ) f x θ 中，x 的範圍與 θ 有關時，則需要引進指標函數來建立概似函數

【Example 3】

(1) If

₁

, , ~ (0,

iid

X " X

n

U θ ) , please find the likelihood function of θ .

(2) If

( )

1

, , ~ ( ; ) ,

0 , /

iid x n

e x

X X f x

o w

θ

θ

θ _{= ⎨} ^⎧

^{− −}

^≥

" ⎩ , please find the likelihood function of

θ .

【sol】

(二) Discussion How to find the MLE

MLE 可依各種不同的情形，討論其求解方法 1.若概似函數可微分，則利用微分可求出 MLE

(1)以 ( ; ,

_{1 2}

, , ) 0

Let n

d L x x x

d θ

θ ^{" = ⇒ 求出} θ ^ˆ

(2)利用二階微分，確認 L ( ; , θ x x

_{1 2}

, " , x

_n

) 在

^

1 2

ˆ ( X X , , , X

_n

)

θ θ = " 達到最大

即 若

2

1 2 ˆ

2

( ; , , ,

_n

) | 0

d L x x x

d θ

^{θ θ}

θ ^"

⁼

^< ⇒ θ ^ˆ 為 θ 的 MLE.

(3)若 L ( ; , θ x x

_{1 2}

, " , x

_n

) 較難微分，則可利用 l ( ; , θ x x

_{1 2}

, " , x

_n

) = ln ( ; , L θ x x

_{1 2}

, " , x

_n

) 來處理。

【Note】對數概似函數 l ( θ ; x ) = log L ( θ ; x )

θ ^ˆ 稱 θ 之 maximum likelihood estimator (MLE) :

max ( ; ) L x L ( ; ) ˆ x

θ

θ θ

∈Ω

= 或 max ( ; ) l x l ( ; ) ^ˆ x

θ

θ θ

∈Ω

=

滿足條件: local maximum : 0 , 0

ˆ 2 2 ˆ

∂ <

= ∂

∂

∂

θ

θ

θ

θ

L L

global maximum: check boundary.

【Example 4】

(1)If

₁

, , ~ ( ; ) ! , 0,1, 2, , 0 ,

iid x

X " X

n

f x θ = θ e

^−θ

x x = " ≤ θ < ∞ zero elsewhere, please find the MLE of θ .

(2) If

₁

, , ~ ( , )

iid

X " X

n

N θ θ , please find the MLE of θ .

【sol】

【Example 5】

Let X X " X be a random sample from a population with probability density

₁

,

₂

, ,

_n

function 1

_{[0, )}

( ; ) ( )

x

f x θ e I

^θ

θ

−

=

∞

x . Moreover, X X

₁

,

₂

, " , X

_m

are observed but all we know about X

_m+1

, " , X

_n

are that they exceed τ .

(1)Write down the likelihood function for θ

(2)Show that the maximum likelihood estimator of θ is

1

( )

ˆ

m i i

X n m m

τ

θ

⁼

⎛ − ⎞

⎜ ⎟

⎝ ⎠

= ∑

【sol】

2.若概似函數不可微分，則需以其它求極值的方法處理

【Example 6】

Let

_{1 2}

1

, , , ~ ( ; ) ,

2

iid x

X X " X

n

f x θ = e

^{− −θ}

− ∞ < < ∞ − ∞ < x θ < ∞ , please find the MLE of θ .

【sol】

1

1 1

1 1

( ; ) ( ; )

2 2

n i

i i

n n x

x

i n

i i

L θ x f x θ e

^{− −θ}

e

^{−= −θ}

= =

= ∏ = ∏ = ∑

因為 max ( ; ) L x

θ

θ 與

1

min

n i i

θ

x θ

=

∑ − ^{有相同的解}

故我們分段討論如下：

(1)若 m < θ ，則

{ }

{ }

( ) ( 2 ) ( )

( ) ( ) ( )

( ) ( ) ( )

( )

i i

i

i i i

i i i

i i i

i i i

i

i i

x x

i i

x

i i i i i i

x m m x x

i

x m m x x

x m m x x

x m m x x

x m

x x m

x x m

x x m x x m x x m

m m x m

m m m

m m m

m

θ θ

θ θ

θ θ

θ θ

θ θ

θ θ

θ θ θ

θ θ θ

θ θ θ

θ

≤ < ≤ >

≤ < ≤ >

≤ < ≤ >

≤ < ≤ >

≤

− − −

= − − −

= − − − + − − − + − − −

= − + + − + −

≥ − + − + −

⎡ ⎤

= − − ⎢ − + − ⎥

⎣ ⎦

= −

∑ ∑

∑

∑ ∑ ∑

∑ ∑ ∑

∑ ∑ ∑

∑ ∑ ∑

( )

x mi

θ m

>

− −

∑ ∑

θ

若 m 為 ( , x x

_{1 2}

" , x

_n

) 的樣本中位數，則

( ) ( )

i i i i

i i

i i

x x x m x m

i i

x x

x x m m m

x x m

θ θ θ

θ

≤ >

− − − ≥ − − − ≥ 0

∴ − ≥ −

∑ ∑ ∑ ∑

∑ ∑

(2) 若 m > θ ，則

{ }

{ }

( ) (2 ) ( )

( ) ( ) ( )

( ) ( ) ( )

( )

i i

i

i i i

i i i

i i i

i i i

i

i i

x x

i i

x

i i i i i i

x x m x m

i

x x m x m

x x m x m

x x m x m

x

x x m

x x m

x x m x x m x x m

m x m m

m m m

m m m

m

θ θ

θ θ

θ θ

θ θ

θ θ

θ θ

θ θ θ

θ θ θ

θ θ θ

θ

< ≤ < ≥

< ≤ < ≥

< ≤ < ≥

< ≤ < ≥

≥

− − −

= − − −

= − − − + − − − + − − −

= − + − − + −

≥ − + − + −

⎡ ⎤

= − ⎢ − + − ⎥ + −

⎣ ⎦

= −

∑ ∑

∑

∑ ∑ ∑

∑ ∑ ∑

∑ ∑ ∑

∑ ∑ ∑

( )

m x mi

m θ

<

− −

∑ ∑

θ

若 m 為 ( , x x

_{1 2}

" , x

_n

) 的樣本中位數，則

( ) ( )

i i i i

i i

i i

x x x m x m

i i

x x

x x m m m

x x m

θ θ

θ

≥ <

− − − ≥ − − − θ ≥ 0

∴ − ≥ −

∑ ∑ ∑ ∑

∑ ∑

(3)由(1),(2)可知

當 θ ^ˆ = ( , x x

_{1 2}

, " , x

_n

) 之樣本中位數時

x

i

− θ

∑ 有最小值 (即 ( ; ) L θ x

_i

有最大值)

{ }

(( 1) 2)

( 2) (( 2) 1)

, ˆ 1

2 ,

n

n n

X if n is odd

X X if n is even

θ

⁺

+

⎧ ⎪

∴ = ⎨

⎪⎩ +

3.若概似函數為 θ 的有界函數，則利用圖形考慮求解

【Example 7】

(1)If

₁

, , ~ (0, )

iid

X " X

n

U θ , 0 ≤ x

_i

≤ θ , please find the MLE of θ .

(2)If

₁

1 1 1 1

, , ~ , , ,

2 2 2 2

iid

n i

X " X U ^⎛ ⎜ ⎝ θ − θ + ^⎞ ⎟ ⎠ θ − ≤ ≤ + x θ − ∞ < < θ ∞ , please find the MLE of θ .

(3)If , please find the

MLE of

1

, , ~ ( ,

1 2

),

1 2

,

1

,

iid

n i

X " X U θ θ θ ≤ ≤ x θ − ∞ < < ∞ − ∞ < θ θ

₂

< ∞

1 2

( , θ θ ) .

(4)If

( )

1

, , ~ ( ; ) ,

0 , /

iid x n

e x

X X f x

o w

θ

θ

θ _{= ⎨} ^⎧

^{− −}

^≥

" ⎩ , please find the MLE of θ .

【sol】

【Example 8】

Suppose X X " X is a random sample from Uniform [ ,

₁

,

₂

, ,

_n

θ θ + 1] . Let

, , and

1

min{

1

,

2

, ,

_n

}

Y = X X " X Y

_n

= max{ X X

₁

,

₂

, " , X

_n

} R Y =

_n

− , please find Y

₁

the MLE of θ .

【sol】

4.若概似函數中 θ 有範圍限制，則需先求出 θ 的 global maximum，再以討論的方 法求解 θ 的 MLE

【Example 9】

If

₁

, , ~ ( ) , 1 2 1

iid

X " X

n

Ber θ ≤ ≤ θ , please find the m.l.e of θ .

【sol】

5.在離散型單一樣本的例子中，或機率密度函數型態不同時，則需就每個 x 樣本 點逐點來討論。

【Example 10】

Let X be a single observation from the following distribution ( ; ) f x θ .

x 1 2 4

Pr( X = x ) θ 0.4+ θ 0.6- 2 θ

where ^{θ ∈} [ ^{0 , 0.3} ] ^{. If}

1

,

2

~ ( ; ⁾

X X

iid

f x θ , please find the m.l.e of θ .

【sol】

6.若 L θ ( ) 可微，其中 θ = ( , θ θ

_{1 2}

, " , θ

_r

) ，則可分別對 θ θ

₁

,

₂

, " , θ

_r

做偏微分求解 θ 之 MLE

解題步驟：

(1)令 ( ; ) 0

i

θ L

∂ =

∂ θ x 或 ln ( ; ) 0

i

θ L

∂ =

∂ θ x 求解 θ ˆ ,

_i

i = 1, 2, " , r

(2)檢驗

2

2

( ; ) 0

i

θ L

∂ <

∂ θ x 或

2

2

ln ( ; ) 0

i

θ L

∂ <

∂ θ x

(3)檢驗 Hessian 矩陣

2

ˆ

( ; )

i j

L θ θ

θ

⎡ ∂ ⎤

⎢ ∂ ∂ ⎥

⎢ ⎥

⎣ θ x ⎦ 或

2

ˆ

ln ( ; )

i j

L θ θ

θ

⎡ ∂ ⎤

⎢ ∂ ∂ ⎥

⎢ ⎥

⎣ θ x ⎦ 之行列式值是否大於

若(2)成立且(3) Hessian 矩陣之行列式值大於 0，則 θ ˆ ,

_i

i = 1, 2, " , r 為 θ

_i

的最大概似 估計量

【Example 11】

If X X

₁

,

₂

, " , X

_n

~

iid

1 2 1 2

( , ), , 0

N θ θ − ∞ < < ∞ θ < θ < ∞ , please find the MLE of

1 2

( , θ θ ) .

【sol】

7.若偏微分無 closed form 時，則可利用數值分析的方法求解參數的 MLE.

【Example 12】

Let X

₁

, X

₂

,..., X

_n

~

iid

gamma ( θ

₁

, θ

₂

) , θ

₁

> 0 , θ

₂

> 0 , please find maximum likelihood estimator of ( , θ θ

_{1 2}

) .

【sol】 ( ) _{( )}

₁

⁽

^{1 2}

⁾

^{1 1 2}

2

~ 1 2

1

1 ...

,

,

_θ ^{θ θ}

θ θ θ

θ

_n ^xi

n

e x x x x

L ⎥

^{− −Σ}

⎦

⎢ ⎤

⎣

⎡

= Γ

( ) _{= −} [ _Γ ( ) ₊ ] ₊ ( ₋ ) ∑ ₋ ∑

2 1

2 1 1 2

1

, log log 1 log

log L θ θ n θ θ θ θ x

_i

θ ^x

ⁱ

( ) ( )

* log 0

0 log log log

2 2 2

1 2

2 1

1 1

1

⎪ ⎪

⎭

⎪⎪ ⎬

⎫

= +

−

∂ =

∂

= +

∂ − Γ

∂

− Γ

∂ =

∂

∑

∑

θ θ

θ θ

θ θ θ θ

θ

i

i

n x

x n n

L

Î No closed form to solve equation *，we can use numerical method to solve

equation.

【Example 13】

Let X and

_i

C

_i

, i = 1, 2, " n , , be two independent random sequences, where X

_i

is from an exponential distribution with a scale parameter λ

₁

and s from an exponential distribution with a scale parameter

C

i

λ

2

. Consider a sequence of pairs

( Z Y

i

^,

i

) , i = 1, 2, " n , , where

,

i i

, 1 ,

, 0

i i i i

i i

i i i i

X if X C if X C

Z and Y

C if C X if C X

< <

⎧ ⎧

= ⎨ ⎩ ≤ = ⎨ ⎩ ≤ , please answers the follows

questions

(1) Suppose we observe X , 1, 2, ,

_i

i = " . Find the MLE of n λ

₁

.

(2) Find the joint probability density function of Z and Y .

(3) Suppose we observe ( Z Y

i

^,

i

) , 1, 2, i = " . Find the MLE of , n λ

₁

and λ

₂

.

【sol】