Large Sample Theory
Homework 5: Maximum Likelihood Estimate, Testing, Asymptotic Distribution Due Date: January 12th
1. Consider the classical Gaussian linear model Yi = µi + i, 1 ≤ i ≤ n, where µi = zTi β and iare i.i.d. Gaussian with mean 0 and variance σ2. Here ziare d-dimensional vectors for covariate values. Suppose that the covariates are ranked in order of importance. (It means that the first covariate is the most important and etc.)
To entertain the possibility that the last d − p don’t matter, βp+1 = · · · = βd = 0. Let βˆ(p) be the least-squares estimate with βp+1= · · · = βd = 0 and ˆYi(p)the corresponding fitted value.
In this fashion, we end up d possible regression models. Now the problem is which one to use. A natural goal to entertain is to obtain new values Y1∗, . . . , Yn∗ at z1, . . . , zn and evaluate the performance of ˆY1(p), . . . , ˆYn(p) as estimates of Y1∗, . . . , Yn∗ and, hence, the model with βd+1 = · · · = βp = 0 by the (average) expected prediction error
EP E(p) = n−1E
n
X
i=1
(Yi∗− ˆYi(p))2.
Here Y1∗, . . . , Yn∗are independent of Y1, . . . , Ynand Yi∗is distributed as Yi, i = 1, . . . , n.
Let RSS(p) = Pni=1(Yi − ˆYi(p))2 be the residual sum of squares. Suppose that σ2 is known.
(a.) Show that
EP E(p) = σ2
1 + p n
+ 1 n
n
X
i=1
(µi− µ(p)i )2
where µ(p)i = zTi βˆ(p) and ˆβ(p) = (β1, . . . , βp, 0, . . . , 0)T. (b). Show that
E[RSS(p)] = σ2
1 − p n
+ 1 n
n
X
i=1
(µi− µ(p)i )2. (c). Show that RSS(p) + (2p/n)σ2 is an unbiased estimate of EP E(p).
(d). Mallow (1973, Technometrics) suggested a model selection rule in which p is se- lected to be the one minimizes RSS(p) + (2p/n)σ2 and then using ˆY(ˆp) as a predictor. Suppose p = 2 and d = 3. Find the probability that P (ˆp = 3) and P (ˆp ≤ 1) when n goes to infinity. (You can assume that those covariates are realized values of 3 independent U N IF (0, 1) random variables. For example, µi = β1zi1 + β2zi2 + β3zi3 where zi1, zi2, and zi3 are independent U N IF (0, 1) random variables.
2. Consider model Y = Xβ + where E() = 0 and V ar() = σ2Jn. Let ˆYi = Xiβ andˆ hii= Xi(XTX)−1XTi .
(a) Show that for any > 0,
P (| ˆYi− E( ˆYi)| ≥ ) ≥ min[P (i ≥ /hii), [P (i ≤ −/hii)].
(Hint: for independent random variables X and Y , P (|X +Y | ≥ ) ≥ P (X ≥ )P (Y ≥ 0) + P (X ≤ −)P (Y < 0).)
(b) Show that ˆYi− E( ˆYi)→ 0 if and only if hP ii→ 0.
1
3. Let (Xi, Yi), 1 ≤ i ≤ n, be iid with Xi and Yi independent, N (θ1, 1), N (θ2, 1), respec- tively. Suppose θ1 ≥ 0 and θ1 ≥ θ2 ≥ 0. Consider testing H0 : θ1 = θ2 = 0 versus H1 : θ1 > 0 or θ2 > 0. Show that whatever be n, under H0, λnis distributed as a mixture of point mass at 0, χ21 and χ22 with probabilities 3/8, 1/2, 1/8, respectively.
4. Let (X11, X12), . . . , (Xn1, Xn2) be i.i.d. from a bivariate normal distribution with un- known mean and covariance matrix. For testing H0 : ρ = 0 versus H1 : ρ 6= 0, where ρ is the correlation coefficient, show that the test rejecting H0 when |W | > 0 is an LR test, where
W =
n
X
i=1
(Xi1− ¯X1)(Xi2− ¯X2)/
" n X
i=1
(Xi1− ¯X1)2+
n
X
i=1
(Xi2− ¯X2)2
#
.
Find the distribution of W under H0.
5. Suppose you are studying the number of visitations of a pollinator to a flower. Your hypothesis is that yellow flowers are better than red flowers (in terms of pollinator at- traction). Previous studies have found that the number of visitors to red flowers follows a normal distribution with a mean of 200 visits per flower and a variance of 50. Suppose in a sample of 20 yellow flowers that the mean number of visits is 202 with a known variance (of visits per flower) of 50. Again, assume the number of visitors is normally distributed.
a. What is the probability of this data under the null hypothesis (yellow and red flowers are equivalent)?
b. What is the critical value for a (one-sided) test of the null hypothesis at the α = 0.05 level?
c. What are the values for (a) and (b) when the variance for yellow flowers (50) is instead a SAMPLE variance (i.e., an estimate of the true variance)? Hint: Would you now use a normal or a t distribution?
d. Suppose that yellow flowers are indeed better. Given the sample size (20) and as- suming the variance (50) is the true value, how small an effect can we detect using a (one-sided) test of significance of α = 0.05 with 80% power?
e. Repeat the calculation in (d) assuming that the variance (50) is now an estimated value, not necessarily the true value.
f. Suppose the true mean and variance for yellow flowers are 201 and 10. How large a sample size is required to have a power of 80 percent of detecting a difference be- tween red and yellow using a test of significance with level α = 0.05? Compute this for both the normal (variance assumed know) and t (variance estimated) settings.
g. If the true variance for yellow is 35, what is the probability that we observe a sample variance of 50 (or larger) given our sample size of 20.
6. Let X1, X2, . . . , Xnbe a random sample from the unif (0, θ) distribution for some θ >
0. Suppose we wish to test
H0 : θ = θ0 versus Ha: θ < θ0 at level (size) α. Suppose that we use test statistic X(n).
a. Derive the test with the probability of a Type I error α.
2
b. What is the probability of a type Type II error for any particular θ = θ1where θ1 is some fixed number less than θ0?
c. What is the power function of this test?
d. What sample size is necessary in order to get β(θ1) = β where β is a fixed number between 0 and 1 and θ1 is a fixed value between 0 and θ0?
7. Let X1, . . . , Xn be the times in months until failure of n similar pieces of equipment.
Since the equipment is subject to wear, we often model X1, . . . , Xnas a random sample of size n from a Weibull distribution with density f (x, λ) = λcxc−1exp(−λxc), x > 0.
Here c is a known positive constant and λ > 0.
a. Find an optimal test for testing H0 : 1/λ ≤ 1/λ0versus Ha: 1/λ > 1/λ0.
b. Suppose that the only table you have is a normal probability table. Can you use this table to carry out the test derived in (a)? Give reasons to justify your answer.
8. Let Xn be a random variable having the Poisson distribution P (nθ), where θ > 0, n = 1, 2, . . .. Show that (Xn− nθ)/√
nθ→ N (0, 1).d
9. Let U1, . . . , Un be i.i.d. random variables having the uniform distribution on [0, 1] and Yn= (Qni=1Ui)−1/n. Show that√
n(Yn− e)→ N (0, ed 2).
10. Set ˆσ =qn−1Pni=1(Xi− ¯X)2. Show that√
n(ˆσ − σ)→ N (0, σd 2/2).
11. Let X1, . . . , Xnbe i.i.d. N (θ, 1) with θ ≥ 0.
(a) Show that the MLE of θ, ˆθn, is ¯X if ¯X > 0 and 0 otherwise.
(b) If θ > 0, show that√
n(ˆθn− θ)→ N (0, 1).L
(c) If θ = 0, the probability is 1/2 that ˆθn = 0 and 1/2 that√
n(ˆθn− θ)→ N (0, 1).L 12. If X1, . . . , Xnare i.i.d. according to U (0, θ) and Tn = X(n), the limiting distribution of
n(θ − Tn) is exponential with density θ−1exp(−x/θ). Use this result to determine the limit distribution of
(a) n[f (θ) − f (Tn)], where f is any function with f(1)(θ) 6= 0;
(b) [f (θ) − f (Tn)] is suitably normalized if f(1)(θ) = 0 but f(2)(θ) 6= 0.
13. Let X1, . . . , Xnbe i.i.d. N (θ, σ2) and consider the estimation of θ2. (a) Find the maximum likelihood estimator.
(b) Obtain the limit distribution of the estimators obtained in (a). (Hint: You may need to consider θ 6= 0 and θ = 0 separately.)
14. Let X1, . . . , Xn be i.i.d. with E(Xi) = θ, V ar(Xi) = σ2 < ∞, and let δn = ¯X with probability 1 − nand δn= Anwith probability n. If nand Anare constants satisfying
n→ 0 and nAn→ ∞,
then δnis consistent for estimating θ, but E(δn− θ)2 does not tend to zero.
15. Suppose that Xnis a random variable having the binomial distribution Bin(n, p), where 0 < p < 1, n = 1, 2, . . .. Define
Yn =
( log(Xn/n) Xn ≥ 1
1 Xn = 0.
Show that Yn a.s.→ log p and√
n(Yn− log p)→ N (0, (1 − p)/p).d 3
16. Let X1, . . . , Xnbe iid random variables with V ar(X1) < ∞. Show that 2
n(n + 1)
n
X
j=1
jXj
→ EXP 1.
17. Let (Y1, Z1), . . . , (Yn, Zn) be i.i.d. with the Lebesgue pdf λ−1µ−1e−y/λe−z/µI(0,∞)(y)I(0,∞)(z), where λ > 0 and µ > 0.
(a) Find the MLE of (λ, µ).
(b) Suppose that we only observe Xi = min(Yi, Zi) and δi = 1 if Xi = Yi and δi = 0 if Xi = Zi. Find the MLE of (λ, µ).
18. Let X be N (0, θ), 0 < θ < ∞. Find the Fisher information I(θ).
4
