Principle of Data Reduction
6.1 Introduction
An experimenter uses the information in a sample X1, . . . , Xnto make inferences about an unknown parameter θ. If the sample size n is large, then the observed sample x1, . . . , xn is a long list of numbers that may be hard to interpret. Any statistic, T (X), defines a form of data reduction or data summary. For example, the sample mean, the sample variance, the largest observation, and the smallest observation are four statistics that might be used to summarize some key features of the sample.
The statistic summarizes the data in that, rather than reporting the entire sample x, it reports only that T (x) = t. For example, two samples x and y will be treated as equal, if T (x) = T (y) is satisfied. The advantages and consequences of this type of data reduction are the topics of this chapter.
We study three principles of data reduction.
1. The Sufficiency Principle promotes a method of data reduction that does not discard information about θ while achieving some summarization of the data.
2. The Likelihood Principle describes a function of the parameter determined by the observed sample, that contains all the information about θ that is available from the sample.
3. The Equivariance Principle prescribes yet another method of data reduction that still preserves some important features of the model.
11
6.2 The Sufficiency Principle
Sufficiency Principle: If T (X) is a sufficient statistic for θ, then any inference about θ should depend on the sample X only through the value T (X). That is, if x and y are two sample points such that T (x) = T (y), then the inference about θ should be the same whether X = x or Y = y is observed.
6.2.1 Sufficient Statistics
Definition 6.2.1 A statistic T (X) is a sufficient statistic for θ if the conditional distribution of the sample X given the value of T (X) does not depend on θ.
To use this definition to verify that a statistic T (X) is a sufficient statistic for θ, we must verify that P (X = x|T (X) = T (x)) does not depend on θ. Since {X = x} is a subset of {T (X) = T (x)},
Pθ(X = x|T (X) = T (x)) = Pθ(X = x and T (X) = T (x)) Pθ(T (X) = T (x))
= Pθ(X = x)
Pθ(T (X) = T (x)) = p(x|θ) q(T (x))|θ),
where p(x|θ) is the joint pmf/pdf of the sample X and q(t|θ) is the pmf/pdf of T (X). Thus, T (X) is a sufficient statistic for θ if and only if, for every x, the above ratio is constant as a function of θ.
Theorem 6.2.1 If p(x|θ) is the joint pdf or pmf of X and q(t|θ) is the pdf or pmf of T (X), then T (X) is a sufficient statistic for θ if, for every x in the sample space, the ratio p(x|θ)/q(T (x)|θ) is constant as a function of θ.
Example 6.2.1 (Binomial sufficient statistic) Let X1, . . . , Xn be iid Bernoulli random vari- ables with parameter θ, 0 < θ < 1. Then T (X) = X1+ · · · + Xn is a sufficient statistic for θ. Note that T (X) counts the number of Xi’s that equal 1, so T (X) ∼ Bi(n, θ). The ratio of pmfs is thus
p(x|θ) q(T (x)|θ) =
Qθxi(1 − θ)1−xi
¡n
t
¢θt(1 − θ)n−t (define t =Pn
i=1xi)
= θPxi(1 − θ)P(1−xi)
¡n
t
¢θt(1 − θ)n−t
= 1
¡n
t
¢ = 1
¡ n
Pxi
¢
Since this ratio does not depend on θ, by Theorem 6.2.1, T (x) is a sufficient statistic for θ.
Example 6.2.2 (Normal sufficient statistic) Let X1, . . . , Xnbe iid N (µ, σ2), where σ2is known.
We wish to show that the sample mean, T (X) = ¯X, is a sufficient statistic for µ.
f (x|µ) = Yn i=1
(2πσ2)−1/2exp(−(xi− µ)2/(2σ2))
= (2πσ2)−n/2exp{−[
Xn i=1
(xi− ¯x)2+ n(¯x − µ)2]/(2σ2)}
Recall that the sample mean ¯X ∼ N (µ, σ2/n). Thus, the ratio of pdf is p(x|θ)
q(T (x))|θ) = (2πσ2)−n/2exp{−[Pn
i=1(xi− ¯x)2+ n(¯x − µ)2]/(2σ2)}
(2πσ2/n)−1/2exp{−n(¯x − µ)2/(2σ2)}
= n−1/2(2πσ2)−(n−1)/2exp{−
Xn i=1
(xi− ¯x)2/(2σ2)},
which does not depend on µ. By Theorem 6.2.1, the sample mean is a sufficient statistic for µ.
Example 6.2.3 (Sufficient order statistics) Let X1, . . . , Xn be iid from a pdf f , where we are unable to specify any more information about the pdf. It then follows that
fX(1),...,X(n)(x) =
n!Qn
i=1fX(xi), if x1 < . . . < xn
0 otherwise
where (x(1), . . . , x(n) is the order statistic. By Theorem 6.2.1, the order statistic is a sufficient statistic.
Of course, this is not much of a reduction, but we should not expect more with so little information about the density f . However, even if we specify more about the density, we still may not be able to get much of a sufficient reduction. For example, suppose that f is the Cauchy pdf f (x|θ) = π(x−θ)1 2 or the logistic pdf f (x|θ) = (1+ee−(x−θ)−(x−θ))2. It turns out that outside of the exponential family of distributions, it is rare to have a sufficient statistic of smaller dimension than the size of the sample, so in many cases it will turn out that the order statistics are the best that we can do.
Theorem 6.2.2 (Factorization Theorem) Let f (x|θ) denote the joint pdf or pmf of a sample X. A statistic T (X) is a sufficient statistic for θ if and only if there exist functions g(t|θ) and h(x) such that, for all sample points x and all parameter points θ,
f (x|θ) = g(T (x)|θ)h(x). (6.1)
Proof: We give the proof only for discrete distributions. Suppose T (X) is a sufficient statistic.
Choose g(t|θ) = Pθ(T (X) = t) and h(x) = P (X = x|T (X) = T (x)). Because T (X) is sufficient, the conditional probability h(x) does not depend on θ. Thus,
f (x|θ) = Pθ(X = x)
= Pθ(X = x and T (X) = T (x))
= Pθ(T (X) = T (x))P (X = x|T (X) = T (x))
= g(T (x)|θ)h(x)
So factorization (6.1) has been exhibited. Also, the last two lines above imply that g(T (x)|θ) is the pmf of T (X).
Now assume the factorization (6.1) exists. Let q(t|θ) be the pmf of T (X). Define AT (x)= {y : T (y) = T (x)}. Then
f (x|θ)
q(T (x)|θ) = g(T (x)|θ)h(x) q(T (x)|θ)
= g(T (x)|θ)h(x) P
AT (x)g(T (y)|θ)h(y) (density transformation)
= g(T (x)|θ)h(x) g(T (x)|θ)P
AT (x)h(y) (since T is a constant on AT (x))
= h(x)
P
AT (x)h(y)
Since the ratio does not depend on θ, by Theorem 6.2.1, T (X) is a sufficient statistic for θ. ¤
To use the Factorization Theorem to find a sufficient statistic, we factor the joint pdf of the sample into two parts. One part does not depend on θ and it constitutes the h(x) function. The other part depends on θ, and usually it depends on the sample x only through some function T (x) and this function is a sufficient statistic for θ.
Example 6.2.4 (Normal sufficient statistic) Let X1, . . . , Xnbe iid N (µ, σ2), where σ2is known.
The pdf can be factored as
f (x|µ) = (2πσ2)−n/2exp{−
Xn i=1
(xi− ¯x)2/(2σ2)} exp{−n(¯x − µ)2/(2σ2)}.
We can define
g(t|θ) = exp{−n(t − µ)2/(2σ2)}
by defining T (x) = ¯x, and
h(x) = (2πσ2)−n/2exp{−
Xn i=1
(xi− ¯x)2/(2σ2)}.
Thus, by the Factorization Theorem, T (X) = ¯X is a sufficient statistic for µ.
Example 6.2.5 (Uniform sufficient statistic) Let X1, . . . , Xn be iid observations from the dis- crete uniform distribution on 1, . . . , θ. That is, the unknown parameter, θ, is a positive integer and the pmf of Xi is
f (x|θ) =
1θ x = 1, 2, . . . , θ 0 otherwise.
Thus, the joint pmf of X1, . . . , Xn is
f (x|θ) =
θ−n xi ∈ {1, . . . , θ} for i = 1, . . . , n 0 otherwise.
Let IA(x) be the indicator function of the set A; that is, it is equal to 1 if x ∈ A and equal to 0 otherwise. Let N = {1, 2, . . .} be the set of positive integers and let Nθ= {1, 2, . . . , θ}. Then the joint pmf of X1, . . . , Xn is
f (x|θ) = Yn i=1
θ−1INθ(xi) = θ−n Yn i=1
INθ. Defining T (x) = maxixi, we see that
Yn i=1
INθ = ( Yn i=1
IN(xi))INθ(T (x)).
Thus we have the factorization
f (x|θ) = θ−nINθ(T (x))(
Yn i=1
IN(xi)).
By the factorization theorem, T (X) = maxiXi is a sufficient statistic for θ.
Example 6.2.6 (Normal sufficient statistic, both parameters unknown) Assume that X1, . . . , Xn are iid N (µ, σ2), and that both µ and σ2 are unknown so the parameter vector is θ = (µ, σ2). Let T1(x) = ¯x and T2(x) = s2 =Pn
i=1(xi− ¯x)2/(n − 1).
f (x|θ) =(2πσ2)−n/2exp{−[
Xn i=1
(xi− ¯x)2+ n(¯x − µ)2]/(2σ2)}
= (2πσ2)−n/2exp{−(n(t1− µ)2+ (n − 1)t2)/(2σ2)}.
Let h(x) = 1. By the factorization theorem, T (X) = (T1(X), T2(X)) = ( ¯X, S2) is a sufficient statistic for (µ, σ2).
The results can be generalized to the exponential family of distributions.
Theorem 6.2.3 Let X1, . . . , Xn be iid observations from a pdf or pmf f (x|θ) that belongs to an exponential family given by
f (x|θ) = h(x)c(θ) exp{
Xk i=1
wi(θ)ti(x)}
where θ = (θ1, . . . , θd), d ≤ k. Then T (X) = (
Xn j=1
t1(Xj), . . . , Xn j=1
tk(Xj))
is a sufficient statistic for θ.
6.2.2 Minimal Sufficient Statistics
In any problem, there are many sufficient statistics. For example,
1. It is always true that the complete sample, X, is a sufficient statistic.
2. Any one-to-one function of a sufficient statistic is also a sufficient statistic.
Recall that the purpose of a sufficient statistic is to achieve data reduction without loss of infor- mation about the parameter θ; thus, a statistic that achieves the most data reduction while still retaining all the information about θ might be considered preferable.
Definition 6.2.2 A sufficient statistic T (X) is called a minimal sufficient statistic if, for any other sufficient statistic T0(X), T (x) is a function of T0(x).
To say that T (x) is a function of T0(x) simply means that if T0(x) = T0(y), then T (x) = T (y).
Let T = {t : t = T (x) for some x ∈ X } be the image of X under T (x). Then T x partitions the sample space into sets At, t ∈ T , defined by At= {x : T (x) = t}. If {Bt0 : t ∈ T0} are the partition sets for T0(x) and {At : t ∈ T } are the partition sets for T (x), then Definition 6.2.2 states that every Bt0 is a subset of some At. Thus, the partition associates with a minimal sufficient statistic, is the coarsest possible partition for a sufficient statistic, and a minimal sufficient statistic achieves the greatest possible data reduction for a sufficient statistic.
Example 6.2.7 (Two Normal sufficient statistics) Suppose that X1, . . . , Xn are observations from N (µ, σ2) with σ2 known. We know that T (X) = ¯X is a sufficient statistic for µ. The factorization theorem shows that T0(X) = ( ¯X, S2) is also a sufficient statistic for µ. T (X) can be written as a function of T0(X) by defining the function r(a, b) = a. Then T (x) = ¯x = r(¯x, S2) = r(T0(x)).
Theorem 6.2.4 let f (x|θ) be the pmf or pdf of a sample X. Suppose there exists a function T (x) such that, for every two sample points x and y, the ratio f (x|θ)/f (y|θ) is a constant as a function of θ if and only if T (x) = T (y). Then T (x) is a minimal sufficient statistic for θ.
Proof: To simply the proof, we assume f (x|θ) > 0 for all x ∈ X and θ. First we show that T (X) is a sufficient statistic. Let T = {t : t = T (x) for some x ∈ X } be the image of X under T (x).
Define the partition sets induced by T (x) as At= {x : T (x) = t}. For each At, choose and fix one element xt∈ At. For any x ∈ T , xT (x) is the fixed element that is in the same set, At, as x. Since x and xT (x) are in the same set At, T (x) = T (xT (x)) and, hence, f (x|θ)/f (xT x|θ) is constant as a function of θ. Thus, we can define a function on X by h(x) = f (x|θ)/f (xT x|θ) and h does not depend on θ. Define a function on T by g(t|θ) = f (xt|θ). Then it can be seen that
f (x|θ) = f (xT x|θ)f (x|θ)
f (xT x|θ) = g(T (x)|θ)h(x) and, by the factorization theorem, T (X) is a sufficient statistic for θ.
Now to show that T X is minimal, let T0(X) be any other sufficient statistic. By the factoriza- tion theorem, there exist function g0 and h0 such that f (x|θ) = g0(T0(x)|θ)h0(x). Let x and y be any two sample points with T0(x) = T0(y). Then
f (x|θ)
f (y|θ) = g0(T0(x)|θ)h0(x)
g0(T0(y)|θ)h0(y) = h0(x) h0(y).
Since this ratio does not depend on θ, the assumptions of the theorem imply that T (x) = T (y).
Thus, T (x) is a function of T0(x) and T (x) is minimal. ¤
Example 6.2.8 (Normal minimal sufficient statistic) Let X1, . . . , Xn be iid N (µ, σ2), both µ and σ2 unknown. Let x and y denote two sample points, and let (¯x, Sx) and (¯2 y, Sy) be the sample2 means and variances corresponding to the x and y samples, respectively. Then the ratio of densities is
f (x|µ, σ2)
f (y|µ, σ2) = exp{[−n(¯x2− ¯y)2+ 2nµ(¯x − ¯y) − (n − 1)(s2x − s2y)]/(2σ2)}.
This ratio will be constant as a function of µ and σ2 if and only ¯x = ¯y and s2x = s2y. Thus, by Theorem 6.2.4, ( ¯X, S2) is a minimal sufficient statistic for (µ, σ2).
Example 6.2.9 (Uniform minimal sufficient statistic) Suppose X1, . . . , Xn are iid uniform observations on the interval (θ, θ + 1), −∞ < θ < ∞. Then the joint pdf of X is
f (x|θ) =
1 θ < xi < θ + 1, i = 1, . . . , n, 0 otherwise,
which can be written as
f (x|θ) =
1 maxixi− 1 < θ < minixi 0 otherwise
Thus, for two sample points x and y, the numerator and denominator of the ratio f (x|θ)/f (y|θ) will be positive for the same values of θ if and only if minixi = miniyi and maxixi = maxiyi. Thus, we have that T (X) = (X(1), X(n)) is a minimal sufficient statistic.
This example shows the dimension of a minimal sufficient statistic may not match the dimension of the parameter. A minimal sufficient statistic is not unique. Any one-to-one function of a minimal sufficient statistic is also a minimal sufficient statistic. For example, T0(X) = (X(n)− X(1), (X(n)+ X(1))/2) is also a minimal sufficient statistic in Example 6.2.9, and T0(X) = (Pn
i=1Xi,Pn
i=1Xi2) is also a minimal sufficient statistic in Example 6.2.8.
6.2.3 Ancillary Statistics
Definition 6.2.3 A statistic S(X) whose distribution does not depend on the parameter θ is called an ancillary statistic.
Alone, an ancillary statistic contains no information about θ. An ancillary statistic is an obser- vation on a random variable whose distribution is fixed and known, unrelated to θ. Paradoxically, an ancillary statistic, when used in conjunction with other statistics, sometimes does contain valuable information for inferences about θ.
Example 6.2.10 (Uniform ancillary statistic) Let X1, . . . , Xn be iid uniform observations on the interval (θ, θ + 1), −∞ < θ < ∞. The range statistic R = X(n)− X(1) is an ancillary statistic.
Recall that the cdf of each Xi is
F (x|θ) =
0 x ≤ θ
x − θ θ < x < θ + 1 1 θ + 1 ≤ x.
Thus, the joint pdf of X(1) and X(n) is
g(x(1), x(n)|θ) =
n(n − 1)(x(n)− x(1))n−2 θ < x(1) < x(n)< θ + 1
0 otherwise
Making the transformation R = X(n)− X(1) and M = (X(n)+ X(1))/2, we see the joint pdf of R and M is
h(r, m|θ) =
n(n − 1)rn−2 0 < r < 1, θ + r/2 < m < θ + 1 − r/2
0 otherwise
Thus, the pdf for R is h(r|θ) =
Z θ+1−r/2
θ+r/2
n(n − 1)rn−2dm = n(n − 1)rn−2(1 − r), 0 < r < 1.
This is a beta pdf with α = n − 1 and β = 2. Thus, the distribution of R does not depend on θ, and R is ancillary.
Example 6.2.11 (Location and scale family ancillary statistic) Let X1, . . . , Xn be iid ob- servations from a location family with cdf F (x − θ), −∞ < θ < ∞. Show that the range statistic R = X(n)− X(1) is an ancillary statistic.
Let X1, . . . , Xn be iid observations from a scale family with cdf F (x/σ), σ > 0. Show that any statistic that depends on the sample only through the n − 1 values X1/Xn, . . . , Xn−1/Xn is an ancillary statistic.
6.2.4 Sufficient, Ancillary, and Complete Statistics
A minimal sufficient statistic is a statistic that has achieved the maximal amount of data reduction possible while still retaining all the information about the parameter θ. Intuitively, a minimal sufficient statistic eliminates all the extraneous information in the sample, retaining only that piece with information about θ. Since the distribution of an ancillary statistic does not depend on θ, it
might be suspected that a minimal sufficient statistic is unrelated to an ancillary statistic. However, this is not necessarily the case. Recall Example 6.2.9 in which X1, . . . , Xn were iid obs from a uniform(θ, θ + 1) distribution. We have pointed out that the statistic (X(n)− X(1), (X(n)+ X(1))/2) is a minimal sufficient statistic, and in Example 6.2.10 we showed that X(n)− X(1) is an ancillary statistic. Thus, in this case, the ancillary statistic is an important component of the minimal sufficient statistic. Certainly, the ancillary statistic and the minimal sufficient statistic are not independent.
The following example shows that an ancillary statistic can sometimes give important informa- tion for inference about θ.
Example 6.2.12 (Ancillary precision) Let X1 and X2 be iid obs from the discrete distribution that satisfies
P (X = θ) = P (X = θ + 1) = P (θ + 2) = 1/3,
where θ, the unknown parameter, is any integer. It can be shown with an argument similar to that in Example 6.2.9 that (R, M ), where R = X(n)− X(1) and M = (X(n)+ X(1))/2, is a minimal sufficient statistic. To see how R might give information about θ, consider a sample point (r, m), where m is an integer. First we consider only m, θ must be one of three values, either θ = m or θ = m − 1 or θ = m − 2. With only the information that M = m, all three θ values are possible values. But now suppose we get the information that R = 2. Then it must be the case that X(1) = m − 1 and X(2) = m + 1, and the only possible value for θ is θ = m − 1. Thus, the knowledge of the value of the ancillary statistic R has increased our knowledge about θ. Of course, the knowledge of R alone would give us no information about θ.
For many important situations, however, our intuition that a minimal sufficient statistic is independent of any ancillary statistic is correct. A description of situations in which this occurs relies on the next definition.
Definition 6.2.4 Let f (t|θ) be a family of pdfs or pmfs for a statistic T (X). The family of probability distributions is called complete if Eθg(T ) = 0 for all θ implies Pθ(g(T ) = 0) = 1 for all θ. Equivalently, T (X) is called a complete statistic.
Example 6.2.13 (Binomial complete sufficient statistic) Suppose that T has a binomial (n, p) distribution, 0 < p < 1. Let g be a function such that Epg(T ) = 0. Then
0 = Epg(T ) = Xn
t=0
g(T ) µn
t
¶
pt(1 − p)n−t = (1 − p)n Xn
t=0
g(t) µn
t
¶¡ p 1 − p
¢t
for all p, 0 < p < 1. Let r = p/(1 − p), 0 < r < ∞. The last expression is a polynomial of degree n in r, where the coefficient of rt is g(t)¡n
t
¢. For the polynomial to be 0 for all r, each coefficient must be 0. Thus, g(t) = 0 for t = 0, 1, . . . , n. Since T takes on the values 0, 1 . . . , n with probability 1, this yields that Pp(g(T ) = 0) = 1 for all p. Hence, T is a complete statistic.
Example 6.2.14 Let X1, . . . , Xn be iid uniform (0, θ) observations, 0 < θ < ∞. Show that T (X) = maxiXi is a complete statistic.
Using an argument similar to that used in one of previous examples, we can see that T (X) = maxiXi is a sufficient statistic and the pdf of T (X) is
f (t|θ) =
ntn−1θ−n 0 < t < θ
0 otherwise.
Suppose g(t) is a function satisfying Eθg(T ) = 0 for all θ. Since Eθg(T ) is a constant of θ, its derivative with respect to θ is 0. Thus we have that
0 = d
dθEθg(T ) = d dθ
Z θ
0
g(t)ntn−1θ−ndt
= (θ−n) d dθ
Z θ
0
ng(t)tn−1dt + ( d dθθ−n)
Z θ
0
ng(t)tn−1dt
= θ−nng(θ)θn−1+ 0
= θ−1ng(θ)
Since θ−1n 6= 0, it must be g(θ) = 0. This is true for every θ > 0, hence, T is a complete statistic.
Theorem 6.2.5 (Basu’s Theorem) If T (X) is a complete and minimal sufficient statistic, then T (X) is independent of every ancillary statistic.
Proof: We give the proof only for discrete distributions. Let S(X) be any ancillary statistic.
Then P (S(X) = s) does not depend on θ since S(X) is ancillary. Also the conditional probability,
P (S(X) = s|T (X) = t) = P (X ∈ {x : S(x) = s}|T (X) = t),
does not depend on θ because T (X) is a sufficient statistic (recall the definition!). Thus, to show that S(X) and T (X) are independent, it suffices to show that
P (S(X) = s|T (X) = t) = P (S(X) = s) (6.2)
for all possible values t ∈ T . Now, P (S(X) = s) =X
t∈T
P (S(X) = s|T (X) = t)Pθ(T (X) = t).
Furthermore, since P
t∈T Pθ(T (X) = t) = 1, we can write P (S(X) = s) =X
t∈T
P (S(X) = s)Pθ(T (X) = t)
Therefore, if we define the statistic
g(t) = P (S(X) = s|T (X) = t) − P (S(X) = s), the above two equations show that
Eθg(T ) =X
t∈T
g(t)Pθ(T (X) = t) = 0
for all θ. Since T (X) is a complete statistic, this implies that g(t) = 0 for all possible values t ∈ T . Hence, (6.2) is verified. ¤
It should be noted that the “minimality” of the sufficient statistic was not used in the proof of Basu’s theorem. Indeed, the theorem is true with this word omitted, because a fundamental property of a complete statistic is that it is minimal.
Theorem 6.2.6 (Complete statistics in the exponential family) Let X1, . . . , Xn be iid ob- servations from an exponential family with pdf or pmf of the form
f (x|θ) = h(x)c(θ) exp¡Xk
j=1
wj(θ)tj(x)¢ ,
where θ = (θ1, θ2, . . . , θk). Then the statistic T (X) = (
Xn i=1
t1(Xi), Xn
i=1
t2(Xi), . . . , Xn
i=1
tk(Xi))
is complete if {w1(θ), . . . , wk(θ)) : θ ∈ Θ} contains an open set in Rk.
Example 6.2.15 (Using Basu’s theorem) Let X1, . . . , Xn be iid exponential observations with parameter θ. Consider computing the expected value of
g(X) = Xn
X1+ · · · + Xn.
We first note that the exponential distributions form a scale parameter family and thus g(X) is an ancillary statistic. The exponential distributions also form an exponential family with t(x) = x and so, by Theorem 6.2.6,
T (X) = Xn i=1
Xi
is a complete statistic and by, Theorem 6.2.3, T (X) is a sufficient statistic. Hence, by Basu’s theorem, T (X) and g(X) are independent. Thus we have
θ = EθXn= EθT (X)g(X) = (EθT (X))(Eθg(X)) = nθEθg(X).
Hence, for any θ, Eθg(X) = 1/n.
6.3 The likelihood Principle
6.3.1 The likelihood function
Definition 6.3.1 Let f (x|θ) denote the joint pdf or pmf of the sample X = (X1, . . . , Xn). Then, given that X = x is observed, the function of θ defined by
L(θ|x) = f (x|θ) is called the likelihood function.
This definition almost seems to be defining the likelihood function to be the same as the pdf or pmf. The only distinction between these two functions is which variable is considered fixed and which is varying. When we consider the pdf or pmf f (x|θ), we are considering θ as fixed and x as the variable; when we consider the likelihood function L(θ|x), we are considering x to be the observed sample point and θ to be varying over all possible parameter values. If we compare the likelihood function at two parameter points and find that
L(θ1|x) > L(θ2|x),
then the sample we actually observed is more likely to have occurred if θ = θ1 than if θ = θ2, which can be interpreted as saying that θ1 is a more plausible value for the true value of θ than is θ2. We carefully use the word “plausible” rather than “probable” because we often think of θ as a fixed value. Furthermore, although f (x|θ), as a function of x, is a pdf, there is no guarantee that L(θ|x), as a function of θ, is a pdf.
LIKELIHOOD PRINCIPLE: If x and y are two sample points such that L(θ|x) is propor- tional to L(θ|y), that is, there exists a constant C(x, y) such that
L(θ|x) = C(x, y)L(θ|y)
for all θ, then the conclusions drawn from x and y should be identical.
Note that C(x, y) may be different for different (x, y) pairs but C(x, y) does not depend on θ.
6.4 The Equivariance Principle
The first type of equivariance might be called measurement equivariance. It prescribes that the inference made should not depend on the measurement scale that is used.
The second type of equivariance, actually an invariance, might be called formal invariance. It states that if two inference problems have the same formal structure in terms of the mathematical model used, then the same inference procedure should be used in both problems. The elements of the model that must be the same are: Θ, the parameter space; {f (x|θ) : θ ∈ Θ}, the set of pdfs or pmfs for the sample; and the set of allowable inferences and consequences of wrong inferences.
For example, Θ may be Θ = {θ : θ > 0} in two problems. But in one problem θ may be the average price of a dozen eggs and in another problem θ may refer to the average height of giraffes.
Yet, formal invariance equates these two parameter spaces since they both refer to the same set of real numbers.
Equivariance Principle: If Y = g(X) is a change of measurement scale such that the model for Y has the same formal structure as the model for X, then an inference procedure should be both measurement equivariant and formally equivariant.
Example 6.4.1 (Binomial equivariance) Let X ∼ Bionmial(n, p) with known n and p. Let T (x) be the estimate of p that is used when X = x is observed. Rather than using the number of successes, X, to make an inference about p, we could use the number of failures, Y = n − X. We
can see that Y ∼ Binomial(n, q = 1 − p). Let T∗(y) be the estimate of q that is used when Y = y is observed, so that 1 − T∗(y) is the estimate of p when Y = y is observed. If x successes are observed, then the estimate of p is T (x). But if there are x successes, then there are n − x failures and 1 − T∗(n − x) is also an estimate of p. Measurement equivariance requires that these two estimates be equal, that is, T (x) = 1 − T∗(n − x), since the change from X to Y is just a change in measurement scale. Furthermore, the formal structures of the inference problems based on X and Y are the same. X and Y both have binomial(n, θ) distributions, 0 ≤ θ ≤ 1. So formal invariance requires that T (z) = T∗(z) for all z = 0, . . . , n. Thus, measurement and formal invariance together require that
T (x) = 1 − T∗(n − x) = 1 − T (n − x).
If we consider only estimators satisfying the above equality, then we have greatly reduced and sim- plified the set of estimators we are wiling to consider. Whereas the specification of an arbitray estimator requires the specification of T (0), . . . , T (n), the specification of an estimator satisfying the above equality requires the specification only of T (0), . . . , T ([n/2]), where [n/2] is the greatest integer not larger than n/2. This is the type of data reduction that is always achieved by the equiv- ariance principle. The inference to be made for some sample points determines the inference to be made for other sample points.
