Intention to Treat and Substitution Bias

The objective of most experimental designs is to estimate the conditional mean impact of training, or E(¢ j X; D = 1). However, in many experiments a signi…cant fraction of the treatment group drops out of the program and does not receive the services being evaluated.²⁰ In general, in the presence of dropping out E(¢ j X; D = 1) cannot be identi…ed using comparisons of means. Instead, the experimental mean di¤erence estimates the mean e¤ect of the o¤er of treatment, or what is sometimes called the “intent to treat.”

For many purposes, this is the policy-relevant parameter. It is informative on how the availability of a program a¤ects participant outcomes. Attrition is a normal feature of an ongoing program.

To obtain an estimate of the impact of training on those who actually receive it, ad-ditional assumptions are required beyond (5.A.1) or (5.A.2a) and (5.A.2b). Let T be an indicator for actual receipt of treatment, with T = 1 for persons actually receiving train-ing, and T = 0 otherwise. Let T^¤ be a similarly de…ned latent variable for control group

20Using the analysis in the preceding subsection, dropping out by experimental treatment group members could be reduced by compensating them for completing training.

members indicating whether or not they would have actually received training, had they been in the treatment group. De…ne

E(¢j X; D = 1; R = 1; T = 1) = E(¢ j X; D = 1; T = 1)

as the mean impact of training on those members of the treatment group who actually receive it. This parameter will equal the original parameter of interest E(¢ j X; D = 1) only in the special cases where (5.A.3), the common e¤ect assumption, holds, or where an analog to (5.A.4) holds so that the decision of treatment group members to drop out is independent of (¢ ¡ E(¢)), the person-speci…c component of their impact.

A consistent estimate of the impact of training on those who actually received it can be obtained under the assumption that the mean outcome of the treatment group dropouts is the same as that of their analogs in the control group, so that

(5.A.6) E(Y j X; D = 1; R = 1; T = 0) = E(Y j X; D = 1; R = 0; T^¤ = 0):

Note that this assumption rules out situations where the treatment group dropouts receive potentially valuable partial treatment. Under (5.A.6),

(5.1) E(Y j X; D = 1; R = 1) ¡ E(Y j X; D = 1; R = 0) P (T = 1j X; D = 1; R = 1)

identi…es the mean impact of training on those who receive it.²¹ This estimator scales up the experimental mean di¤erence estimate by the fraction of the treatment group receiving training. When all treatment group members receive training, the denominator equals one and the estimator reduces to the simple experimental mean di¤erence. Estimator (5.1) also shows that the simple mean di¤erence estimator provides a downward biased estimate of the mean impact of training on the trained when there are dropouts from the treatment group, because the denominator always lies between zero and one. Heckman, Smith and Taber (1998) present methods for estimating distributions of outcomes and for testing the identifying assumptions in the presence of dropping out. They present evidence on the validity of the assumptions that justify (5.1) in the National JTPA Study data.

In an experimental evaluation, the converse problem can also arise for the control group members. In an ideal experiment, no control group members would receive either the ex-perimental treatment or close substitutes to it from other sources. In practice, a signi…cant fraction of controls often receives similar services from other sources. In this situation, the mean earnings of control group members no longer correspond to E(Y0 j X; D = 1) and neither the experimental mean di¤erence estimator nor the adjusted estimator (5.1)

identi-…es the impact of training relative to no training for those who receive it. However, under certain conditions discussed in Section 3, the experimental estimate can be interpreted as the mean incremental e¤ect of the program relative to a world in which it does not exist.

21See, e.g., Mallar (1978), Bloom (1984) and Heckman, Smith and Taber (1998).

As in the case of treatment group dropouts, identifying the impact of training on the trained in the presence of control group substitution requires additional assumptions beyond (5.A.1) or (5.A.2a) and (5.A.2b). Let S = 1 denote control group members receiving substitute training from alternative sources and let S = 0 denote control group members receiving no training and let Y2be the outcome conditional on receipt of alternative training.

Consider the general case with both treatment group dropping out and control group substitution. In this context, one approach would be to invoke the assumptions required to apply non-experimental techniques as described in Section 7 to the treatment group data to obtain an estimate of the impact of the training being evaluated on those who receive it.

Heckman, Hohmann, Khoo and Smith (1998) employ this and other strategies using data from the National JTPA Study.

Alternatively, two other assumptions allow use of the control group data to estimate the impact of training on the trained. The …rst assumption is a generalized common e¤ect assumption, where to distinguish individuals we restore subscript i

(5.A.3⁰) Y_1i¡ Y0i= Y_2i¡ Y0i= ¢_i ´ ¢ for all i.

This assumption states that (a) the impact of the program being evaluated is the same as the impact of substitute programs for each person and (b) that all persons respond exactly the same way to the program (a common e¤ect assumption). The second assumption is a generalized version of (5.A.4), where

(5.A.4⁰) E(Y1¡ Y0 j X; D = 1; T = 1; R = 1) = E(Y2¡ Y0 j X; D = 1; S = 1; R = 0):

This assumption states that the mean impact of the training being evaluated received by treatment group members who do not drop out equals the mean impact of substitute train-ing on those control group members who receive it. Both (5.A.3⁰) and (5.A.4⁰) are strong assumptions. To be plausible, either would require evidence that the training received by treatment group members was similar in content and duration to that received by control group members. Note that (5.A.3⁰) implies (5.A.4⁰). Under either assumption, the ratio (5.2) E(Y j X; D = 1; R = 1) ¡ E(Y j X; D = 1; R = 0)

Pr(T = 1j X; D = 1; R = 1) ¡ Pr(S = 1 j X; D = 1; R = 0)

identi…es the mean impact of training on those who receive it in both the experimental treatment and control groups, provided that the denominator is not zero. The similarity of estimator (5.2) to the instrumental variable estimator de…ned in Section 7 is not accidental;

under assumptions (5.A.3⁰) or (5.A.4⁰), random assignment is a valid instrument for training because it is correlated with training receipt but not with any other determinants of the outcome Y . Without one of these assumptions, random assignment is not, in general, a valid instrument (Heckman, 1997; Heckman, Hohmann, Khoo and Smith, 1998). To see this point, consider a model in which individuals know their gain from training, but because the treatment group has access to the program being evaluated, it faces a lower cost of training. In this case, controls are less likely to be trained, but the mean gross impact

would be larger among control trainees than among the treatment trainees. Drawing on the analysis of Section 7, this correlation violates the condition required for the IV estimator to identify the parameter of interest.